Stream output on Metal, which has no transform feedback

What changed

mtl12 translates Direct3D 12 to Apple's Metal so that Windows games can run under Wine on Apple Silicon. Stream output (SO) is a D3D12 feature that writes chosen vertex shader outputs into up to four buffers, in draw order. Each draw appends at a per-buffer "filled size" counter that the GPU advances. Games use it for GPU particle systems and skinning caches. Test suites use it too, to read back what a vertex shader computed. Until now mtl12 rejected any pipeline that declared stream output.

The interesting part

Metal has no transform feedback. A Metal vertex function can write to device memory, but nothing in a normal draw numbers the output vertices the way D3D12 does. Strips share vertices between triangles, D3D12 writes them as expanded lists (triangle i of a strip is vertices i, i+1+i%2, i+2−i%2), and vertex invocations run in no fixed order.

So each SO pipeline gets a second vertex function. I added an option to SPIRV-Cross (the shader cross-compiler we use, patch 0013) that turns the shader's outputs into plain buffer writes at record × stride + offset, and I build that function into a pipeline with rasterization turned off. Before the draws, two small compute kernels run for each SO draw:

• a one-thread "plan" kernel counts the records, walks strip-cut indices, and decides how many whole primitives fit in every bound buffer (a primitive that doesn't fit in all of them is written to none). It then advances the filled sizes;
• an "expand" kernel gathers every record's vertex-buffer bytes and its vertex and instance IDs into scratch buffers, in D3D12's order.

The SO draw is then an indirect draw over the gathered data, one vertex per record, and the normal draw follows unless the pipeline disabled rasterization.

ExecuteIndirect draws, whose counts only the GPU knows, use the same kernels. The plan kernel walks the commands one after another, giving each a run of consecutive records, and the expand kernel finds a record's command with a binary search. The CPU cannot size the scratch buffers from vertex counts, so it sizes them from the room left in the target buffers. Every captured record has to fit there anyway.

Two things surprised me. First, Metal rejects a buffer argument that points to pointers (device uint* const device*). The table of target addresses has to be ulong and cast at each write. Second, a vkd3d-proton test expects indexed draws to run even with no index buffer bound, or with an index format of UNKNOWN, reading index 0. mtl12 used to drop those draws.

Two more came from vkd3d-proton tests once stream output worked. Draws with no viewports set still have to write stream output, and mtl12 was dropping them. After an ExecuteIndirect, the state its command signature sets (root constants, root descriptors, vertex and index buffer views) is expected to read as cleared, not as whatever the last command set.

Numbers

Full run at the end of the session: 536 pass, 82 fail, 170 skip of 788 tests. The session before ended at 522 pass, 89 fail, 176 skip of 787. No test got worse. Thirteen conformance tests changed to passing, among them the five whose names mention stream output, test_vertex_id (three variants), test_execute_indirect_state and test_vbv_stride_edge_cases. One new api test (stream_output, 5 cases) was added. Failed checks in the two conformance suites went from 3,843 to 3,811.