ExecuteIndirect through Metal indirect command buffers, and a compiler that crashes instead of failing

What changed

mtl12 translates Direct3D 12 to Apple's Metal so that Windows games can run under Wine on Apple Silicon. ExecuteIndirect is D3D12's GPU-driven draw call: a buffer holds a list of commands, and each command can set root constants, vertex buffers and an index buffer before its draw. Until now mtl12 ran each command as a Metal indirect draw and ignored the vertex and index buffer arguments.

Now a small compute kernel reads the D3D12 commands and writes a Metal indirect command buffer (ICB, a list of draws that the GPU itself can fill in). The render pass then runs the whole list with one executeCommandsInBuffer call. Each command gets its own vertex buffers, strides and index buffer, including null views and 16- or 32-bit indices. A new api test, execute_indirect_state, was written first and covers these cases. It also passes under Metal's validation layer. Command signatures that don't hold exactly one draw or dispatch now fail with E_INVALIDARG, as on Windows.

The interesting part

Two things in Metal behaved in ways its documentation doesn't mention. I isolated both in standalone spikes.

Odd bind counts shift strides. An ICB declares how many vertex buffers its commands may bind. With 29, 31, 11 or 13, the second command in the list fetched vertices with each slot's stride shifted by one slot. With 30, 12, 14 or 16, both commands were correct. The validation layer reported nothing. My guess is a packing bug in the per-command layout, but I have only seen the effect. mtl12 now uses 30, which leaves 20 vertex buffer slots for applications.

A shader class that crashes the compiler. The full test run found one regression, vkd3d-proton's test_multisample_resolve_formats. A pipeline that supports ICBs, with a pixel shader that runs per sample, calls get_sample_position() and writes depth, makes Apple's GPU compiler print "second-level indirections not handled in EI uniform ABI" and crash the process. It doesn't return an error. Under Wine this surfaced as an access violation and sometimes a hang. The spike compiles each shader variant in a child process: dropping either the sample-position call or the depth write avoids the crash, and pipelines without ICB support are fine. Because nothing returns, retrying after a failure isn't possible. mtl12 therefore checks the generated Metal shader source before compiling and builds these pipelines without ICB support. Their ExecuteIndirect commands then run one by one, without per-command vertex or index buffers. That gap is marked FIXME. A second new api test, execute_indirect_per_sample, reproduces the crash, hanging without the fix and passing with it.

Numbers

• Full run at the end of the session: 522 pass, 89 fail, 176 skip of 787 tests, 0 newly failing. The previous full run, at the end of entry 016, was 515/91/177 of 783. Of the 7 tests that changed status, 4 are new api tests and 3 are vkd3d-proton tests fixed in entry 017. None of the vkd3d-proton ExecuteIndirect tests changed status: test_execute_indirect_state still fails because it reads its results through stream output, which mtl12 doesn't support yet.
• A full run in the middle of the session, before the compiler-crash workaround, scored 520/90/176 of 786 with the one regression.
• Odd-bind-count spike: counts 11, 13, 29 and 31 are wrong from the second command on; 12, 14, 16 and 30 are correct (32 fails descriptor validation).