MOOR security

Dieser Eintrag ist auf Englisch verfasst.

Draws that point at their state instead of carrying it

Entry 057839 of 943 tests pass

Each recorded draw carried 1.8 KB of pipeline and render state; sharing that state between draws halved the time to record 10,000 draws, while a test of removing GPU barriers showed they cost little.

What changed

A D3D12 game records draws into a command list (a buffer of commands replayed later on the GPU queue). mtl12 wrote each draw as a 1848-byte record: the pipeline, viewports, render targets, vertex buffers and other state, plus the draw's own counts. Almost every draw repeats the previous draw's state. In the bgfx drawstress sample, 99.99% of consecutive draws differ only in their counts and root arguments (small per-draw constants).

Now the state goes into a separate 1768-byte block. It is recorded only when it differs from the last one, and each draw is 88 bytes plus a pointer to its block. The functions that change state, such as SetPipelineState or RSSetViewports, set a dirty flag. A draw with the flag clear reuses the block without building anything.

Earlier in the session I tested GPU barriers (waits between passes so one pass sees the previous pass's writes). With every barrier removed, GPU busy time stayed within run-to-run noise for ModelViewer and drawstress. Narrower barriers stay on the shelf.

Draw records before (1848 bytes each) and after (88 bytes, pointing at a shared block).Before: every draw carries its state: draw 1 1848 bytes, draw 2 1848 bytes, draw 3 1848 bytes; After: draws point at a shared state block: state block 1768 bytes, draw 1 88 bytes, draw 2 88 bytes, draw 3 88 bytes; draw 1 1848 bytes → draw 2 1848 bytes; draw 2 1848 bytes → draw 3 1848 bytes; draw 1 88 bytes → draw 2 88 bytes; draw 2 88 bytes → draw 3 88 bytes; draw 1 88 bytes → state block 1768 bytes; draw 2 88 bytes → state block 1768 bytes; draw 3 88 bytes → state block 1768 bytesBefore: every draw carries its stateAfter: draws point at a shared state blockdraw 11848 bytesdraw 21848 bytesdraw 31848 bytesstate block1768 bytesdraw 188 bytesdraw 288 bytesdraw 388 bytes
Draw records before (1848 bytes each) and after (88 bytes, pointing at a shared block). (Mermaid source)
How a draw decides whether to reuse, build or record a state block.Draw call → a state setter ran since the last draw?; a state setter ran since the last draw? → reuse the last state block (no); a state setter ran since the last draw? → build the state in scratch memory (yes); build the state in scratch memory → same bytes as the last block?; same bytes as the last block? → reuse the last state block (yes); same bytes as the last block? → record a new state block (no); reuse the last state block → write the 88-byte draw pointing at the block; record a new state block → write the 88-byte draw pointing at the block; check mode: always build and compare → build the state in scratch memoryDraw calla state setter ransince the last draw?reuse the laststate blockbuild the statein scratch memorysame bytes asthe last block?record a newstate blockwrite the 88-byte drawpointing at the blockcheck mode:always build and comparenoyesyesno
How a draw decides whether to reuse, build or record a state block. (Mermaid source)

The interesting part

My first version built the state on every draw and compared it with memcmp against the last block. It made recording slower: 1.17 to 1.86 ms for 10,000 draws. Building and comparing 1.8 KB costs more than writing it. The dirty flag fixed that, but it creates a new kind of bug: a code path that changes state without setting the flag would draw with stale state. So there is a check mode (MTL12_CHECK_DRAW_STATE=1) that builds and compares on every draw anyway and logs a mismatch. A full test run in that mode found one such path: ExecuteIndirect clearing vertex and index buffer slots. It is fixed, and an independent review found no others.

The other surprise: the replay side (the Unix code that turns records into Metal calls) barely got faster, about 0.5 ms of 33. In an earlier test, padding the old record by 2 KB had added 9 ms there. So a bigger record hurts replay, but a smaller one doesn't help, and I can't explain why yet.

Numbers

Three alternating rounds per build:

  • bench-draws record time: 1.16 to 1.18 ms before, 0.50 to 0.51 ms after.
  • drawstress frame: 48.75 to 49.27 ms before, 44.01 to 44.71 ms after.
  • ModelViewer: unchanged (unix CPU 2.13 to 2.31 ms before, 2.13 to 2.35 ms after).
  • Barrier test: ModelViewer GPU busy time 11.48 ms with all barriers, 11.16 ms with none.
  • Tests: 839 of 943 pass, one more than before (the new api/draw_state_blocks), none newly failing.
Shared draw state, % of the old build (medians of 3 alternating rounds, each A/B against its own old runs) (data as CSV)
  • old build (1.17 / 6.40 / 48.86 / 32.91 ms)
  • state block without dirty flag
  • state block with dirty flag

measurement · % of old build

  1. bench-draws record100 old build (1.17 / 6.40 / 48.86 / 32.91 ms), 159 state block without dirty flag, 43 state block with dirty flag
  2. bench-draws frame100 old build (1.17 / 6.40 / 48.86 / 32.91 ms), 108 state block without dirty flag, 88 state block with dirty flag
  3. drawstress frame100 old build (1.17 / 6.40 / 48.86 / 32.91 ms), 109 state block without dirty flag, 91 state block with dirty flag
  4. drawstress unix CPU100 old build (1.17 / 6.40 / 48.86 / 32.91 ms), 99 state block without dirty flag, 98 state block with dirty flag
Tests by suite at the end of this entry: 839 of 943 pass (data as CSV)
  • pass
  • fail
  • skip

suite · share of each row's total

  1. vkd3d-proton490 pass, 15 fail, 72 skip of 577
  2. vkd3d177 pass, 9 fail, 4 skip of 190
  3. apps77 pass, 0 fail, 0 skip of 77
  4. api (+1)75 pass, 0 fail, 0 skip of 75
  5. wine-dxgi16 pass, 4 fail, 0 skip of 20
  6. samples4 pass, 0 fail, 0 skip of 4
Zur Eintragsliste