MOOR security

Dieser Eintrag ist auf Englisch verfasst.

Five walks over 110 MB a frame: letting the recorder say which pre-passes to run

Entry 053837 of 941 tests pass

Replay walked every recorded command five times per frame before drawing anything; flags summed while recording removed the walks and cut a 61,000-draw bgfx frame from 74.7 to 48.2 ms.

What changed

The layer records a game's D3D12 command lists on the Windows side into a stream of fixed-size records, and replays them into Metal on the macOS side. Before replay, and after every barrier (a D3D12 command that orders GPU work), five pre-passes walk the commands up to the next barrier. They convert indices for primitive restart, expand indirect draws (draws whose arguments come from a GPU buffer), resolve predication (conditional rendering), and prepare stream output (vertex data written back to a buffer). Each pass walked the whole range even when it had nothing to do.

bgfx's "drawstress" example draws about 61,000 cubes a frame with no barrier between them. Temporary timing marks put the five walks at 416 ns per draw, next to 554 ns for replaying the draw itself (see the cost chart).

The first fix walked the range once and ran only the passes it found work for. That scan alone still cost 121 ns per draw. The second moves the question to the recording side: each recorded command ORs a few flag bits into a word for its range, which lives in the barrier record that opens the range or, for the first range, is passed along with the list. Replay reads one word and never walks the range unless a pass has work (see the diagram).

How replay decides which pre-passes run: five walks, one scan, then flags summed while recording.after: the recording side sums flags: record a command, OR its flags into the range's word, barrier record or list header holds the word, replay reads one word, runs only flagged passes; before: one scan: walk the range once, read one field per record, run only passes with work; Before: five walks: indices, indirect, predication, indirect draws, stream output; record a command → OR its flags into the range's word; OR its flags into the range's word → barrier record or list header holds the word; barrier record or list header holds the word → replay reads one word, runs only flagged passes; walk the range once, read one field per record → run only passes with work; indices → indirect; indirect → predication; predication → indirect draws; indirect draws → stream outputafter: the recording side sums flagsbefore: one scanBefore: five walksrecord a commandOR its flags intothe range's wordbarrier record orlist header holds the wordreplay reads one word,runs only flagged passeswalk the range once,read one field per recordrun only passeswith workindicesindirectpredicationindirect drawsstream output
How replay decides which pre-passes run: five walks, one scan, then flags summed while recording. (Mermaid source)

The interesting part

Why does reading one field per record cost 121 ns? A draw record is 1848 bytes: 16 viewports and scissors, 32 vertex buffers, 8 render targets, stream output state. 61,000 of them are about 110 MB a frame, so every record read is a cache miss, no matter how little the code does with it. Shrinking the record would make replay cheaper as well; that is not started.

Every flag has a test that fails when the flag is dropped on purpose. One gap turned up: no test ended a list on a command that needs a pre-pass, so skipping the last fold at Close went unnoticed. api/execute_indirect_state now ends its list on an ExecuteIndirect behind a barrier.

Numbers

Medians on an M1 Max (the middle column measured before that change was committed):

scenebeforeone scanafter
drawstress frame74.7 ms54.7 ms48.2 ms
drawstress unix-side CPU58.7 ms38.9 ms32.3 ms
bench-draws frame (10000 draws)7.47 msnot measured6.29 ms

drawstress's GPU time also fell, from 4.30 to 3.80 ms; I have not looked into why.

Full test run: 837 pass, 28 fail, 76 skip of 941, the same as the previous entry. Nothing newly fails or passes.

bgfx drawstress per frame, ms (about 61,000 draws, M1 Max, medians) (data as CSV)
  • frame
  • unix-side CPU

version · ms per frame (median)

  1. five walks74.68 frame, 58.72 unix-side CPU
  2. one scan54.72 frame, 38.88 unix-side CPU
  3. recorded flags48.24 frame, 32.33 unix-side CPU
Pre-pass cost per draw in drawstress, ns (M1 Max, under Rosetta) (data as CSV)

pre-pass walk · ns per draw

  1. convert_indices121
  2. expand_indirect70
  3. predicate_commands58
  4. encode_indirect_draws63
  5. stream_output102
  6. the single scan121
Tests by suite at the end of this entry: 837 of 941 pass (data as CSV)
  • pass
  • fail
  • skip

suite · share of each row's total

  1. vkd3d-proton490 pass, 15 fail, 72 skip of 577
  2. vkd3d177 pass, 9 fail, 4 skip of 190
  3. apps77 pass, 0 fail, 0 skip of 77
  4. api73 pass, 0 fail, 0 skip of 73
  5. wine-dxgi16 pass, 4 fail, 0 skip of 20
  6. samples4 pass, 0 fail, 0 skip of 4
Zur Eintragsliste