MOOR security

Dieser Eintrag ist auf Englisch verfasst.

Three ways to make a draw cheaper, none kept

Entry 054837 of 941 tests pass

Three ideas to cut the CPU cost of a draw call in a D3D12-to-Metal layer: one did nothing measurable, one broke rendering, and one helped a benchmark in one run and hurt it in another.

What changed

The layer's source ends the session where it started. Replay (the macOS side that turns recorded D3D12 commands into Metal calls) costs about 520 ns of CPU per draw in bgfx's drawstress example (about 61,000 draws a frame). I skipped each part of the draw path in turn and measured what went away (see the cost chart). No part dominates, and most of the time goes into Metal calls, which run under Rosetta (Apple's x86-to-ARM translator) just as the layer does.

I tried three changes (see the diagram):

  1. Cached upload memory. A spike (a small standalone program) showed that an atomic operation right after a store to write-combined memory (memory the CPU writes around its caches) costs 81 to 92 ns, against 7 ns otherwise. Replay writes each draw's root arguments to such a buffer, and atomics follow. Making the buffer cached changed nothing measurable, so I didn't keep it.
  2. One argument-table bind per encoder. The Metal 4 header says the table's contents are snapshotted at each draw, so binding it once should be enough. In practice ModelViewer's GPU time fell from 12.3 to 1.8 ms: most draws had silently vanished. The reason is still unknown.
  3. Zero less of each draw record. Each recorded draw is a 1848-byte struct, cleared in full before it is filled in. Clearing only the fields read without a count looked like a 6% cut in recording time. Back-to-back A/B runs said otherwise (see the A/B chart).
The three changes and what became of each.drawstress: about 520 ns of unix-side CPU per draw → Upload buffer not write-combined (an atomic after a WC store: 7 ns to 81-92 ns); drawstress: about 520 ns of unix-side CPU per draw → Bind the argument table once per encoder instead of once per draw; drawstress: about 520 ns of unix-side CPU per draw → Zero only part of the 1.8 KB draw record; Upload buffer not write-combined (an atomic after a WC store: 7 ns to 81-92 ns) → no measurable change not kept; spike kept; Bind the argument table once per encoder instead of once per draw → most draws lost; GPU 12.3 to 1.8 ms reverted; reason unknown; Zero only part of the 1.8 KB draw record → record +7.5%, frame -2.6%, unix CPU +0.6% review: uninitialised slots; reverteddrawstress: about 520 ns of unix-side CPU per drawUpload buffer not write-combined(an atomic after a WC store: 7 ns to 81-92 ns)Bind the argument table once per encoderinstead of once per drawZero only part of the 1.8 KB draw recordno measurable changenot kept; spike keptmost draws lost; GPU 12.3 to 1.8 msreverted; reason unknownrecord +7.5%, frame -2.6%, unix CPU +0.6%review: uninitialised slots; reverted
The three changes and what became of each. (Mermaid source)

The interesting part

The first measurement of change 3 showed bench-draws recording at 1.09 ms against 1.18 ms. An hour later, the same code measured 1.30 ms, and the old code 1.20 ms. On one machine, one run of a benchmark can drift by 15% within the hour, so only runs made minutes apart count as a comparison. The review also found a real hole: ExecuteIndirect could extend the vertex-buffer count past the part that was copied, which left slots uninitialised. So the change was slower on one benchmark, only slightly faster on the other, and less safe. I reverted it.

The perf harness caught change 2, as bench-room's GPU timestamps stopped advancing. I did not run the conformance tests on it.

Numbers

  • Spike: atomic after a write-combined store 81.3 and 92.0 ns, 7.0 ns otherwise; reads from write-combined memory 69 ns per cache line.
  • A/B for change 3: bench-draws record 1.20 to 1.29 ms, drawstress frame 48.6 to 47.3 ms, drawstress unix CPU 32.7 to 32.9 ms.
  • Full test run: 837 pass, 28 fail, 76 skip of 941, the same as the previous entry.
Unix-side CPU per drawstress draw, ns (about 520 in total; M1 Max, under Rosetta) (data as CSV)

part of replay · ns per draw

  1. outside cmd_draw (walk and dispatch)145
  2. render area; viewports; root upload92
  3. render_encoder46
  4. argument table and encoder state110
  5. draw paths and the Metal draw call126
One atomic add after a 28-byte store, ns (x86_64 under Rosetta, M1 Max) (data as CSV)

case · ns each

  1. atomic add alone (cached)7
  2. store to cached memory then atomic7
  3. atomic add alone (write-combined)7
  4. store to write-combined memory then atomic81.3
Partial zeroing of draw records against the full memset, back to back (before = 100) (data as CSV)

measurement · after, as % of before (median of the A/B runs; lower is faster)

  1. bench-draws record107.5
  2. drawstress unix-side CPU100.6
  3. drawstress frame97.4
Tests by suite at the end of this entry: 837 of 941 pass (data as CSV)
  • pass
  • fail
  • skip

suite · share of each row's total

  1. vkd3d-proton490 pass, 15 fail, 72 skip of 577
  2. vkd3d177 pass, 9 fail, 4 skip of 190
  3. apps77 pass, 0 fail, 0 skip of 77
  4. api73 pass, 0 fail, 0 skip of 73
  5. wine-dxgi16 pass, 4 fail, 0 skip of 20
  6. samples4 pass, 0 fail, 0 skip of 4