A Metal draw costs more when its vertex shader reads the base instance, whatever the value

What changed

No code. This session measured where the remaining CPU time of bgfx's draw stress test goes (64,000 draws a frame) and wrote it down: a spike, an addendum to the previous entry's design and two notes.

Last session, each draw's root arguments started reaching the shader through the draw's base instance, the start value Metal adds to the instance index. Before that, mtl12 changed a buffer address in Metal's argument table before every draw. That made drawstress's unix-side CPU time (the part of mtl12 that runs as a macOS library under Rosetta and calls Metal) drop from 12.7 to 8.6 ms a frame. A build that didn't bind the root arguments at all, so its pictures were wrong, reached 5.3 ms. I had called that 5.3 ms the ceiling and the 3.3 ms between it and 8.6 ms unexplained.

The interesting part

The ceiling wasn't a fair comparison. Its pipelines didn't read the base instance, and reading it has a cost even when the value is constant. A small standalone program encodes 50,000 indexed draws with six vertex functions:

• a draw whose vertex function reads only the vertex ID costs about 117 ns under Rosetta;
• one that reads the base instance, the instance ID or the base vertex costs about 151-158 ns;
• a base instance of 0 costs as much as a different one on each draw.

Natively it is 32 against 59-61 ns. I didn't read Metal's code, so where the cost comes from is a guess. It looks like Metal supplies these values per draw once the function uses them.

In drawstress the gap is larger. Skipping the Metal draw call leaves 3.33 ms of our own replay work. Draws through pipelines that read no base instance add about 2.5 ms (40 ns a draw), and draws through the base-instance pipelines add about 6.3 ms (98 ns a draw). The address change it replaced cost more, so the design stays. Metal 4's encoders have no setBytes, and I found no other way to give each draw its own data.

The second finding was about the other side of the frame. About 13.7 ms of the 23 ms frame runs on the same thread outside our unix library: bgfx, Wine, Rosetta, and our own Windows-side DLL, which records the commands. A sampling profiler couldn't split that time, because under Rosetta the sampled addresses in Windows code aren't the code's own addresses. Timers around the 18 recording methods bgfx calls per draw put our part at 15% of the frame, about 3.5 ms.

Numbers

• Spike, ns per draw (median of 5 command buffers, two rounds): Rosetta 117.0-121.4 (vertex ID only) against 150.8-164.6 (base instance, instance ID, base vertex); native 31.7-33.3 against 56.3-68.6.
• Drawstress unix CPU per frame, two runs each: as committed 9.58-9.78 ms; offset pipelines with base instance 0 9.35-9.63 ms; pipelines that read no base instance 5.82-5.87 ms; no Metal draw call 3.33-3.35 ms.
• Our recording, steady state: 0.153-0.154 of wall time; frame 23.05 ms, unix CPU 9.33 ms in the same run.
• Skipping setArgumentTable while the table hasn't changed: 8.78 / 9.31 ms against 9.03 / 8.92 ms, so no gain.
• Full suite: 848 pass, 28 fail, 76 skip of 952, the same tests as at the end of entry 063 (no source changed).