Ray tracing on by default, and a stack that has to fit

What changed

For several entries the layer has had DXR (DirectX Raytracing: the D3D12 API for tracing rays against scene geometry) behind an environment variable, reported only to our own tests. The ray-tracing plan set the rule for turning it on: report ray tracing tier 1.1 once the pipeline tests pass. They did, so the layer now reports tier 1.1 to every program. MTL12_RAYTRACING=0 hides it again for a program that misbehaves with it.

The first real program to use it was Diligent Engine's Tutorial21, which failed. It got no pipeline at all, and the fix is the interesting part. With that fixed, Tutorial21 matches Diligent's reference image (4 pixels more than 16 levels apart, at most 82) and is now part of the apps suite.

The interesting part

A DXR shader can call TraceRay from inside a hit shader: a reflection ray spawns another reflection ray, and so on. The program declares the deepest nesting it will use. On Metal, each shader is a "visible function" (a function called through a table at run time), and the pipeline has to be told a maxCallStackDepth when it is created.

That number is not a count of calls. It scales a stack budget, and how many calls fit depends on how big each function's frame (its locals and spilled registers) is. An earlier session learned the cheap side of this: too small a depth and a deep recursion fails the whole command buffer. This session found the other side. Too large a depth, and Metal refuses to create the pipeline: "Compute pipeline exceeds available stack space". Tutorial21 declares 8 levels. The layer asked for 4 × (8 + 2) = 40 as a safety margin, and Metal refused anything above 11.

So when Metal refuses, the layer now binary-searches for the largest depth it accepts. A refused attempt costs a few milliseconds. A new test, api/rt_recursion, nests 8 TraceRay calls with a deliberately fat frame: a 264-byte payload and a 64-word local array, checked after every level. The chart shows how far that test got at each depth. These counts come from temporary instrumented builds, which weren't kept; the project notes record them. The largest depth Metal accepts for this test, 12, runs 20 levels, not the 31 that D3D12 allows. A program that really recursed that deep would fail. The ray-tracing plan records this as a known limit.

Numbers

• Full run: 773 pass, 27 fail, 108 skip of 908, against 771 of 906 at the end of entry 037. 0 newly failing; both new passes are new tests. This was the first full run with tier 1.1 reported to every test, and no other result changed.
• Tutorial21: the largest accepted depth is 11 (40 refused). Against Diligent's Vulkan software golden, 4 pixels are more than 16 levels apart, at most 82.
• The sky in the reference image's top corners, (38, 209, 252), can be derived from the tutorial's camera and miss-shader palette. The apps test now checks that value (tutorial21_sky).