Tessellation passes: 31 more tests, and a bug that was a zero

What changed

Entry 026 described how a tessellated D3D12 draw runs on Metal: five steps in one render encoder, with the hull shader and the tessellator as object functions (the compute-like first stage of Metal's mesh pipeline). At that point, 8 of 33 tessellation tests passed. This entry covers the fixes that took it to 31 of 32. The one left contradicts another suite. Tests that combine tessellation with geometry shaders still fail, because geometry shaders are next.

Most failures were at the stage interfaces, where one shader's outputs must line up with the next one's inputs. SPIRV-Cross (the SPIR-V to Metal shader translator) rejects the "Component" decoration in tessellation stages. That is the decoration that lets D3D pack two variables into one four-component register. Packed inputs now get a location of their own, and the producing stage follows them by name. The last and most surprising failure was one integer. The translator reports rows = 0 for a plain (non-array) signature element, and the linker stepped past unused outputs by rows. Every unused output then landed on the same location, and the hull shader's output record no longer matched the domain shader's input record.

Stream output (D3D's way of writing a stage's vertices to a buffer) now works from domain shaders. A second copy of the domain shader writes records instead of positions. It is drawn as points, and each point looks up its domain vertex in the tessellator's index buffer. A one-thread object function plans how many whole primitives fit into the target buffers.

A later review found that a tessellated draw ignored SetPredication (the API that skips draws based on a GPU value) and still rendered. The fix is one pointer: the tessellator's prefix-sum step reads the draw's predicate record and writes an index count of zero when the predicate fails. A new test covers it.

The interesting part

One vkd3d test and one vkd3d-proton test make the same call and expect different results. It creates a tessellation pipeline with a TRIANGLE topology type. vkd3d expects E_INVALIDARG, while vkd3d-proton expects success because native drivers accept it. I follow vkd3d-proton and leave the vkd3d test failing on purpose.

Numbers

• Full run: 610 pass, 53 fail, 145 skip of 808. Before: 580 pass of 807. All 31 tests that went from fail to pass are tessellation tests.
• Tessellation subset: 8 of 33 at the start, 31 of 32 after the fixes, 34 of 35 with the new predication test.
• The new predication test failed 64 checks before the fix and passed all 196 after.
• Two tile-mapping tests failed in the full run. They also fail with the code from before this work, while another app was loading the GPU, so I count them as an existing race rather than a regression. This is unverified beyond that.