A hull shader without a hull stage
D3D12 tessellation now runs on Metal inside one render pass, using object functions (the first stage of Metal's mesh pipeline) as compute that barriers can order against vertex work.
What changed
D3D12 tessellation has three parts: a hull shader that turns input patches into control points and tessellation factors, a fixed-function tessellator that turns factors into a mesh of domain points, and a domain shader that places each point. Metal has none of the hull part. Its own tessellation reads factors from a buffer and supports triangles and quads only: no isolines, no point output. The usual answer is a compute kernel for the hull shader, but a compute encoder would end the render pass (the unit of work that owns the render targets) before every tessellated draw and start a new one after it.
I tried something else first, in a spike (a throwaway test program): run the hull shader as an object function, the first stage of Metal's mesh pipeline, inside the same render encoder. An object function is ordinary threadgroup compute that happens to live in a render pipeline. The design then builds a tessellated draw out of five steps in that one encoder, with a barrier between each:
- the vertex shader, drawn as points with rasterization off, writing its outputs to a buffer;
- the hull shader as an object function;
- the tessellator as three more object functions (count, prefix sum, emit);
- the domain shader as the vertex function of an indexed indirect draw;
- the pixel shader, as usual.
The tessellator is the D3D11 reference tessellator (Microsoft, MIT) in Mesa's OpenCL C port (MIT), adapted to Metal. It covers all three domains, the four partitioning modes, point output and both windings. A new SPIRV-Cross patch (0022) emits hull shaders as object functions and domain shaders as vertex functions reading a point table.
The interesting part
Whether barriers inside a render encoder really order vertex → object → vertex work was the whole question. The spike ran 200 iterations of 262,144 vertices through that chain and compared every value. With the barriers: 0 iterations with a mismatch. Without them: all 200 did. The object function also needed no payload and no mesh grid. Its mesh function never runs, and the pipeline can share the pass's attachments with rasterization off.
intra-pass barriers · iterations with a wrong result
- with barriers0
- without barriers200
Numbers
- Spike: 0 of 200 iterations wrong with intra-pass barriers, 200 of 200 without.
- First working version: 8 of 33 tessellation tests pass.
- The quick test subset on that version: 69 pass, 3 fail, 19 skip of 91, no new failures. There was no full run for this work.