MOOR security

Dieser Eintrag ist auf Englisch verfasst.

Culling triangles in a pass that has no pixel shader

Entry 072868 of 963 tests pass

mtl12 now honours Direct3D 12 cull distances in depth-only passes by generating an empty pixel shader for SPIRV-Cross to add the discard to, and a gallery scene found a second cull-distance bug on the way.

What changed

A cull distance is a number a vertex shader writes per vertex (SV_CullDistance). Direct3D 12 throws away a whole triangle when one of its cull distances is negative at all three vertices. Metal has clip distances but no cull distances, so mtl12 emulates them in the pixel shader (the program that colours each pixel): a SPIRV-Cross patch recovers each distance's value at the three corners from the interpolated value and its screen-space derivatives, and discards the pixel if the triangle should be culled (patch 0008).

That left out pipelines with no pixel shader at all, such as depth-only passes that only fill the depth buffer. They ignored cull distances and printed a fixme. Now mtl12 hands SPIRV-Cross an empty fragment shader, 32 hand-written SPIR-V words (SPIR-V is the shader format between the Direct3D bytecode and Metal's shading language). It goes through the same compile step as an application's shader, so the existing patch adds the discard, and the depth-bounds and sample-mask code come along too. Colour writes are masked, because Direct3D writes no colour without a pixel shader.

The two routes to the cull discard.vertex, domain or geometry shader writes SV_CullDistance → pixel shader?; pixel shader? → application pixel shader DXIL to SPIR-V (dxil-spirv) (yes); pixel shader? → empty fragment module 32 hand-written SPIR-V words (no (before: ignored, fixme)); application pixel shader DXIL to SPIR-V (dxil-spirv) → SPIR-V to MSL (SPIRV-Cross) patch 0008: recover each distance at the 3 vertices, discard if one is negative at all of them patch 0045: keep declared but unread inputs; empty fragment module 32 hand-written SPIR-V words → SPIR-V to MSL (SPIRV-Cross) patch 0008: recover each distance at the 3 vertices, discard if one is negative at all of them patch 0045: keep declared but unread inputs; SPIR-V to MSL (SPIRV-Cross) patch 0008: recover each distance at the 3 vertices, discard if one is negative at all of them patch 0045: keep declared but unread inputs → Metal pipeline no app pixel shader: colour writes maskedvertex, domain or geometry shaderwrites SV_CullDistancepixel shader?application pixel shaderDXIL to SPIR-V (dxil-spirv)empty fragment module32 hand-written SPIR-V wordsSPIR-V to MSL (SPIRV-Cross)patch 0008: recover each distance at the 3 vertices,discard if one is negative at all of thempatch 0045: keep declared but unread inputsMetal pipelineno app pixel shader: colour writes maskedyesno (before: ignored, fixme)
The two routes to the cull discard. (Mermaid source)
Gallery scene cullnops: a depth pre-pass without a pixel shader culls a grid of triangles by their distance from the white circle (SV_CullDistance); the colour pass draws all of them dim and, with depth test EQUAL, bright where the pre-pass left depth: triangles fully outside are culled, triangles the circle crosses are kept whole; animated, the circle grows and shrinks. All scenes in the gallery

The interesting part

The gallery scene for this feature found a second bug. Its pixel shader takes the vertex shader's whole output struct, cull distance included, but doesn't read the distance. Games do this all the time. Its pipeline failed to compile with "use of undeclared identifier 'gl_CullDistance'". SPIRV-Cross leaves inputs a shader never reads out of the input struct, and patch 0008 then referred to a variable that wasn't there. Patch 0045 marks those inputs as used. No test in the suites had this pattern. The API test now has a second pipeline with such a shader, and without the patch that pipeline fails to create.

Numbers

  • Full run: 868 pass, 27 fail, 68 skip of 963, against 867/27/68 of 962 at the end of the last session; 1 newly passing (api/cull_distance_depth_only, new), 0 newly failing.
  • api/cull_distance_depth_only: 129 checks. Without the generated shader, 32 of the first pipeline's 64 checks failed; without patch 0045, the second pipeline fails to create.
Tests by suite at the end of this entry: 868 of 963 pass (data as CSV)
  • pass
  • fail
  • skip

suite · share of each row's total

  1. vkd3d-proton498 pass, 14 fail, 65 skip of 577
  2. vkd3d178 pass, 9 fail, 3 skip of 190
  3. api (+1)92 pass, 0 fail, 0 skip of 92
  4. apps80 pass, 0 fail, 0 skip of 80
  5. wine-dxgi16 pass, 4 fail, 0 skip of 20
  6. samples4 pass, 0 fail, 0 skip of 4
Zur Eintragsliste