Barycentrics on a GPU API with no per-vertex inputs

What changed

mtl12 translates Direct3D 12 to Apple's Metal, so Windows games can run under Wine on Apple Silicon. This entry covers two sessions after entry 022 (entry 023, the room sample, was written separately). Each feature started with a failing test.

• Native 16-bit types. Shaders compiled with -enable-16bit-types use real 16-bit floats and integers. They mostly worked already; the device just didn't report the cap. Two translation settings had to change: buffer views may start on any 2-byte boundary, and fp32 → fp16 → fp32 round trips must not go through a helper that flushes tiny values to zero.
• Barycentrics. See below.
• Shader Model 6.7. The two features 6.7 requires, QuadAny/QuadAll and helper lanes in wave operations, already worked. Only the reported version changed.
• Stencil reference from the pixel shader. SV_StencilRef lets a pixel shader choose the stencil value per pixel. The shader translator already mapped it to Metal's [[stencil]] output; the cap was missing. A new test writes 64 different values into two stencil formats.

The interesting part

SV_Barycentrics gives a pixel its three weights inside the triangle. GetAttributeAtVertex(attr, i) returns the value that vertex i wrote, before interpolation. Metal has barycentric weights, but no way to read a single vertex's output: a fragment input is either interpolated or taken from the first vertex only.

The workaround (SPIRV-Cross patch 0019, diagram below) uses screen-space linearity. An attribute interpolated without perspective is a linear function of the no-perspective weights. The pixel's 2x2 quad supplies derivatives of both. Solving a 2x2 system gives the change per weight, and from that the three vertex values. The error was at most 6.7e-5 for values up to about 250, under the 9.8e-5 vkd3d-proton's test allows.

Then a surprise, which a new test found. A pixel shader that reads only perspective barycentrics got wrong weights. Metal only applies perspective correction when the fragment function also declares [[position]] or reads another perspective input. Without either, the first two weights came back divided by each vertex's w. Patch 0020 now declares the position whenever barycentrics are read.

A review found the fix's own bug. In a per-sample shader, SPIRV-Cross adds the sample's offset to that position through the sample ID. That ID was never declared, so the shader failed to compile. The patch now requests it too, and the test covers a per-sample shader.

Numbers

• Full run at the end: 577 pass, 82 fail, 147 skip of 806 tests. At the end of entry 022: 558, 79 and 162 of 799.
• All 19 new passes: seven new tests (api native_16bit, barycentrics, barycentrics_perspective_only, attribute_at_vertex_index, sm67_quad_helpers, stencil_export, and the room sample of entry 023), and twelve vkd3d-proton tests that skipped before: five for 16-bit types, test_sv_barycentric, two SM 6.7 tests, both stencil export tests, test_shader_instructions_dxil and test_buffers_oob_behavior_vectorized_byte_address. Nothing that passed before fails.
• Three tests that skipped before now run and fail, each for a known Metal limit: test_sm67_helper_lane_wave_ops (672 checks: Metal splits a clipped full-screen triangle into two quads), test_large_buffer_descriptors (315: Metal only maps private heaps into sparse buffers) and test_denorm_behavior_dxil (1: 32-bit denormals are flushed).
• New api tests: barycentrics 707 checks, stencil_export 800.
• Without patch 0020, barycentrics_perspective_only failed 192 checks.