21 Diligent Engine apps on Metal, checked against Diligent's own reference images

What changed

Diligent Engine is an open-source (Apache-2.0) rendering engine with a D3D12 backend and about thirty small sample programs. I built it with mingw (a GCC cross-compiler for Windows programs) with only the D3D12 backend, and ran the programs under Wine on top of mtl12, this project's D3D12-to-Metal 4 layer. Upstream does not build that backend with mingw, so the build supplies a small replacement for ATL's COM smart pointers and a few patches, each marked in tools/diligent/.

Twenty tutorials and the NuklearDemo sample now pass the apps suite: they exit cleanly, print no errors, have the right background colour at two probe pixels, and match a reference image.

Earlier in the session:

• Logic ops. D3D12 can combine a pixel with the render target using bitwise operations (AND, XOR, ...). Metal has no such blend state, so the pixel shader reads the target with framebuffer fetch and applies the operation itself.
• A ray-tracing plan: Metal's ray-tracing API works on the M1 even though the M1 has no ray-tracing hardware.

The interesting part

I didn't have to invent the reference images. Diligent's sample framework has a golden-image mode. It renders one frame at time zero, presents it and exits, so even animated tutorials give the same frame every time. Diligent's CI stores that frame for every backend, including D3D12 on WARP (Microsoft's CPU rasterizer). Run the same way, 512×512 with the same flags, our Metal frame can be compared pixel for pixel against an image made by a different team on a different implementation.

Most tutorials land within 13 of 255 levels. The exceptions were informative:

• Minified, rotated textures differ by up to 83 levels in a few hundred pixels, because mip and anisotropic filtering are implementation-defined.
• Tutorial29 (order-independent transparency) differs from WARP in 349 pixels. But WARP itself differs from Diligent's Vulkan, OpenGL and WebGPU images in 136 edge pixels, and our frame matches those three. When the reference disagrees with everything else, check the other references before blaming the layer.

A review pointed out that my first allowances (a flat 64 to 768 bad pixels) were not tied to measurements. Every app gave identical output over two runs, so each allowance is now the measured count plus 16 and a tenth.

Numbers

• Full run: 682 pass, 27 fail, 137 skip of 846 tests, against 659 / 27 / 138 of 824 at the end of entry 031. None newly failing; 23 newly passing: 21 Diligent apps, api/logic_op and vkd3d's test_output_merger_logic_op.
• Pixels more than 16 levels from WARP's image (of 262,144): Tutorial29 349, Tutorial05 303, Tutorial06 303, Tutorial08 228, Tutorial04 191, Tutorial09 1, the other 15 apps 0 (two identical runs).
• NuklearDemo: at most 1 level from WARP anywhere.

Not covered, with the reasons: three programs (tutorials 16 and 23, and the GLTFViewer sample) use HLSL that Wine's shader compiler rejects, such as reversebits and shader model 5.1 resource arrays. Five need Diligent modules this build leaves out (RenderStateNotation, SuperResolution). Two need ray tracing, one needs variable-rate shading, and two have no golden-image mode.