MOOR security

Dieser Eintrag ist auf Englisch verfasst.

Reading past the end of a vertex buffer, the Direct3D way

Entry 069858 of 958 tests pass

Direct3D 12 promises zeros when a draw reads past the end of a vertex buffer and Metal promises nothing, so mtl12 now has each vertex shader check its own reads.

What changed

Two gaps in plain drawing closed this session.

First, the four "adjacency" topologies (primitive types that send each line or triangle together with its neighbours' vertices, for geometry shaders) now draw when there is no geometry shader. Direct3D 12 then ignores the neighbours. mtl12 picks out the real vertices in the same GPU pre-pass that already rewrites index buffers for strip cuts and triangle fans.

Second, vertex fetch bounds. A vertex buffer view in Direct3D 12 is an address, a stride and a size. When a draw reads an element that ends past the size, the spec says the shader gets 0 in the format's components and the default (0, 0, 0, 1) in the rest (D3D11.3 functional spec 8.19.2). Metal 4 binds a vertex buffer as an address and a stride only, with no length. A read past the end returns whatever memory follows.

mtl12 now rewrites the vertex function that SPIRV-Cross generates. The inputs Metal fetched are copied first. Then each one is checked: its index × stride + offset + format size must fit in the view, or it is replaced by the defaults. The index is the vertex id, or for per-instance data the start instance plus the instances since it divided by the step rate. The views travel to the shader as a small table, bound at a buffer slot that would otherwise belong to input slot 19. Layouts that use slot 19 are left unchecked. Indirect draws bind the table from their own records.

How the view sizes and the input layout meet in the vertex function's check.D3D12: IASetVertexBuffers address, SizeInBytes, stride per slot → draw state: the views (va, size, stride) x 20; CreateGraphicsPipelineState input layout: slot, offset, format → SPIRV-Cross MSL vertex function; SPIRV-Cross MSL vertex function → guard_vertex_fetches: stage_in copied, each input checked; draw state: the views (va, size, stride) x 20 → Metal vertex function (upload, bind at buffer 28 (ExecuteIndirect: the record's own)); guard_vertex_fetches: stage_in copied, each input checked → Metal vertex function; Metal vertex function → index x stride + offset + size ≤ view size?; index x stride + offset + size ≤ view size? → Metal's fetch (yes); index x stride + offset + size ≤ view size? → 0 in the format's components, the rest of (0, 0, 0, 1) (no)D3D12: IASetVertexBuffersaddress, SizeInBytes, stride per slotdraw state: the views(va, size, stride) x 20CreateGraphicsPipelineStateinput layout: slot, offset, formatSPIRV-Cross MSL vertex functionguard_vertex_fetches:stage_in copied, each input checkedMetal vertex functionindex x stride + offset + size≤ view size?Metal's fetch0 in the format's components,the rest of (0, 0, 0, 1)upload, bind at buffer 28(ExecuteIndirect: the record's own)yesno
How the view sizes and the input layout meet in the vertex function's check. (Mermaid source)
Gallery scene vertexbounds: a grid of 40 x 22 vertices coloured from a vertex buffer, read through a view that covers only part of it; vertices past the view's end read (0, 0, 0, 1), shown as flat grey, and the triangles across the boundary blend into it. All scenes in the gallery

The interesting part

I break each check on purpose once and watch the test fail. Six such breaks failed the new test straight away. A seventh did not: after a checked pipeline has put its table where slot 19's buffer goes, the next pipeline must rebind slot 19, and skipping that rebind passed every check. Nothing in the test read slot 19. A second pipeline that does fails 40 checks with the break. A cache that forgets to invalidate is a missing guard too.

The cost was the other worry. Earlier measurements put a per-draw CPU price on vertex functions that read the instance builtins. A stash-and-rebuild A/B of bgfx's drawstress example showed none: 8.74 ms of layer CPU per frame without the checks and 8.62 ms with them.

Numbers

  • Full run: 858 pass, 27 fail, 73 skip of 958, up from 856. The only per-test changes are the two new tests passing.
  • api/vertex_bounds: 1793 checks pass. Without the checks, 632 of its first 1281 failed. Nine deliberate breaks each fail it.
  • api/adjacency: 1629 checks pass, and 812 of its first 1309 failed before the fix.
Tests by suite at the end of this entry: 858 of 958 pass (data as CSV)
  • pass
  • fail
  • skip

suite · share of each row's total

  1. vkd3d-proton494 pass, 14 fail, 69 skip of 577
  2. vkd3d177 pass, 9 fail, 4 skip of 190
  3. api (+2)87 pass, 0 fail, 0 skip of 87
  4. apps80 pass, 0 fail, 0 skip of 80
  5. wine-dxgi16 pass, 4 fail, 0 skip of 20
  6. samples4 pass, 0 fail, 0 skip of 4