Inside the HTTP/3 conformance lab

Most conformance tooling drives a client and checks what comes back. This does the opposite. It is a server that is deliberately, precisely wrong — once per port, 59 times over — and the measurement is whatever the client does when it walks into one.

That inversion is the whole design. A client cannot be asked whether it rejects a second control stream; it can only be shown one. Every anomaly below exists because a clause in an RFC says a client must do something about it, and the only way to find out is to produce the situation on the wire and watch.

One anomaly, one port

The listener runs inside the production proxy and binds 59 UDP sockets at 4460 + offset, ending at 4518. Offsets are pinned by a unit test, because the port number is a public identifier: renumbering would silently point somebody's script at a different anomaly and it would still return results.

Three tiers, three places to be wrong

Where an anomaly is injected decides what can be concluded from the answer. A TLS anomaly is a configuration the handshake either survives or does not. A QUIC anomaly is mostly transport configuration and the socket underneath it. An HTTP/3 anomaly is bytes written after the handshake, on a stream chosen for the clause being tested.

What a catalogue entry has to declare

An anomaly is not a function; it is a record, and the record is what makes the verdict checkable. Nothing in the suite may score a cell on a clause the entry did not name in advance.

Five classes, because a pass does not always mean the same thing

A test whose clause says MAY can never produce a fail. That is enforced in the reduction, not left to whoever writes the entry.

The socket underneath

Several QUIC clauses are about conditions no configuration can express: a datagram that never arrives, two that arrive out of order, a path that marks congestion. Those live in a wrapper around the UDP socket which implements poll_recv and poll_send and is free to lie about both.

Why the stack is forked

Three impairments have no reachable code path in a correct implementation, which is exactly why they are worth testing and exactly why they cannot be configured. They live in vendored forks.

From what happened to what it means

The reduction is two-stage on purpose. Thirteen observations describe what the connection did; five verdicts say what that is worth. Keeping them apart is what lets the suite distinguish a client that objected in the wrong place from one that did not object, and both from a run that never put the client in the situation.

A fail is never inferred from silence. Where a clause names a specific error code, a client that closes the QUIC connection instead has objected — the report says so — but the code the clause names never reached the wire, and the test asks what the client signalled.

Proving a client read something

Several HTTP/3 clauses are about a stream the client is not obliged to read promptly. If it never reads, it cannot have violated anything, and scoring it would be an accusation the run cannot support. The proof used is flow-control credit on the anomaly’s own stream, because a receiver extends credit only once its application has consumed what is already there.

Attribution, and the ways it goes wrong

A verdict is worth nothing if it belongs to the wrong connection, and on a port driven by twelve clients in sequence that is not automatic. These are the rules the session layer holds to.

Sessions key on SNI plus peer address

A run is a walk through the catalogue, not a single connection. The driver reads the report once, at the end, and that read is the only signal the endpoint gets that a run is over.

A client is not one connection per test

Chromium opens two to four. A late reconnect that learns nothing must never overwrite the connection that established the finding, so no inconclusive verdict can replace a conclusive one — whatever observation produced it.

Counters are per peer, never per port

A port aggregate makes a second client arriving mid-wait look like the first still talking, so every counter is scoped to the address it came from.

An error path must not return the value of “nothing to do”

A socket operation that fails and one that had nothing to do are counted separately and logged separately, so a broken path cannot read as an idle one.

Driving it

The driver is a binary that runs one client command per anomaly, substituting the URL and port. Whatever the command connects with is what gets measured — there is no shim, no library binding, no mock.

h3-conformance \
    --host conformance.pqcrypta.com \
    --client 'curl -s --http3-only --max-time 15 -o /dev/null {url}' \
    --filter q-stateless-reset \
    --json

Exit status is deliberately ignored. A client failing a test frequently should exit non-zero, and treating that as an error would discard the result the run exists to collect. The verdict is read from the server side, which is the only side that knows what was emitted.

Reproducibility is measured, not asserted

The whole matrix is driven repeatedly and compared cell by cell. A cell counts as reproducible only when every repeat returned the same verdict; a cell missing from any repeat counts as a disagreement rather than being dropped, because a client that failed to launch once is precisely the instability being measured.

The warm-up is discarded on evidence rather than superstition: the first run after a restart returns inconclusive for clients that pass on every later run, so it is measured and dropped.

The fleet

12 implementations, each driven as its own binary against the same ports, version-pinned to a commit where the project has one. Nothing is simulated: whatever the command connects with is what gets measured, and its exit status is ignored, because a client failing a test frequently should exit non-zero.

A run does not publish itself

The weekly job stages; it cannot write the live matrix. Promotion is a separate program, and it asks, because a run finishing is not a decision to publish it.

What this vantage cannot see

Stated rather than worked around, because a limit that is not published reads as a result. Each of these is a place where the honest answer is inconclusive and the tempting one is a verdict.

Every run is addressable

A driven run ends with its own report and badge, keyed by the session the endpoint assigned. Those URLs are noindex, nofollow: the space is unbounded and each one describes a single execution rather than the state of the field.

Full report: https://conformance.pqcrypta.com/report/<session>
Badge:       https://conformance.pqcrypta.com/badge/<session>.svg

The published matrix is the other thing entirely: 12 implementations driven by us, on a schedule, promoted by hand.

Where to go next

The matrix is every client against every anomaly. The anomaly index is what the server emits and the clause it is judged against. Findings is only the failures. Post-quantum certificates is the one result that deserved an argument rather than a row.