Inside the HTTP/3 conformance lab

Most conformance tooling drives a client and checks what comes back. This does the opposite. It is a server that is deliberately, precisely wrong — once per port, 59 times over — and the measurement is whatever the client does when it walks into one.

That inversion is the whole design. A client cannot be asked whether it rejects a second control stream; it can only be shown one. Every anomaly below exists because a clause in an RFC says a client must do something about it, and the only way to find out is to produce the situation on the wire and watch.

One anomaly, one port

The listener runs inside the production proxy and binds 59 UDP sockets at 4460 + offset, ending at 4518. Offsets are pinned by a unit test, because the port number is a public identifier: renumbering would silently point somebody's script at a different anomaly and it would still return results.

Three tiers, three places to be wrong

Where an anomaly is injected decides what can be concluded from the answer. A TLS anomaly is a configuration the handshake either survives or does not. A QUIC anomaly is mostly transport configuration and the socket underneath it. An HTTP/3 anomaly is bytes written after the handshake, on a stream chosen for the clause being tested.

What a catalogue entry has to declare

An anomaly is not a function; it is a record, and the record is what makes the verdict checkable. Nothing in the suite may score a cell on a clause the entry did not name in advance.

Five classes, because a pass does not always mean the same thing

A test whose clause says MAY can never produce a fail. That is enforced in the reduction, not left to whoever writes the entry.

The socket underneath

Several QUIC clauses are about conditions no configuration can express: a datagram that never arrives, two that arrive out of order, a path that marks congestion. Those live in a wrapper around the UDP socket which implements poll_recv and poll_send and is free to lie about both.

Why the stack is forked

Three impairments have no reachable code path in a correct implementation, which is exactly why they are worth testing and exactly why they cannot be configured. They live in vendored forks.

From what happened to what it means

The reduction is staged on purpose. Thirteen observations describe what the connection did; five verdicts say what that is worth. Keeping them apart is what lets the suite distinguish a client that objected in the wrong place from one that did not object, and both from a run that never put the client in the situation. Between them sits the record the verdict is derived from, which is kept.

A fail is never inferred from silence. Where a clause names a specific error code, a client that closes the QUIC connection instead has objected — the report says so — but the code the clause names never reached the wire, and the test asks what the client signalled.

And a silence is only read once the client has had its chance to break it. Each test declares a reaction opportunity, which is an event rather than an interval: the exchange completing, the anomaly’s own stream being demonstrably read, early data arriving and being refused in the handshake, or — for the tests whose anomaly is itself an absence — the server still being there a set interval later. A verdict may not depend on an observation that became impossible before the client had the means to make it, so a silence-based failure requires the client to have still been running when its opportunity arrived.

The record a verdict is derived from

A verdict is computed, not written down. What is written down is the evidence: the observation, the error code the clause names, whether the read proof answered, whether early data arrived and how much, and the four instants — when the anomaly was established, when the exchange completed, when the observation ended, when the client process exited. The oracle is a function of that record and reads nothing else.

The record survives the run. Every cell of the published matrix carries the evidence its verdict came from, the version of the scoring rules that produced it, and, where a rule moved the verdict rather than the observation deciding it, the reason code that says which. A live session expires in minutes; the matrix is the copy that is kept, so that is where the evidence has to live.

That one-way arrow is the point. When a verdict differs between two runs, the two records can be compared field by field, and the answer is either that the oracle was given different inputs or that it reached a different conclusion from the same ones. The first is the client or the harness behaving differently on the day; the second is a scoring change, which has to be justified against the clause rather than against the result. Without the record, both look identical — and a suite that cannot tell them apart can quietly fit its rules to its outcomes.

Proving a client read something

Several HTTP/3 clauses are about a stream the client is not obliged to read promptly. If it never reads, it cannot have violated anything, and scoring it would be an accusation the run cannot support. The proof used is flow-control credit on the anomaly’s own stream, because a receiver extends credit only once its application has consumed what is already there.

Attribution, and the ways it goes wrong

A verdict is worth nothing if it belongs to the wrong connection, and on a port driven by 13 clients in sequence that is not automatic. These are the rules the session layer holds to.

Sessions key on SNI plus peer address

A run is a walk through the catalogue, not a single connection. The driver reads the report once, at the end, and that read is the only signal the endpoint gets that a run is over.

A client is not one connection per test

Chromium opens two to four. A late reconnect that learns nothing must never overwrite the connection that established the finding, so no inconclusive verdict can replace a conclusive one — whatever observation produced it.

Counters are per peer, never per port

A port aggregate makes a second client arriving mid-wait look like the first still talking, so every counter is scoped to the address it came from.

An error path must not return the value of “nothing to do”

A socket operation that fails and one that had nothing to do are counted separately and logged separately, so a broken path cannot read as an idle one.

Driving it

The driver is a binary that runs one client command per anomaly, substituting the URL and port. Whatever the command connects with is what gets measured — there is no shim, no library binding, no mock.

h3-conformance \
    --host conformance.pqcrypta.com \
    --client 'curl -s --http3-only --max-time 15 -o /dev/null {url}' \
    --filter q-stateless-reset \
    --json

Exit status is deliberately ignored. A client failing a test frequently should exit non-zero, and treating that as an error would discard the result the run exists to collect. The verdict is read from the server side, which is the only side that knows what was emitted.

Reproducibility is measured, not asserted

The whole matrix is driven repeatedly and compared cell by cell. A cell counts as reproducible only when every repeat returned the same verdict; a cell missing from any repeat counts as a disagreement rather than being dropped, because a client that failed to launch once is precisely the instability being measured.

Every one of those runs measured the build that produced the published matrix, which is how they are selected: a repeat against a different binary is not a repeat. That build has since been replaced on the server, and the figures stay with the matrix they describe until a run of the new build is promoted. The warm-up is discarded on evidence rather than superstition — the first run after a restart returns inconclusive for clients that pass on every later one, so it is measured and dropped.

The fleet

13 implementations, each driven as its own binary against the same ports, version-pinned to a commit where the project has one. Nothing is simulated: whatever the command connects with is what gets measured, and its exit status is ignored, because a client failing a test frequently should exit non-zero.

A run publishes only through the gate

The weekly job cannot write the live matrix. It measures the running build four times — a warm-up that is thrown away, then three scored repeats — and hands the last run to a separate program that publishes it only if every check passes. A run that fails one is not published: the live matrix keeps the last state that passed, and the refusal raises an alert on the monitor.

What this vantage cannot see

Stated rather than worked around, because a limit that is not published reads as a result. Each of these is a place where the honest answer is inconclusive and the tempting one is a verdict.

Every run is addressable

A driven run ends with its own report and badge, keyed by the session the endpoint assigned. Those URLs are noindex, nofollow: the space is unbounded and each one describes a single execution rather than the state of the field.

Full report: https://conformance.pqcrypta.com/report/<session>
Badge:       https://conformance.pqcrypta.com/badge/<session>.svg

The published matrix is the other thing entirely: 13 implementations driven by us, on a schedule, published only when every check passes.

Where to go next

The matrix is every client against every anomaly. The anomaly index is what the server emits and the clause it is judged against. Findings is only the failures. Post-quantum certificates is the one result that deserved an argument rather than a row.