HTTP/3 Client Conformance

Running it

Each test has its own UDP port. Connect to it, and the server records what your client did. Results are filed under a session started from your address, so the test connections need nothing special.

# start a session
SESSION=$(curl -sX POST https://conformance.pqcrypta.com/session \
  | sed 's/.*"id":"//;s/".*//')

# walk the catalogue
curl -s https://conformance.pqcrypta.com/catalog.json \
  | grep -o '"port": [0-9]*' | grep -o '[0-9]*' | while read PORT; do
      curl -s --http3-only --max-time 15 \
        "https://conformance.pqcrypta.com:$PORT/" -o /dev/null
    done

# read the result
curl -s https://conformance.pqcrypta.com/report/$SESSION.json

The report is also available as a page at /report/<session> and as an SVG badge at /badge/<session>.svg.

How a verdict is reached

The server is the judge. A library that crashes on an unknown frame is in no position to report on itself, so every verdict comes from what the server observed.

Every test ends with a liveness probe: after the anomaly, the server expects one ordinary request on the same connection. For the extensibility tests this is the whole game — "ignored it" and "died on it" look identical on the wire up to that point, and without the probe a crashed client would read as compliant.

Those last two exist because the model was wrong twice. Scoring a MAY as a MUST failed clients for making a legal choice, and scoring "the request never reached the window" as a failure accused clients of accepting violations that were never sent. A suite that wrongly accuses the thing it tests is worse than no suite, because nobody then believes the failures that are real.

The catalogue

Read live from the suite, so this table cannot drift from what is actually served. Every entry cites the clause it exercises.

HTTP/3 layer

QUIC layer

What this cost to get right

Building it turned up more bugs in the suite than in any client. Every failure it reported during development was ours, and each would have produced a false verdict:

• Any stream counted as the liveness probe — but every client opens its control stream before reading a byte of our SETTINGS, so every extensibility test passed the moment a client connected.
• The critical streams were held in the wrong scope, so dropping them closed the control stream — our violation, scored against the client.
• Verdicts were decided at the probe, before the client had read the anomaly, so a correct rejection was recorded as acceptance.
• A hand-written Huffman table produced the wrong code for several characters. It looked entirely plausible; only the RFC 7541 test vectors caught it.
• The QPACK dynamic table was used without checking whether the client had granted any capacity. The default is zero.

Every expected_code in the catalogue has since been checked against the published RFC text rather than recollection, and three tests are pinned to the sentence that requires them so a later edit cannot drift.

Status

This is new and openly incomplete. Tests whose anomaly is not yet implemented serve a correct control stream and record inconclusive — never a pass, and never a failure for something that was not sent.

If your client scores a failure you believe is wrong, say so. Given the above, the suite is the more likely culprit, and a false accusation is a bug worth fixing quickly.