HTTP/3 Client Conformance

What the catalogue covers

What a result means

Running it

Each test has its own UDP port. Connect to it, and the server records what your client did. Results are filed under a session started from your address, so the test connections need nothing special.

A port per test rather than a path per test, because several anomalies are properties of the endpoint rather than of a connection — the set of QUIC versions offered is decided before any connection exists, so a test of it has to own the whole port.

# start a session
SESSION=$(curl -sX POST https://conformance.pqcrypta.com/session | jq -r .id)

# walk the catalogue
curl -s https://conformance.pqcrypta.com/catalog.json \
  | jq -r '.tests[].port' | while read PORT; do
      curl -s --http3-only --max-time 15 \
        "https://conformance.pqcrypta.com:$PORT/" -o /dev/null
    done

# read the result
curl -s https://conformance.pqcrypta.com/report/$SESSION.json | jq .totals

Without jq: grep -o '"port":[0-9]*' | cut -d: -f2 pulls the ports out of the catalogue, and the session id is the first "id" in the /session response.

The report is also available as a page at /report/<session> and as an SVG badge at /badge/<session>.svg.

In CI

h3-conformance does the above and exits non-zero if anything failed. It is a harness rather than a client: it runs a command you supply, once per test, so whatever your library connects with is what gets measured.

# drive curl
h3-conformance

# drive your own client
h3-conformance --client 'my-client --url {url}'

# one area, machine-readable
h3-conformance --filter h-qpack --json

Exit 0 clean, 1 a test failed, 2 the suite was unreachable — distinct so an outage never reads as a regression in your client. Inconclusive results never fail the run.

The report

A session collects results as you walk the ports, and is readable three ways: as a page at /report/<session>, as JSON at /report/<session>.json, and as an SVG badge at /badge/<session>.svg. All three are public and CORS-open, so CI can fetch them without a key.

{
  "session": "8fbecf53c82f7c6481ecca51727c6742",
  "catalog_size": 51,
  "totals":   { "pass": 22, "fail": 0, "inconclusive": 5, "not_run": 0 },
  "by_class": { "correctness": { "pass": 6, "fail": 0, "inconclusive": 3 }, ... },
  "results": [
    { "id": "h-grease-frame", "class": "extensibility", "tier": "http3",
      "spec": "RFC 9114 §7.2.8", "verdict": "pass",
      "detail": "Ignored the unrecognised element and completed the follow-up request." }
  ]
}

Test ids are stable and appear in the URL, the JSON and the badge, so they are safe to pin a CI assertion to.

Embedding the badge

![HTTP/3 conformance](https://conformance.pqcrypta.com/badge/$SESSION.svg)

A badge is a picture of one run, not a live status: re-run the suite and use the new session to refresh it.

How results find their session

By default results are filed under the session started from your address, which is why the walk above needs nothing special. If your tests run from several machines, or from behind a shared address, connect with the server name <session>.conformance.pqcrypta.com instead — the session travels in the SNI, and results follow it rather than the address.

A session expires an hour after it was created, not an hour after its last result — activity does not extend it. Start the session at the beginning of the walk rather than long before, and fetch the report while the run is fresh; nothing is archived.

How a verdict is reached

The server is the judge. A library that crashes on an unknown frame is in no position to report on itself, so every verdict comes from what the server observed.

Every test ends with a liveness probe: after the anomaly, the server expects one ordinary request on the same connection. For the extensibility tests this is the whole game — "ignored it" and "died on it" look identical on the wire up to that point, and without the probe a crashed client would read as compliant.

A failure needs positive evidence. It is tempting to read “no rejection arrived” as “the client accepted it”, and that inference is unsound. Where the anomaly rides in the response your client had to read to be served, a completed exchange does show it was seen. Where it went to the control stream — unidirectional, and read on no particular schedule — a one-shot request can finish before the stream is picked up at all, which looks identical to accepting the violation. The same is true of a rejection whose CONNECTION_CLOSE is lost in transit, since QUIC only repeats that frame in answer to an incoming packet. Neither case is reported as a failure.

The classes exist because the required answer is not the same everywhere, and sometimes it is the opposite. Ignoring a reserved HTTP/3 frame is a pass — reserved frame types exist precisely to be ignored. Ignoring a reserved QUIC frame is a failure, because QUIC reserves no ignorable frame types and an unknown frame carries no length to skip it by. Two tests that look alike on the wire require opposite behaviour, so a single pass/fail rule would score one of them backwards.

The catalogue

Read live from the suite, so this table cannot drift from what is actually served. Every entry cites the clause it exercises.

HTTP/3 layer

QUIC layer

When a run proves nothing

Every port in the catalogue emits its anomaly—51 of 51. Nothing here is a placeholder.

Whether your run reaches a given anomaly is a separate question, and some depend on what your client happens to do. A few examples, all of which report inconclusive rather than guessing:

• 0-RTT rejection needs early data to reject, which needs a session ticket from an earlier connection to the same port. A client that connects once has none.
• ECN reporting is required only where the ECN field is accessible to the peer, and a network can strip the codepoint in transit. Silence there is not attributable to your client.
• QPACK dynamic table references need the client to have granted the encoder some table capacity. Advertise zero, and there is nothing the server is permitted to reference.
• Connection ID rotation and path validation need a connection that lives long enough, or moves. A single short request does neither.
• An anomaly on the control stream that your client closed without objecting to. The control stream is unidirectional and nothing obliges a client to read it on any schedule, so a one-shot request can finish before it is picked up — which looks exactly like accepting the violation. The server holds the response back for a moment first, to give a conformant reader its chance, but where the two cannot be told apart the result is inconclusive rather than a failure. A suite that reports “accepted a protocol violation” on the strength of silence is guessing.

An inconclusive result says which of these applied. It is never counted as a pass and never as a failure, and it does not fail a CI run.

If your client scores a failure you believe is wrong, say so — a suite that accuses the thing it tests is worth fixing quickly.

Scope, and what this is not

Fifty-one tests is a beginning, not coverage. QUIC and HTTP/3 together are enormous, and the catalogue is chosen rather than exhaustive. What is missing divides into two quite different things, and collapsing them into one list of “not done yet” would overstate what this suite can ever become.

Testable, and not yet written

The backlog is shorter than it was: ECN congestion marking, repeated key updates and mid-connection MAX_STREAMS credit have since been built. What is left is a Retry packet a client ought to discard — RFC 9000 §17.2.5.2 has a client ignore any Retry after the first — and a handful of similar single-clause cases.

Compatible version negotiation is a case of its own, and no longer counted as backlog. RFC 9000 §6.2 does not define what a client does when the offered version is one it supports; it says only that “how to perform version negotiation is left as future work”. RFC 9368 supplies that work, so the test becomes possible once the stack implements version_information — and not before. Writing it now would mean measuring behaviour no clause requires, which is how a conformance suite starts inventing requirements.

Outside what this vantage point can establish

Not a backlog, and listing it as one would be misleading. These are behaviours a server watching a client cannot legitimately provoke or observe, so a test claiming to cover them would be reporting on something other than what it says.

• QPACK decoder-stream errors. That stream is the client's output. The error exists for an encoder objecting to a decoder's instructions, and nothing here can make a client emit a malformed one.
• WebTransport session handling beyond the setting that gates it. Extended CONNECT is initiated by the client; a generic HTTP/3 client never opens a session, so the port would report that nothing was exercised in perpetuity.
• A client's own migration after a NAT rebind. The server can send a PATH_CHALLENGE — and does — but only the client, or its network, can change the address it sends from.
• The amplification limit. A requirement on servers, so there is nothing about it to ask of the thing under test.

• This is not a certification. Passing every test here means a client handled fifty-one specific situations correctly. It does not mean the implementation is conformant, and nothing on this site should be quoted as though it did.
• A failure is a specification violation, not a security vulnerability. Some may have security relevance; most are simply behaviour a specification prohibits. Calling every failure a vulnerability would make the whole dataset less credible, and would be unfair to the implementations measured.
• It tests clients, not servers. For the other direction — what a server advertises and actually does — the HTTP/3 scanner is the tool.
• The suite is itself an implementation, and so is subject to the same doubt as anything it measures. A verdict you believe is wrong is worth reporting: a suite that accuses the thing it tests is worse than no suite, because the real failures stop being believed.