HTTP/3 Client Conformance

What the catalogue covers

What a result means

Running it

Each test has its own UDP port. Connect to it, and the server records what your client did. Results are filed under a session started from your address, so the test connections need nothing special.

A port per test rather than a path per test, because several anomalies are properties of the endpoint rather than of a connection — the set of QUIC versions offered is decided before any connection exists, so a test of it has to own the whole port.

# start a session
SESSION=$(curl -sX POST https://conformance.pqcrypta.com/session | jq -r .id)

# walk the catalogue
curl -s https://conformance.pqcrypta.com/catalog.json \
  | jq -r '.tests[].port' | while read PORT; do
      curl -s --http3-only --max-time 15 \
        "https://conformance.pqcrypta.com:$PORT/" -o /dev/null
    done

# read the result
curl -s https://conformance.pqcrypta.com/report/$SESSION.json | jq .totals

Without jq: grep -o '"port":[0-9]*' | cut -d: -f2 pulls the ports out of the catalogue, and the session id is the first "id" in the /session response.

The report is also available as a page at /report/<session> and as an SVG badge at /badge/<session>.svg.

In CI

h3-conformance does the above and exits non-zero if anything failed. It is a harness rather than a client: it runs a command you supply, once per test, so whatever your library connects with is what gets measured.

# drive curl
h3-conformance

# drive your own client
h3-conformance --client 'my-client --url {url}'

# one area, machine-readable
h3-conformance --filter h-qpack --json

Exit 0 clean, 1 a test failed, 2 the suite was unreachable — distinct so an outage never reads as a regression in your client. Inconclusive results never fail the run.

The report

A session collects results as you walk the ports, and is readable three ways: as a page at /report/<session>, as JSON at /report/<session>.json, and as an SVG badge at /badge/<session>.svg. All three are public and CORS-open, so CI can fetch them without a key.

{
  "session": "8fbecf53c82f7c6481ecca51727c6742",
  "catalog_size": 59,
  "totals":   { "pass": 22, "fail": 0, "inconclusive": 5, "not_run": 0 },
  "by_class": { "correctness": { "pass": 6, "fail": 0, "inconclusive": 3 }, ... },
  "results": [
    { "id": "h-grease-frame", "class": "extensibility", "tier": "http3",
      "spec": "RFC 9114 §7.2.8", "verdict": "pass",
      "detail": "Ignored the unrecognised element and completed the follow-up request." }
  ]
}

Test ids are stable and appear in the URL, the JSON and the badge, so they are safe to pin a CI assertion to.

Embedding the badge

![HTTP/3 conformance](https://conformance.pqcrypta.com/badge/$SESSION.svg)

A badge is a picture of one run, not a live status: re-run the suite and use the new session to refresh it.

How results find their session

By default results are filed under the session started from your address, which is why the walk above needs nothing special. If your tests run from several machines, or from behind a shared address, connect with the server name <session>.conformance.pqcrypta.com instead — the session travels in the SNI, and results follow it rather than the address.

A session expires an hour after it was created, not an hour after its last result — activity does not extend it. Start the session at the beginning of the walk rather than long before, and fetch the report while the run is fresh; nothing is archived.

How a verdict is reached

The server is the judge. A library that crashes on an unknown frame is in no position to report on itself, so every verdict comes from what the server observed.

Every test ends with a liveness probe: after the anomaly, the server expects one ordinary request on the same connection. For the extensibility tests this is the whole game — "ignored it" and "died on it" look identical on the wire up to that point, and without the probe a crashed client would read as compliant.

A failure needs positive evidence. It is tempting to read “no rejection arrived” as “the client accepted it”, and that inference is unsound. Where the anomaly rides in the response your client had to read to be served, a completed exchange does show it was seen. Where it went to the control stream — unidirectional, and read on no particular schedule — a one-shot request can finish before the stream is picked up at all, which looks identical to accepting the violation. The same is true of a rejection whose CONNECTION_CLOSE is lost in transit, since QUIC only repeats that frame in answer to an incoming packet. Neither case is reported as a failure.

That last case is asked about rather than assumed. RFC 9000 §10.2.1 has a closing endpoint re-send its CONNECTION_CLOSE in answer to an incoming packet — and only then — so where a connection ended with no reason recorded the server sends a PING and waits. A client that did object answers with the close it already sent; a client that never objected stays quiet, and now the silence is evidence instead of an absence of it.

In the published run that probe was needed 136 times. It drew a close 36 times and a transport acknowledgement 53 times, so the question demonstrably arrived in 65% of them.

Both numbers are needed to read either. A peer that acknowledges the PING and sends no close is alive, which is what makes “no close observed” a measurement rather than an assumption. A probe that drew neither would have reached nobody, and the same silence would mean this instrument is broken — the identical mistake the probe was added to stop making, one level up. So the suite counts its own question as well as the answer.

The distinction is carried down to the individual result, not just kept in this total. A correctness cell whose peer acknowledged the probe says so, and says the client was still there and had not rejected the anomaly; one whose peer answered nothing says the weaker thing instead, that a client which exited without closing looks from here exactly like one that read the anomaly and carried on. The two were a single sentence until 2026-09-19, and it made the stronger claim in both.

The classes exist because the required answer is not the same everywhere, and sometimes it is the opposite. Ignoring a reserved HTTP/3 frame is a pass — reserved frame types exist precisely to be ignored. Ignoring a reserved QUIC frame is a failure, because QUIC reserves no ignorable frame types and an unknown frame carries no length to skip it by. Two tests that look alike on the wire require opposite behaviour, so a single pass/fail rule would score one of them backwards.

The catalogue

Read live from the suite, so this table cannot drift from what is actually served. Every entry cites the clause it exercises.

HTTP/3 layer

QUIC layer

TLS layer

When a run proves nothing

Every port in the catalogue emits its anomaly—59 of 59. Nothing here is a placeholder.

Whether your run reaches a given anomaly is a separate question, and some depend on what your client happens to do. A few examples, all of which report inconclusive rather than guessing:

• 0-RTT rejection needs early data to reject, which needs a session ticket from an earlier connection to the same port. A client that connects once has none.
• ECN reporting is required only where the ECN field is accessible to the peer, and a network can strip the codepoint in transit. Silence there is not attributable to your client.
• QPACK dynamic table references need the client to have granted the encoder some table capacity. Advertise zero, and there is nothing the server is permitted to reference.
• Connection ID rotation and path validation need a connection that lives long enough, or moves. A single short request does neither.
• An anomaly on a unidirectional stream — the control stream, the QPACK encoder stream, a push stream — that your client closed without objecting to. Nothing obliges a client to read those on any schedule, so a one-shot request can finish before one is picked up, which looks exactly like accepting the violation. So the server asks: it holds the response back for a moment, keeps writing on that stream, and watches for flow-control credit on it. Credit means the client consumed the stream, and carrying on after that is a failure. A window filled with no credit back, while the client stayed connected, means it never read the stream. Where neither can happen — the client left first — the result is inconclusive rather than a failure. A suite that reports “accepted a protocol violation” on the strength of silence is guessing.

An inconclusive result says which of these applied. It is never counted as a pass and never as a failure, and it does not fail a CI run.

If your client scores a failure you believe is wrong, say so — a suite that accuses the thing it tests is worth fixing quickly.

How much of this moves when nothing does

A verdict that changes between two runs of the same binaries is not a verdict, and a suite that never measures that is asking to be taken on faith. 3 consecutive full runs were kept for this — same server build, same client binaries, same catalogue, nothing changed between them — and every cell was compared across all 3.

Every one of the 708 cells held one verdict across all 3 runs. Getting there took closing a series of races in the harness itself — a count taken per address that two connections could share, a fact recorded only when a connection's handler finished, a probe that could lose to the client's own acknowledgement — each of which had shown up as one cell reading differently from one run to the next.

This is measured, not asserted, and it is recomputed from the kept runs every time this page is served. If the runs are removed the section disappears rather than quoting a number nobody can check.

Scope, and what this is not

59 tests is a beginning, not coverage. QUIC and HTTP/3 together are enormous, and the catalogue is chosen rather than exhaustive. What is missing divides into two quite different things, and collapsing them into one list of “not done yet” would overstate what this suite can ever become.

Testable, and not yet written

The backlog is shorter than it was: ECN congestion marking, repeated key updates and mid-connection MAX_STREAMS credit have since been built. What is left is a Retry packet a client ought to discard — RFC 9000 §17.2.5.2 has a client ignore any Retry after the first — and a handful of similar single-clause cases.

Compatible version negotiation is a case of its own, and no longer counted as backlog. RFC 9000 §6.2 does not define what a client does when the offered version is one it supports; it says only that “how to perform version negotiation is left as future work”. RFC 9368 supplies that work, so the test becomes possible once the stack implements version_information — and not before. Writing it now would mean measuring behaviour no clause requires, which is how a conformance suite starts inventing requirements.

Outside what this vantage point can establish

Not a backlog, and listing it as one would be misleading. These are behaviours a server watching a client cannot legitimately provoke or observe, so a test claiming to cover them would be reporting on something other than what it says.

• Whether a client read a unidirectional stream, when it left before that could be answered. The one signal that separates read from received is flow-control credit on that stream, and the server provokes it by writing until the client must either grant more or stop reading: a grant proves the stream was consumed, and a window filled with none, while the client stays connected, proves it was not. A client that closes before its window can be filled, or before the wait ends, leaves the distinction unmade — and the cell says so rather than guessing either way.
• QPACK decoder-stream errors. That stream is the client's output. The error exists for an encoder objecting to a decoder's instructions, and nothing here can make a client emit a malformed one.
• WebTransport session handling beyond the setting that gates it. Extended CONNECT is initiated by the client; a generic HTTP/3 client never opens a session, so the port would report that nothing was exercised in perpetuity.
• A client's own migration after a NAT rebind. The server can send a PATH_CHALLENGE — and does — but only the client, or its network, can change the address it sends from.
• The amplification limit. A requirement on servers, so there is nothing about it to ask of the thing under test.

• This is not a certification. Passing every test here means a client handled 59 specific situations correctly. It does not mean the implementation is conformant, and nothing on this site should be quoted as though it did.
• A failure is a specification violation, not a security vulnerability. Some may have security relevance; most are simply behaviour a specification prohibits. Calling every failure a vulnerability would make the whole dataset less credible, and would be unfair to the implementations measured.
• It tests clients, not servers. For the other direction — what a server advertises and actually does — the HTTP/3 scanner is the tool.
• The suite is itself an implementation, and so is subject to the same doubt as anything it measures. A verdict you believe is wrong is worth reporting: a suite that accuses the thing it tests is worse than no suite, because the real failures stop being believed.