Every result here was produced by pointing a real client binary at the adversarial conformance suite and recording what the server observed. Nothing is hand-entered, and no client is asked to report on itself.
This is not a ranking. Several tests are discretionary — the specification permits more than one answer. The interesting cells are the ones where implementations disagree.
The implementations
| Client | Stack | Language | Version | Result |
|---|---|---|---|---|
ngtcp2 |
ngtcp2 + nghttp3 (direct) | C | 3c23148 | 55 pass 4 fail 0 n/a |
chromium |
Chromium QUICHE | C++ | build 1248 | 54 pass 4 fail 1 n/a |
quico |
quico + lemon-tls | JavaScript | 0.7.0 (npm) | 57 pass 0 fail 2 n/a |
neqo |
neqo (Firefox) | Rust | 0.31.1 (cac2b74) | 54 pass 3 fail 2 n/a |
picoquic |
picoquic + picotls | C | 8162550 | 42 pass 14 fail 3 n/a |
quic-go |
quic-go | Go | quic-go v0.61.0 (h1:9So2anK4Tp22) | 47 pass 7 fail 5 n/a |
quiche |
Cloudflare quiche | Rust | 0.30.0 (1c8e976) | 46 pass 6 fail 7 n/a |
xquic |
Alibaba xquic + BoringSSL | C | 2060a44 | 45 pass 7 fail 7 n/a |
lsquic |
LiteSpeed lsquic + BoringSSL | C | 4.9.4 (2b30189) | 45 pass 6 fail 7 n/a |
aioquic |
aioquic | Python | 1.3.0 (PyPI wheel) | 44 pass 6 fail 9 n/a |
msquic |
.NET HttpClient + msquic | C# | 2.7.0 (7e6525d) | 34 pass 14 fail 11 n/a |
curl |
ngtcp2 + nghttp3 | C | libcurl/8.18.0 ngtcp2/1.11.0 | 31 pass 14 fail 14 n/a |
quinnself-test |
quinn + h3 (our fork) | Rust | noq fork (e88b1bf) | 59 pass 0 fail 0 n/a |
Per-test results
Rows where the implementations disagree are marked ◆. Hover a cell for what the server actually observed.
Verdict and implementation work together: choosing a verdict alone shows every test where any client scored it, and adding an implementation narrows that to one column — “every failure in Chromium”, or “every RFC 9114 correctness test where implementations disagree”. Column headings sort.
q-version-negotiation
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-retry
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Not run | Pass | Pass | Pass |
q-reserved-transport-param
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-reserved-frame
◆
|
Pass | Pass | Pass | Pass | Pass | Pass | Fail | Fail | Pass | Pass | Fail | Pass | Pass |
q-cid-rotation
◆
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Pass |
q-stateless-reset
◆
|
Pass | Pass | Fail | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Pass | Pass |
q-flow-control
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Pass | Pass | Pass |
q-ack-frequency
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-ecn
|
Unsupported | Pass | Unsupported | Pass | Pass | Unsupported | Unsupported | Pass | Pass | Unsupported | Unsupported | Pass | Unsupported |
q-pmtu-blackhole
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Pass | Pass |
q-path-challenge
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-zero-rtt-reject
◆
|
Unsupported | Pass | Pass | Pass | Fail | Unsupported | Pass | Pass | Fail | Pass | Pass | Unsupported | Pass |
q-multipath
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-key-update
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-stream-limit
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-loss-recovery
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-connection-migration
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-invalid-transport-param
◆
|
Pass | Pass | Pass | Fail | Fail | Pass | Fail | Pass | Pass | Fail | Pass | Fail | Pass |
q-zero-rtt-replay
|
Unsupported | Pass | Pass | Pass | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Unsupported | Pass |
q-ecn-congestion
|
Unsupported | Pass | Unsupported | Pass | Pass | Unsupported | Unsupported | Pass | Pass | Unsupported | Unsupported | Pass | Unsupported |
q-key-update-repeated
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
q-max-streams-credit
|
Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-grease-settings
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-grease-frame
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-reserved-uni-stream
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Pass | Pass | Pass |
h-duplicate-setting
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-control-frame-unexpected
◆
|
Fail | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Pass | Pass | Fail | Pass |
h-missing-settings
◆
|
Fail | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Pass | Pass | Fail | Pass |
h-second-control-stream
◆
|
Fail | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Unsupported | Pass | Fail | Pass |
h-qpack-dynamic-table
|
Unsupported | Pass | Pass | Pass | Unsupported | Unsupported | Pass | Unsupported | Pass | Pass | Pass | Unsupported | Pass |
h-qpack-huffman
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-oversized-field-section
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-trailers
◆
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Pass | Pass |
h-early-hints
◆
|
Pass | Pass | Fail | Pass | Pass | Pass | Pass | Fail | Pass | Pass | Pass | Pass | Pass |
h-goaway
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-max-push-id
◆
|
Fail | Pass | Pass | Pass | Fail | Pass | Pass | Fail | Pass | Pass | Pass | Fail | Pass |
h-settings-on-request-stream
◆
|
Fail | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-data-before-headers
◆
|
Fail | Pass | Pass | Pass | Pass | Fail | Pass | Pass | Pass | Pass | Pass | Fail | Pass |
h-cancel-push-unsolicited
◆
|
Fail | Pass | Fail | Fail | Fail | Fail | Pass | Fail | Fail | Pass | Pass | Fail | Pass |
h-qpack-blocked-stream
|
Unsupported | Pass | Pass | Pass | Unsupported | Unsupported | Pass | Unsupported | Pass | Pass | Pass | Unsupported | Pass |
h-response-stream-reset
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-priority-update
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-extended-connect
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-push-promise-unsolicited
◆
|
Fail | Pass | Fail | Fail | Fail | Fail | Pass | Fail | Fail | Pass | Fail | Fail | Pass |
h-datagram-setting-invalid
◆
|
Fail | Pass | Pass | Pass | Fail | Pass | Pass | Fail | Pass | Fail | Fail | Fail | Pass |
h-qpack-encoder-overflow
◆
|
Fail | Pass | Pass | Pass | Unsupported | Fail | Pass | Fail | Pass | Pass | Fail | Fail | Pass |
h-goaway-increasing
◆
|
Fail | Pass | Fail | Pass | Pass | Pass | Fail | Fail | Pass | Pass | Pass | Fail | Pass |
h-push-stream-unpromised
◆
|
Fail | Pass | Fail | Fail | Pass | Fail | Pass | Fail | Fail | Unsupported | Pass | Fail | Pass |
h-qpack-static-index-invalid
◆
|
Fail | Pass | Pass | Pass | Fail | Pass | Pass | Fail | Pass | Pass | Fail | Fail | Pass |
q-packet-reordering
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
h-qpack-encoder-bad-name-index
◆
|
Fail | Pass | Pass | Pass | Unsupported | Fail | Pass | Fail | Pass | Pass | Fail | Fail | Pass |
t-hybrid-only
|
Unsupported | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Unsupported | Pass |
t-classical-only
|
Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass |
t-hybrid-large-hello
|
Unsupported | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Unsupported | Pass |
t-group-not-offered
◆
|
Unsupported | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Unsupported | Pass |
t-corrupt-hybrid-share
|
Unsupported | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Pass |
t-hybrid-share-length
◆
|
Unsupported | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Fail | Pass | Unsupported | Pass |
t-cert-compression-pq
|
Unsupported | Pass | Unsupported | Unsupported | Unsupported | Unsupported | Pass | Unsupported | Pass | Unsupported | Unsupported | Unsupported | Pass |
t-grease-group
|
Unsupported | Pass | Unsupported | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Unsupported | Unsupported | Pass |
Reading this honestly
- A fail is a specification violation, not a security vulnerability. Some may have security relevance; most are simply behaviour a specification prohibits. Treating every failure as a vulnerability would make the whole dataset less credible.
- Every cell here is conclusive. Each client was put in the situation every test is about, and its answer was seen: a pass, a fail, or unsupported where the client does not do the thing the test needs. The suite can still return inconclusive to anyone running it — a client that connects once cannot be asked about 0-RTT — but none of this matrix's cells carry it.
-
Rejecting at the QUIC layer is not a pass either.
Some clients answer an HTTP/3 violation by closing the QUIC connection
rather than sending an HTTP/3 error code —
H3_MISSING_SETTINGSarrives as a transportINTERNAL_ERROR. The client plainly objected, and the report says so, but the code the clause names never reached the wire. Where the clause requires a specific code, that is a fail: the test asks what the client signalled, and a transport close signals nothing an HTTP/3 peer can read. - This paragraph used to say those cells read inconclusive. They did, until the observation was split out from the one that means “responded correctly”. Folding them together had reported “Responded correctly” about a code the server had never seen, so a client that sent an HTTP/3 code and got it slightly wrong scored worse than one that sent none at all. The wording above outlived that change by a few days and is corrected here rather than quietly replaced, because a rule page that has been wrong should say when.
- A fail is never inferred from silence. Every failure in this table rests on something the server watched the client do — a rejection carrying the wrong error code, datagrams that kept arriving after a Stateless Reset, a valid response abandoned. Absence of a rejection is consistent with a client that accepted the violation and with one that never saw it, so it is not reported as either.
- A silence only counts once the client has had its chance. Each test declares a reaction opportunity: not a stopwatch, but an event the server watched happen, after which a client that meant to object had the means to. For most tests it is the request and response completing. Where the anomaly rides a stream the client is not obliged to read promptly, it is the proof that the client read that stream. For the 0-RTT ports it is early data actually arriving and being refused in the handshake. For the tests whose anomaly is an absence — a Stateless Reset, a vanished path — it is the server still being there a set interval later, because there is no exchange to complete. A silence-based failure requires the implementation to have still been running when its opportunity arrived. One that terminated first is recorded inconclusive, because a peer that has gone and a peer that read the violation and tolerated it look identical from a socket, and only the second is a failure.
-
An exit afterwards erases nothing. The rule asks whether
the observation was still possible when the chance to make it came, not
how long the client stayed after. A client that leaves at seven seconds
of an eight-second window keeps its failure if its exchange completed at
one — the evidence was already collected. Where this changed a
verdict, the cell carries
client_exit_before_reaction_opportunityalongside the timestamps and the evidence the verdict was derived from, so it can be audited rather than taken on trust. - Client wrappers are deliberately thin. Each is a few lines that make one request and exit. Everything under test belongs to the library, not the wrapper; anything clever in a wrapper would be measuring the wrapper.
-
One of these is ours. The
quinnrow runs the same forked stack that serves the suite. It is included because excluding it would be hiding a result, but it is the one row to read with the most suspicion.
Generated 2026-10-22T04:21:51Z. Run the suite against your own client at conformance.pqcrypta.com.