The same model, different accelerators.
How much do the answers actually change?

Nobody publishes this number. We are going to.

If you move an LLM from NVIDIA to AMD, Huawei Ascend, Intel Gaudi or AWS Trainium, the outputs will not be identical. Floating-point arithmetic is not associative, so different kernels reduce in a different order and the logprobs shift. That is expected and unavoidable.

The question that matters is how much — and whether the change you are seeing is ordinary numerical noise or a genuine porting fault. Today there is no public reference to answer that against.

What the gate detects — measured on real hardware

Before publishing accuracy deltas we had to establish that our own measurement works. It is calibrated against a corpus of real logprobs captured from Qwen3-8B running on an NVIDIA A40, not from synthetic fixtures. We inject faults that mimic real porting failures and record whether the gate catches them, and at what magnitude.

Rig: vLLM, temperature=0, seed=1234, top_logprobs=20, 16 prompts (EN + 中文). The corpus is published, so every number below is reproducible offline.

Those fault magnitudes are measured against a simulated correct port — bfloat16 rounding applied to the A40’s own output. We then tested that simulation against two real GPUs, and it did not survive. See below.

The first real cross-hardware measurement

NVIDIA A40 (Ampere) against NVIDIA RTX 6000 Ada. Same vendor, same 48 GB, same vLLM image, same Qwen3-8B, same seed, both warm and both individually bit-exact. The only variable is the silicon.

Two NVIDIA GPUs running an identical stack produce different text on 26% of prompts (95% CI 16.8–38.4%). Not merely different distributions — different sentences.

Measured twice on separately rented pods. Restricted to the 16 prompts used in the first run, the second reproduced it to nine decimal places (0.122442606) — so this is a deterministic property of the two GPU models, not an artefact of one rental. Both sides were bit-exact on their own warm repeat, 61/61.

Divergence tracks output length, not prompt type. We widened the corpus expecting open-ended prompts to dominate; they did not. Two short factual questions diverged — including “What is the chemical formula for water?” at 13 generated tokens. Diverged prompts averaged 49 tokens against 41 overall. So the practical guidance is: the longer the generation, the more divergence to expect — a classification workload and a long-form generation workload do not carry the same risk.

This is the number the industry does not publish, and it reframes the question. Before you can ask whether an Ascend or AMD port is faithful, you have to know what faithful looks like between two cards from the same vendor. It is not zero.

It also broke our own threshold, so we are publishing that too

We had calibrated the gate at 2.2e-03 using bfloat16 rounding as a stand-in for two hardware kernels disagreeing. Measured against real silicon:

Simulated rounding understated the real effect by 95×. Worse, genuine cross-silicon variation is larger than a real temperature bug — so a single distance metric, on its own, cannot tell a correct port on new hardware apart from a broken one.

How we fixed it — and the hole we found in the obvious fix

The tempting fix was to drop distribution distance entirely and gate on the two metrics with a perfect record on real hardware: top-1 agreement (1.000 at every compared position, three independent runs) and probability-mass preservation. We tested that against every fault before shipping it. It has a hole.

A transposed-operator bug — logprob values scrambled, token identity untouched — moves both of those by exactly zero. Permuting values within a row changes neither which token sits at rank 0 nor the row’s sum. Only a metric that matches probability by token identity catches it.

So the gate is compound: three metrics, any one failing means FAIL. Distribution distance stays — recalibrated, not loosened. Its ceiling moved from 0.0022 to 0.4402: the geometric mean of measured real cross-hardware noise (0.195) and the weakest fault it must catch (0.993), clear by 2.26× on each side. The old number was never a wrong idea; it was anchored to a simulation that was 95× too optimistic.

Disclosed rather than buried: a 1% temperature error is not reliably caught by any of the three once thresholds tolerate real cross-hardware noise. No threshold wide enough for 0.195 can also catch a fault scoring 0.004. Our test suite asserts this gap exists, so it cannot close or widen silently.

Caveat we will not bury: 61 prompts still leaves a 95% interval of 16.8–38.4%. The rate did not measurably change when we went from 16 prompts to 61 — the interval simply narrowed — so treat 26% as an estimate with real uncertainty, not a specification.

Three things we found while calibrating, which we would rather publish than hide

vLLM is bit-exact — but only once the prefix cache is warm. Repeat a request on a warm server and every logprob is identical to the last bit (16 of 16 prompts). But a prompt’s first execution takes zero cache hits and returns measurably different logprobs — up to 0.087 in token probability, while the generated text stays identical. That is 24× larger than the weakest fault we need to catch.

This is the trap in any naive A-vs-B accuracy comparison: whichever backend happens to be colder looks broken, and the benchmark measures the cache rather than the silicon. Our harness warms both sides on every prompt and discards the result before measuring. Once warm, the reference contributes zero noise — which is what makes it possible to attribute a difference to the target at all.

Cosine similarity cannot detect scaling errors. It is scale-invariant by definition — multiply every logprob by 1.01 or by 2.0 and it returns exactly 1.0000000000 either way. It is widely used for this purpose and it is the wrong instrument. We dropped it.

Comparing logprobs by rank compares the wrong tokens. Deep in the top-k many tokens are near-tied, so a negligible perturbation swaps their order — we recorded rank 3 holding '高' on one side and ' like' on the other. And in log space a 0.5 shift at logprob −26 (a probability near 1e-12, a token that will never be sampled) scores the same as a 0.5 shift at the most likely token, which changes the output. Measured on real hardware, that metric’s noise and its faults overlap: no threshold separates them. We match tokens by identity and compare probabilities instead.

The accuracy delta table

The batch-size row matters as much as the vendor row. Published research shows BF16 accuracy varying by up to 9% from GPU count and batch size alone. Without it, a cross-vendor number looks damning when it may be ordinary.

Method

• Black-box, over the wire. We compare two OpenAI-compatible endpoints. The harness never sees the silicon — only logprobs over HTTP.
• Statistical, never bitwise. Bitwise equivalence across vendors is not achievable. Anyone claiming it is misleading you.
• Worst case, not just mean. A mean over prompts hides the single bad position that actually matters. We report p50, p99 and max.
• Full provenance. Weights SHA256, engine version, driver and toolkit versions, batch size, seed, and the prompt-set hash — published with every number.
• Raw data published, not only the summary table.

ruitong port Qwen3-8B \
  --reference nvidia-a100=http://ref:8000 \
  --candidate amd-mi300x=http://cand:8000 \
  --output results/qwen3-8b__a100__mi300x.json

Exit codes: 0 equivalent · 1 gate failed · 2 could not run. The third is deliberately distinct — "the port is broken" and "we could not tell" demand opposite responses.

Who this is for

Teams migrating inference between accelerators who need to answer, to someone who will ask: did the model's behaviour change, and by how much? If that question arrives from a risk, audit or procurement function, a spreadsheet from a debugging tool is not an answer.

The reference table is public and free. If you need it run on your model and your hardware, with a signed, provenance-bound report, get in touch — or buy directly below.

Pricing

Priced per engagement, not a subscription you have to justify before you have a number. Each report is a real comparison — run on rented reference hardware plus your candidate endpoint — never a simulated placeholder.

Payment via Stripe (card). Full refund if we cannot produce a report — the D8 "could not run" case is on us, not you.

Prefer crypto? Send USDT (Ethereum / ERC-20) to 0xdA2b75956eaba2E50075E3258a496C89168d5E80 and email the transaction hash to hello@ruitong.io — we confirm and start the same day.