Speculative decoding
ProAn illustration of speculative decoding: a fast draft model guesses a few tokens, the target model checks them in one pass, and accepted guesses join the answer.
Live · keyboard ready
Speculative decoding
Draft guesses 4 tokens ahead, target checks them in one pass
Output0 passes
- Explaining why a product answers faster: docs, model pages, inference dashboards and blog posts about latency.
- Showing a recorded speculative run from your own server, with acceptance and tokens per pass for a chosen draft length.
- Teaching the trade off between draft length, acceptance rate and speedup with a slider people can try.
- Use text-stream to show an answer streaming to a user; this card explains the mechanism behind the speed.
- Use token-inspector to look at token probabilities and alternatives for one answer.
- Use model-router to explain how requests are sent to different models; speculative decoding runs two models on every request.
- Use attention-map or mixture-of-experts for what happens inside a single forward pass.
Installation
Pro source and install commands unlock with a Pro plan.
example.tsx
import { SpeculativeDecoding } from "@/components/arc/speculative-decoding/speculative-decoding"; // Simulated: pass the answer and let the seeded draft guess.<SpeculativeDecoding tokens={[{ text: "The" }, { text: " build", ease: 0.6, miss: " deploy" }, { text: " passed" }, { text: "." }]} defaultDraftLength={5} defaultCompare/> // Replayed: pass rounds recorded from your own server.<SpeculativeDecoding rounds={[ { draft: ["The", " build", " failed"], accepted: 2, target: " passed" }, { draft: [], accepted: 0, target: "." }, ]} onComplete={stats => console.log(stats.speedup)}/>SpeculativeDecoding
An illustration of speculative decoding in one card. A draft model lane proposes a few ghost tokens quickly; a target model lane checks them in one sweep. Accepted guesses get a check and travel into the answer, the first miss is struck through and replaced by the target's token, and the rest fade away. Rounds repeat until the sentence is done. A compare switch races a second line where the target writes one token per pass, while the speedup, the share of drafts accepted and the tokens per target pass roll up below. Every token in the answer is underlined in the color of the model that wrote it.
PropTypeDefaultDescription
tokensDecodeToken[]sampleTokensThe target model's answer, one entry per token with its leading space ({ text: " deploy" }). Optional ease (0 to 1) is how likely the draft guesses the token, and miss is what it proposes instead.roundsDecodeRound[]–A recorded run to replay instead of the simulation: per round the draft tokens, how many were accepted from the start, and the target's own token. Hides the draft length slider.draftLengthnumber–Tokens the draft guesses per round, 2 to 8 (controlled). Changing it replans and replays from the start.defaultDraftLengthnumber4Draft length on first render.onDraftLengthChange(length: number) => void–Called when the slider moves.compareboolean–Shows the target only line and races it on the same clock (controlled).defaultComparebooleanfalseCompare state on first render.onCompareChange(compare: boolean) => void–Called when the compare switch changes.autoplaybooleantrueStarts the run once, when the card is mostly in view. Never starts on its own with reduced motion.seednumber12Seed for the simulated guesses. The same seed, tokens and draft length always give the same run.draftCostnumber0.1Cost of one draft token as a share of a target pass. It sets how long drafting takes on screen and counts toward the speedup.passDurationnumber640Milliseconds one target pass takes on screen.titlestring"Speculative decoding"Card title and the group's accessible name.draftLabel / draftDetailstring"Draft model" / "Small, fast"Name and detail of the draft lane. Keep them generic or clearly illustrative.targetLabel / targetDetailstring"Target model" / "Large"Name and detail of the target lane.status"ready" | "loading" | "error""ready"Loading dims the lanes and shimmers the status line; error shows the message with a retry. Controls are disabled in both.errorstring–Message shown when status is "error".onRetry() => void–Called by the retry button in the error state. The button is hidden without it.onComplete(stats: DecodeStats) => void–Called each time the speculative line finishes, with passes, drafted, accepted, tokens, acceptance, tokensPerPass and speedup.classNamestring–Class on the root element.styleCSSProperties–Inline style on the root element.- Tab
- Moves through play or pause, restart, the draft length slider and the compare switch.
- Enter, Space
- On play or pause and restart, activates them. On the compare switch, toggles the race.
- Arrow leftorArrow right, Home, End
- On the draft length slider, changes the length by one or jumps to 2 or 8. The run replays with the new length.
- The card is a named group. The lanes are hidden from assistive technology and summarized in hidden text once each round has its verdicts: what the draft proposed, how many the target accepted and which token it wrote instead.
- Each output line carries the text written so far as its accessible name, so a screen reader reads words instead of tokens.
- Stats are a description list with spoken values ("2.6 times", "67 percent"); the rolling digits are decorative.
- Verdicts never rely on color: accepted columns show a check, the rejected guess is struck through with an x beside the target's token, and discarded guesses fade.
- A polite live region announces play, pause, restart, the comparison, a new draft length with its expected acceptance and speedup, and the finished result. It never announces every round.
- Pointer focus draws no ring; keyboard focus shows Arc's shared focus ring.
- Draft chips appear one by one, fading in from 4px to the left, with a short shimmer on the text while they wait.
- The target's sweep fills each cell in order from a per frame CSS variable; verdict icons and the replacement token crossfade in as the sweep reaches them, and the strike draws from the left.
- Accepted tokens and the target's token travel from their chips into their slot in the answer on a critically damped spring, about 20ms apart. The answer is laid out in full from the start, so the line never reflows.
- Lane height follows wrapping chips on the smooth spring; the target only line opens and closes on the same spring. Stats roll per digit with tabular numerals.
- Status text swaps with a short crossfade; no character morphs, blur or typewriter effects.
- Reduced motion never autoplays; when played, tokens appear in place without travel, the sweep steps per column, and the shimmer, slide and rolls are off.
- Lane and line labels sit in a 120px column on wide cards. Below 560px the lane labels become a legend above the columns and line labels sit above their text with the pass count on the right.
- Draft columns wrap onto a second line when they run out of room; the lane height follows on a spring.
- The status line reserves two lines on narrow cards so wrapping copy never moves the lanes. Stats stay three across; the slider takes a full row above the compare switch.
- The clock runs on requestAnimationFrame in a ref. React state changes only when something on screen changes (a chip, a verdict, a landed token); the sweep is a CSS variable written each frame.
- Plans are memoized on a serialized copy of the inputs, so an inline tokens array does not restart the run each render.
- The loop stops when the run ends or is paused, and frame steps are capped so a background tab never jumps ahead.
Notes for AI
Give your coding assistant the Markdown reference instead of screenshots.
- This is an explainer, not a live decoder. With tokens it simulates a draft model from ease values and a seed; with rounds it replays a real run. Say "illustrative" when the numbers are simulated.
- Token text keeps its leading space, as tokenizers do. Punctuation without a space stays on the same line as the word before it.
- The speedup counts draft time: tokens divided by the sum of one pass per round plus draftCost per drafted token. Longer drafts accept a smaller share because everything after the first miss is discarded, so the speedup peaks at a middle length.
- Keep model names generic (Draft model, Target model) unless the run really came from named models.
- The simulation always leaves the final token to the target, so every round ends with a token from the target.
The full library index for assistants is at /llms.txt.