01
DraftSmall edge model
Candidate tokens are generated one after another at the edge.
Edge-cloud speculative decoding
Adapt before the round.
Stop after a mismatch.
Speculative decoding lets a small edge model propose candidate tokens while a large cloud model verifies them in parallel. RAPID makes proactive edge drafting adapt to local slowdown and stop when cloud feedback invalidates the branch.
- Before generation
- CAAC selects depth
- During expansion
- IPDG checks feedback
- At the cloud
- Verification is unchanged
Round trace
Proactive branch Di+1
EDGEsequential expansion
Di
CLOUDparallel verification
Verify Di
IPDG / BOUNDARY CHECK
A root mismatch arrives while E3 is running. IPDG finishes E3, then stops before E4 and restores the saved boundary.
Small model drafts
Large model verifies
Accepted prefix continues
Speculative decoding, briefly
Draft sequentially.
Verify in parallel.
A lightweight model proposes several candidate tokens. The target model checks all candidate positions in one pass, keeps the accepted prefix, and determines the final output.
02
VerifyLarge cloud model
The target model evaluates the candidate positions in parallel.
03
AcceptVerified output
The accepted prefix is kept. Tokens after the first rejection are discarded.
Two edge-cloud PDG problems
Plan for slowdown.
Stop invalid work.
CAAC and IPDG solve different problems at different points in a decoding round. Neither changes how the cloud target model verifies tokens.
Before generation
CAAC
Congestion-Aware Adaptive Controller
Problem 01
A draft depth chosen before the round can become too long when contention slows the edge. Local drafting then outlasts the cloud response and delays the next request.
CAAC response
Measure local per-token latency and recent token acceptance, then adjust the next draft depth before generation begins.
Physical path
Has local generation slowed?
τcur
local latency
fast estimate
slow baseline
ESR → η
slowdown gate
Logical path
Is a deeper draft useful?
A
acceptance
PID
depth proposal
B̂
proposed depth
FUSION
Apply the slowdown gate, then select the next integer depth.
next depth
Li+1d
1 · · · 8
A recent slowdown opens a gap between the fast estimate and the slow baseline. CAAC uses that gap to reduce depth, while acceptance feedback keeps useful drafting from becoming too shallow.
Opencaac.py
During expansion
IPDG
Interruptible Proactive Draft Generation
Problem 02
Cloud feedback can invalidate a proactive branch while the edge is still expanding it. NPDG continues to the planned depth, so the remaining work is discarded.
IPDG response
Check feedback after each complete expansion. After a mismatch, start no further step and restore the saved tree and KV-cache boundary.
-
01
Save boundary
Record the active tree and aligned KV-cache extent.
-
02
Expand once
Complete one tree-expansion step before checking feedback.
-
03
Check the result
A match keeps the branch. A mismatch closes the expansion gate.
-
04
Reconcile
Retain and remap, or restore the boundary and discard.
Match Continue to the selected depth
Mismatch Stop before the next expansion
ipdg.py
One RAPID round
The modules meet at a clean boundary.
CAAC sets a maximum depth. IPDG may stop earlier, but only after a complete expansion step. The cloud verifier and its token acceptance rule do not change.
Inspect the round coordinator-
t0
Select depth
CAAC uses latency and acceptance from the previous round.
-
t1
Send and expand
The edge sends Di and starts the proactive branch.
-
t2
Check feedback
IPDG checks after each completed expansion.
-
t3
Keep or restore
The branch is remapped after a match or discarded after a mismatch.
Experimental results
Measured gains across models, workloads, and edge devices.
RAPID was evaluated on five datasets, four target and draft model pairs, and two edge platforms. Every reported configuration was repeated three times.
Maximum measured throughput relative to fixed-budget drafting across the tested settings.
Maximum latency reduction relative to fixed-budget drafting in the Jetson evaluation.
ERCR across four target and draft model pairs.
Component ablation
Each mechanism addresses its own loss.
Under sustained edge contention, IPDG stops invalid work at expansion boundaries. CAAC contributes most of the recovery by adapting the draft plan before generation. Their combination gives the highest throughput.
Qwen3-14B/0.6B. Three-run macro mean over C4, OASST, and SpecBench.Throughputtok/s
RAPID records the highest mean throughput and lowest mean latency in 11 of the 12 Jetson model-by-workload comparisons. The remaining comparison is a near tie with EdgeLLM. These are results reported in the paper. This repository contains the mechanism-level reference implementation, not the experiment harness.
Minimal reference implementation
Read the mechanism without the experiment harness.
The repository keeps the paper-aligned controller, boundary protocol, coordinator, tests, and a small simulation. Raw results, machine-specific scripts, and unrelated method prototypes are intentionally excluded.