Open pilot · v1.0.0
AI Handoff Continuity Benchmark
A reproducible test of whether an AI model can recover the exact state required to continue interrupted work from a transcript, compressed memory, or structured handoff.
- Cases
- 3
- Conditions
- 3
- Prompts per run
- 9
- Scoring
- Deterministic
What this tests
Can the next model resume without reconstructing the work?
Each case describes the same interrupted task in three forms. The model selects candidate IDs for the objective, current state, decisions, evidence, constraints, next action, owner, and unresolved questions. A dependency-free runner scores those selections against a published answer key. No LLM judge is used.
Comparison design
Three ways to carry the same work forward
Transcript
The original conversation, including superseded decisions, incidental detail, and chronological corrections.
Compressed memory
A short narrative summary that retains the broad direction but omits some exact evidence, constraints, or open questions.
Structured handoff
An explicit continuation record containing objective, current state, decisions, evidence, constraints, next action, owner, and open questions.
Method
Inspect every input and reproduce every score
Authored pilot cases
The initial dataset covers an incident response, product research, and a data-model change. Every distractor and expected answer is visible.
Constrained answers
Models choose published candidate IDs. This isolates state recovery from prose style and makes scoring repeatable.
Deterministic scoring
Single-value fields require an exact match. Multi-value fields use set F1. Missing answers score zero.
Run it
One dataset, one runner, no dependencies
Node.js 22 or newer is sufficient. Generate the nine prompts, run them with a fixed model configuration, assemble the returned JSON, and score the submission.
curl -O https://handover.sh/benchmark/v1/run.mjs
curl -O https://handover.sh/benchmark/v1/dataset.json
node run.mjs --prompts ./prompts
node run.mjs --template ./submission.json
node run.mjs --submission ./submission.json100-point rubric
Continuation-critical information carries the most weight
Current status
The methodology is public. Cross-model results come next.
This release is an open pilot, not a leaderboard. The scorer has been validated against the answer key, but we will not publish model comparisons until runs include pinned model versions, raw submissions, repeated trials, and documented execution settings.
- Cases are authored rather than sampled from an independent corpus.
- The candidate format measures state recovery, not free-form task quality.
- The three conditions intentionally retain different amounts of detail.
- Handover created the benchmark and benefits if structured handoffs perform well.
Research context