Human review

Human-in-the-loop review for AI agent output

A reliable human-in-the-loop review workflow preserves the exact AI-generated artifact, anchors each finding to the revision and selected evidence, assigns an authorized resolver, and requires a later immutable revision before the finding is closed. The agent reads open annotations before continuing, and the human can verify which revision addressed each comment.

By Handover product team at 44pixelsUpdated August 4, 20268 minute read

Continue from evidence

Try the workflow before setup

Inspect a complete no-account handover, then carry its exact state into your first workspace or connect an agent.

Open live demo Connect an agent

What is human-in-the-loop review for an AI agent?

Human-in-the-loop review is a control in which an identified person inspects an agent's proposed output or action and can request changes, approve progression, or stop the workflow. For durable work products, the review should preserve what the person actually saw, the evidence behind it, the requested correction, and the actor who responded.

What should a human reviewer check first?

The reviewer should confirm the objective, evidence, assumptions, and acceptance criteria before polishing wording. A fluent output can still be wrong, unsupported, incomplete, or outside the task's permissions.

How should feedback be anchored?

Feedback should identify one immutable revision and artifact, preserve the selected quote or code range, and include enough surrounding context to survive later revisions. That anchor makes the review legible to both the original author and another agent.

What makes an annotation actionable for an agent?

An actionable annotation states the problem, expected change, and evidence or constraint the agent must use. Mentions can notify an owner, but they should never widen access to the underlying handover.

Name the claim, line, or decision that needs attention.
Explain why it is incomplete, risky, or inconsistent.
Specify the expected correction or verification step.
Assign an owner when the next actor is known.

How does an agent close the review loop?

The agent retrieves open annotations, publishes a new revision without mutating the reviewed artifact, and resolves only the threads the new revision actually addresses. The resolution should reference that revision so a reviewer can verify the correction.

handover.get
handover.read_artifact
handover.annotations { status: "open" }
handover.continue { expectedRevisionId: "rev_current" }
handover.update_comment {
  status: "resolved",
  resolvedRevisionId: "rev_corrected"
}

Handover review workflow showing three immutable revisions, Markdown, SQL and JSON artifacts, a human review note, an agent correction, and resolution linked to revision 3 — A real Handover product view: Sam anchors a timezone correction to the reviewed work, Alex publishes revision 3, and the thread records which revision resolved the finding.

What is the difference between feedback and approval?

Artifact feedback identifies a finding and requested change; runtime approval authorizes a proposed action. Permissions, destructive-action gates, and tool-call approval belong in the client, policy engine, or access layer. Comments preserve review work and evidence, but they should never be treated as authorization to execute a consequential action.

How do you test that the review loop really closed?

Inspect the original revision, open finding, corrected revision, and resolution link as separate records. The test passes only when authorship is attributable, the correction is visible in a later revision, resolved findings name that revision, unresolved findings remain open, and mentions or assignments did not widen access.

The original artifact remains inspectable.
The corrected artifact has a later immutable revision ID.
The review and resolution actions show authenticated identities.
Every resolved annotation links to the revision that addressed it.
A second reviewer can reproduce the evidence behind the correction.

Primary references