An AI agent governance framework for shared company context

What is AI agent governance?

AI agent governance is the operating system for deciding which agents may exist, what they may access, which actions they may take, how humans supervise consequential work, and what evidence proves the controls worked. It combines organizational ownership, risk management, identity and access management, review workflows, monitoring, and incident response. A principles document is useful, but it is not governance until the rules are enforced and testable.

What should an AI agent inventory contain?

Inventory each deployed or experimental agent as a governed resource. Give it a stable identifier and record enough context to evaluate its access, lifecycle, and blast radius without reverse-engineering a prompt or repository.

• Business owner, technical owner, and review owner.
• Purpose, expected outcome, users, and deployment environment.
• Models, hosts, MCP servers, APIs, tools, and data sources.
• Human identity or named service identity used at every boundary.
• Read, write, publish, delete, execute, and administrative capabilities.
• Risk tier, approval gates, monitoring signals, and incident contact.
• Credential rotation, last review, expiry, disable, and deletion dates.

How should companies tier AI agent risk?

Tier the deployed use case, not the model name. Risk grows with data sensitivity, action reversibility, autonomy, external impact, and the difficulty of detecting a wrong result. A retrieval assistant reading public documents is materially different from an unattended agent that can publish externally, change production data, or approve spending.

Tier 1 — advisory
Reads low-sensitivity context; no consequential writes.

Tier 2 — assisted action
Prepares changes or internal reports; a named human approves execution.

Tier 3 — bounded autonomy
Executes reversible actions inside explicit scope, budget, and monitoring limits.

Tier 4 — consequential autonomy
Touches sensitive data, production systems, money, legal commitments, or public
communications. Requires formal review, strong isolation, and tested shutdown.

How should identity and access be designed for AI agents?

Treat each unattended agent as a distinct non-human principal. Use a single-purpose, revocable identity; grant only the scopes and workspaces required for its current job; derive authorship from the accepted credential; and separate read, create, continue, publish, delete, and administrative permissions. Never let a caller choose its own author identity or company boundary in tool input.

• Use per-user OAuth for interactive people when the host supports it.
• Use one named service identity per unattended agent or automation boundary.
• Keep credentials outside prompts, source code, URLs, and shared configuration.
• Bind tokens to their intended resource and validate them on every request.
• Apply the same authorization checks to search indexes as direct record reads.
• Disable unused identities before deletion and verify that access actually stops.

Where should human approval be required?

Require approval at the point of consequential action, not as a generic chat instruction. The reviewer should see the proposed action, affected resource, evidence, uncertainty, alternatives, and rollback plan. Approval should be attributable, time-bounded, and scoped to that action; it should not silently grant the agent broader future access.

• External publication or communication.
• Destructive or hard-to-reverse changes.
• Production, financial, legal, employment, or customer-impacting actions.
• Access to a new sensitive data class or company area.
• Privilege elevation, credential creation, or delegation to another agent.
• Resolution of a high-severity review finding or policy exception.

What evidence should an agent leave behind?

Keep an attributable continuation record for work another person or agent may need to verify. Preserve the objective, current state, decisions, evidence, constraints, open questions, next action, exact artifacts, model or tool boundary, and review status. Revisions should be immutable so later corrections do not erase what a reviewer originally saw.

How should AI agent governance be tested?

Test controls as workflows, not checkboxes. Use a second identity to confirm isolation, attempt an out-of-scope search and write, exercise approval gates, revoke the credential, and verify the agent can no longer read or act. Then reconstruct one consequential decision from the retained evidence and review history.

• Authorization test: denied scopes and workspaces remain undiscoverable.
• Attribution test: every revision and comment names the authenticated principal.
• Continuity test: a successor recovers the exact state without the original chat.
• Approval test: gated actions cannot execute before the named approval.
• Revocation test: a disabled credential fails immediately across API, CLI, and MCP.
• Recovery test: the owner can stop the agent and resume from the last trusted state.

What should happen when an AI agent is retired?

Offboarding should disable the identity first, preserve records needed for audit or continuation, transfer ownership of active work, remove secrets and scheduled jobs, and only then delete the identity after the retention window. Reusing the same credential for a replacement agent destroys attribution and can silently inherit excessive access.