Governed Autonomy in an AI-Native Factory

RC23.1 — Bind every delegation to an authorization envelope

“The agent can use the repository” is not an authorization statement.

A production delegation should be represented as an authorization envelope containing:

1. the accountable principal and the agent acting on that principal's behalf;
2. a bounded task and intended outcome;
3. permitted and prohibited actions;
4. tools, data, services, environments, and network destinations;
5. credentials and delegated identities;
6. financial, compute, token, time, and action budgets;
7. the maximum consequence the delegation may create;
8. preconditions and evidence required before action;
9. verification required during and after action;
10. escalation, interruption, and containment routes;
11. expiry, renewal, contraction, and revocation;
12. evidence retained for reconstruction; and
13. correction, challenge, restoration, and remedy for affected parties.

Identity and authority are different. An authenticated agent can still be acting outside purpose, using excessive privilege, relying on an expired delegation, or carrying authority that cannot be traced to a principal. NIST's 2026 concept paper on software and AI agent identity and authorization highlights authentication, authorization, on-behalf-of relationships, dynamic policy, least privilege, intent, logging, and non-repudiation as open design requirements.C23-S01 It is a concept paper, not final guidance or proof that one architecture works.

Tool permission also needs more precision than “read” or “write.” NIST's tool-use taxonomy distinguishes read-only, constrained-write, and write permissions, and separates trusted from untrusted environments.C23-S02 A constrained write to a disposable test environment is not equivalent to a write to production. Reading a public dependency graph is not equivalent to reading secrets, customer records, or arbitrary files.

The envelope should therefore be evaluated against the real action path. A credential may appear narrow while the tool it unlocks can call another service, interpret untrusted content, or resume a privileged workflow. The authorized unit is the reachable consequence, not the name of the permission.

RC23.2 — Keep capability evidence separate from production permission

Capability evidence is necessary. It is not an authorization system.

SWE-bench evaluates whether systems can resolve real repository issues under a defined harness.C23-A01 SWE-bench Live adds fresh tasks and makes contamination and time visible.C23-A02 Failure analysis asks how agents fail and how evaluation harnesses affect the result.C23-A04 METR's task-completion time-horizon research conditions estimates on a task distribution and named success probability.C23-A03

These sources answer different questions. They should not be pooled into a single “autonomy score.”

For every capability claim, retain:

• the task suite and task distribution;
• the model, agent harness, tools, and settings;
• the evaluation date;
• the success criterion and probability;
• the estimated task duration;
• failures, exclusions, and evaluator uncertainty; and
• the reason transfer to the local task might fail.

METR warns that its current measurements above 16 hours are unreliable.C23-A03 That limit matters because the phrase “long-horizon agent” can sound more precise than the measurement. A horizon estimate is conditional evidence about evaluated tasks; it is not a claim that a system may safely work for that duration in production.

A local authorization decision adds questions the benchmark does not answer:

• What assets can be changed?
• How quickly can consequences spread?
• What can be reversed?
• Which evidence will be available at the decision point?
• What work must a human verify?
• What happens under adversarial or simply unexpected input?
• Who bears the cost of intervention and recovery?
• Can an affected person challenge and correct the result?

Capability can justify another evaluation. It cannot grant authority automatically.

F23.1 — Use a ladder that can move down

F23.1 is a task-specific authority-and-assurance ladder. It is not a maturity path.

Open figure

Its five operating states are:

1. Propose. The agent recommends; a named authority executes.
2. Prepare. The agent creates a change in an isolated environment; independent evidence is required before promotion.
3. Execute reversible. The agent may act inside a narrow envelope with automatic verification and tested restoration.
4. Execute consequential. Separate authorization, stronger evidence, live containment, and accountable intervention are required.
5. No delegation. Consequence, uncertainty, rights, law, irreversibility, or missing recourse prohibits agent execution.

The labels do not imply that state four is the destination. A task can move from execute-reversible to prepare when the environment changes, evidence expires, a new dependency appears, or recovery stops working. A task can move from propose to no delegation when the affected-party route is inadequate. A less capable model can be acceptable inside a narrower envelope; a more capable model can require re-evaluation because it changes reachable actions.

Every move needs a decision record:

• what changed;
• which evidence changed;
• which alternatives were considered;
• who authorized the move;
• which assurance increased or decreased;
• when the decision expires; and
• what would force authority down again.

The ladder is an aid for governing one task. It is not an organizational score, an agent rank, a procurement shortcut, or evidence that greater autonomy is desirable.

RC23.3 — Threat-model the action path, not only the model

An agent consumes instructions and context, maintains state, chooses tools, and acts through software. Threats can enter or propagate at each boundary.

AgentDojo provides reproducible tasks and security test cases for indirect prompt injection, making the tension between task success and security measurable in a benchmark setting.C23-A05 AgentDyn extends dynamic security evaluation and reports utility–security trade-offs under changing tasks.C23-A06 OWASP's agentic threat taxonomy includes behavior hijacking, tool misuse, identity and privilege abuse, memory and supply-chain risks, and cascading failures.C23-S07 MITRE ATLAS provides a maintained vocabulary of adversarial techniques for scenario design.C23-S09

These are testing resources, not production frequency estimates.

Public vulnerability records make several mechanisms concrete:

• CVE-2026-40117 describes a prompt-influenced agent with an unrestricted file-reading path capable of exfiltrating files; the affected system was fixed.C23-V01
• CVE-2026-12045 describes prompt injection bypassing an intended read-only database boundary through a multi-statement payload.C23-V02
• CVE-2026-57495 describes an external message resuming a highly privileged coding-agent workflow without adequate sender authorization.C23-V03
• CVE-2026-14898 describes indirect prompt injection forming a remote-image request capable of data disclosure; the public record reported no known in-the-wild exploitation at the time.C23-V04

Each record belongs to a product, version, configuration, and disclosure. The cases cannot be added together to calculate an incident rate. They do show why an authorization envelope must include untrusted inputs, reachable tools, downstream requests, credentials, resumption rules, and data egress—not only the model prompt.

F23.2 maps the full action path:

Open figure

principal → inputs → agent/model → memory and policy → tools and credentials → data and environments → downstream action → event evidence → response and recourse

For every path, ask what can influence the decision, what the action can reach, what evidence survives, and how authority is removed.

Multi-agent coordination adds trust edges

Dividing work among agents can clarify roles or increase specialization. It also adds identities, messages, delegated authority, shared state, routing, and failure propagation.

OWASP's multi-agent threat-modeling guide identifies the additional complexity and attack surfaces created by coordination.C23-S08 That supports a modeling obligation, not a verdict about the architecture.

A multi-agent record should name:

• every agent and accountable principal;
• the authority each agent receives and may delegate;
• message origin, integrity, destination, and expiry;
• shared memory and write ownership;
• tools and credentials reachable from each node;
• trust assumptions at each routing decision;
• disagreement and deadlock handling;
• maximum fan-out, depth, duration, and spend;
• failure and compromise propagation;
• containment and revocation across the group; and
• reconstruction of the final action across all participants.

Do not let an orchestrator become an accountability eraser. “Agent B decided” does not explain why Agent A delegated, which evidence B used, which policy allowed the tool call, or who owned the consequence.

The architecture may reduce one risk while increasing another. Specialized agents can narrow permissions, yet additional messages can carry malicious or stale context. Independent checking can detect some errors, yet agents sharing the same model, data, or assumptions may fail together. These are hypotheses to test locally.

Production evidence must reconstruct action and consequence

Observability is not the volume of logs. It is the ability to answer a consequential question with trustworthy evidence.

For an agentic task, retain events for:

• attempted, completed, denied, and abandoned actions;
• user, system, external, and retrieved inputs;
• model, harness, policy, prompt, memory, and tool versions;
• tool calls, parameters, results, retries, and downstream effects;
• credential and authorization decisions;
• human interventions and their timing;
• validation, approval, rejection, and override;
• anomalies, errors, incidents, and affected resources;
• rollback, compensating action, restoration, and verification;
• token, model, tool, infrastructure, review, and recovery cost; and
• evidence links needed for challenge, audit, and learning.

Google Cloud's agent-observability documentation names practical fields such as model interactions, tool use, token usage, latency, errors, and exchanged data.C23-S11 NIST's evaluation-probe project aims to produce structured audit trails linking agent claims and actions to supporting evidence.C23-S05 Product documentation and active research show useful fields and directions; neither proves that a deployed audit trail is complete or trustworthy.

Evidence integrity needs its own control. Logs can omit denied attempts, lose cross-service identity, expose sensitive prompts, or be writable by the system they are meant to audit. Retention can create privacy or security risk. A reconstruction test should start with a real question—“Why was this production change executed?”—and confirm that an independent reviewer can trace authority, inputs, evidence, actions, policy, outcome, and response.

If the trace ends at “the agent decided,” the system is not observable enough for the consequence.

Reversibility includes response and recourse

Rollback is powerful and often incomplete.

A code change may be reverted while disclosed data cannot be recalled. A database can be restored while decisions made from corrupted records persist. A service can recover while a user remains locked out or wrongly classified. A compensating action may repair a balance without repairing delay, loss, or trust.

For each authorized action, classify:

• technically reversible;
• reversible only within a time window;
• recoverable through a compensating action;
• restorable with possible data or service loss; or
• materially irreversible.

Then specify:

1. how the system pauses and credentials are revoked;
2. how spread is contained;
3. who investigates;
4. which evidence is preserved;
5. how restoration is verified;
6. who is notified;
7. how an affected person can correct or challenge the result;
8. who can authorize remedy;
9. how learning changes the envelope; and
10. what evidence is required before reauthorization.

NIST AI RMF supplies a lifecycle governance basis for accountability, monitoring, response, and attention to affected communities.C23-S06 It does not establish that a recourse route works. A production team must test whether a person can find the route, obtain relevant evidence, reach an accountable owner, pause continuing harm, receive correction or restoration, and appeal an unresolved decision.

Recourse failure is an authority failure. If consequential action cannot be challenged or repaired, the envelope should contract or prohibit delegation.

Count assurance and recovery in the economics

Autonomy can reduce elapsed work in one step and increase cost elsewhere.

The FinOps Foundation's AI guidance emphasizes allocation, forecasting, usage, optimization, and connection to business value.C23-S10 For an agentic task, the unit record should include:

• model and token use;
• tool and service charges;
• infrastructure and retained evidence;
• evaluation and test execution;
• human supervision and verification;
• policy exceptions and security review;
• intervention and incident response;
• rollback, restoration, and rework;
• delay and failed-task cost;
• displaced workload in downstream teams; and
• harm response and remedy.

A low cost per model call can coexist with expensive verification. A faster change can increase queueing in review or recovery. A constrained envelope can make the task uneconomic. That is a valid result.

Do not convert these fields into a universal autonomy return. State the local value hypothesis, baseline, alternatives, observation window, affected groups, missing costs, and decision rule. Continue, narrow, redesign, or stop based on the whole task—not the cheapest visible component.