Governing autonomy

Added · Last updated
Prerequisites

The actor has already delegated work to an agent, and the open question is how much it may do without asking. The actor has also judged the risks: which actions are cheap to redo, what it costs if the agent keeps going unchecked, and which effects cannot be reversed. That judgement happens before this sequence, and no pattern here makes it.

The actor has handed work to an agent and now has to decide how much the agent may do without asking.

Setting the limits

  1. Set limits on what the agent may do — Bounded autonomy

    Set limits on scope, spend, time, and risk for each capability, rather than one blanket permission. Show the actor the limits in plain terms before they take effect. Commit to returning the work to the actor whenever a limit is reached; the handoff steps below carry this out.

  2. Show spending against each limit — Cost transparency

    Show how much has been spent against each limit: an estimate before the run, a running total during it, and the remaining allowance. Track only the limits that matter to the actor. The estimate also appears in the preview shown before approval.

Checking costly actions

Applies when

Even within its limits, the agent reaches actions that matter: work that costs a lot, and effects that are hard to reverse. Checkpoints go where the consequences justify them, and routine actions go ahead without one.

  1. Rate each action by its consequences — Action consequences

    Rate each action by how long it takes to recover from, how much it affects, and whether it sets off further changes, using the risk judgement made before the sequence began. The rating decides which actions get a preview, which need approval, and which go ahead without either.

  2. Preview the work before it runs — Plan preview · optional

    Before the agent spends anything, show the steps, what they will touch, the expected results, and the cost estimate, so the actor knows what they are approving. Preview only work rated high enough. A continuous run skips this step, because its rules serve as its plan and the actor reviewed them when setting the limits.

  3. Ask for approval — Approval gate

    Nothing rated as consequential goes ahead without the actor's approval, given per action, per batch, or per set of changes. Approval requests come from three places: a previewed plan, a review of changes from delegated work, and a limit that has been reached. Approving makes the actor responsible for the outcome, which is why the record later needs to show what each approval covered.

  4. Keep the run within what was approved

    If the agent starts doing something the approval did not cover, treat it the same as reaching a limit: pause the run and decide what happens next.

Handing work over

Applies when

The run has paused before finishing. A limit was reached, the agent's confidence dropped too low, a decision came up that needs a person, or the actor stepped in. This is the handback the actor was promised when setting the limits. Each pause needs its own decision about who continues the work and with what permissions.

  1. Decide what happens to the paused run

    Decide for each pause. Two other outcomes lead out of this sequence: the actor takes over the work directly, or gives up on the goal.

    Combine any of:

    • Extend the limit and continue — Approval gate

      Turn the reached limit into an approval request: show what has been spent and what more is needed, then either extend the limit for this run or stop. If the same extension is asked for again and again, note it for the review of limits later on.

    • Hand the work to someone else — Handoff

      Pass the work on with its context: what has been done, what is left, and why it was handed over. The person taking over may be the actor, or someone with permissions or knowledge the agent lacked. Make clear who is responsible for keeping watch after the handover.

  2. Hand the work back to the agent — Handoff, its takeback direction

    Treat giving the work back to the agent as a separate step. Confirm that whatever caused the pause has been dealt with, agree on what the agent resumes and under which limits, and for a while give it narrower limits and have it show more of its work.

Adjusting the limits over time

Applies when

Many limits, approvals, and handovers have built up across runs. The actor is responsible for outcomes they only reviewed briefly, and without a proper record, the next set of limits will be based on impressions.

  1. Keep a record of every decision — Activity log

    Record every limit set, approval given, handover, and limit reached, with who acted, what they did, what it applied to, and when. The record should show exactly what each approval covered.

  2. Adjust the limits based on the record — Bounded autonomy, its incremental regrant

    Widen the limits after a history of runs without problems, and narrow them when the risk goes up. Make each change explicit and show it to the actor. Start with the extensions that keep being requested: a limit that has to be extended on every run is set in the wrong place.

Leaves: The actor decides how much the agent may do based on evidence. They are still responsible for work they only saw at its checkpoints, and that remains the cost of delegating.

Connections