Calibrating reliance

Added · Last updated
Prerequisites

An AI is in use and its output will be acted on: an answer taken into a decision, a suggestion accepted into the work, a change approved into a document. The stakes of acting on wrong output have been judged for each surface, and they set how much of this sequence each surface needs. That judgement happens before this sequence, and no pattern here makes it.

An AI is in use, and its output is about to be acted on by an actor who did not produce it and cannot tell how far to trust it. These steps are taken for each AI surface rather than once for the whole system: every other AI sequence arrives here where its output meets the actor.

Making the output checkable

  1. Mark what made it — Provenance marking · optional

    Decide first whether machine-made content is marked, because once anything is marked, unmarked content reads as human-made. Decide it where the output travels, such as an export, a shared document, or a mixed thread, and make the mark survive copying. Skip this where the surface already shows its origin, as a conversation does.

  2. Put the evidence on the claims

    Three marks answer three questions about the same claims: where they came from, how sure the system is, and how it got there. Let the stakes decide which ones this surface carries, and use each only where its question is the one the actor would ask.

    In any order:

    • Ground the claims — Citation

      Link each claim that needs it to the passage it rests on. Decide which claims need one as a policy, such as quotations, figures, and surprising or contested claims, and let a claim with nothing to point to say so.

    • Let the confidence show — Expressed uncertainty

      Mark doubtful spans with hedges, ranges, or visual marks, so scrutiny lands where it is needed. A claim the grounding step could not link to anything needs this mark.

    • Expose the path — Transparent reasoning · optional

      Where the output was reached through several steps, show them as a trail the actor can scroll, expand, and challenge. Skip this for single-move output, such as a completion or a label.

  3. Answer why on demand — Explanation, its AI explanations

    Let the actor open an explanation of the output, such as the factors behind a recommendation and the limits of what it drew on, and choose how deep it goes. Place it after the evidence, because a convincing explanation makes the actor less likely to open the sources.

Interposing the judgement

Applies when

The evidence is on the output and mostly unread: the marks reassure whether or not anyone follows them, and accepting still costs less than evaluating. The actor could judge and the stakes call for it, but nothing asks them to.

  1. Make the acceptance cost a thought — Cognitive forcing functions

    Put a moment of active thought between the output and its acceptance: a wait, a reveal on demand, an answer given first, or a reason written down. Use it only where the stakes keep justifying it. Reasons written down here are kept for the record steps.

  2. Sharpen the friction

    Two further forms, in no set order: one structures a single review, the other keeps the actor's skill up across many.

    In any order:

    • Structure what the review checks — Checklist · optional

      Where the ways a decision can go wrong are known in advance, list them, and make each item point somewhere the actor would not have looked. Skip this where they are too varied to list.

    • Withhold the answer sometimes — Human goes first · optional

      Where the surface is meant to support the actor's own competence, now and then withhold the recommendation, ask for the actor's answer, and show the two side by side. Skip this for one-off tasks.

Reading the reliance back

Applies when

Acceptances, overrides, opened sources, and cleared checks have built up over the surface's use. Without a record, whether trust matches accuracy is judged on impressions.

  1. Keep the record of the judgements — Activity log

    Record what was accepted and what was overridden, which sources were opened, and the reasons given, each with who acted, what they did, what it applied to, and when. This is the only record that shows acceptances nobody read.

  2. Move the friction on evidence — Cognitive forcing functions, where it is spent

    Add friction where acceptances go through unread, and move or remove a check that every actor clears in one motion, as an explicit decision. An override that keeps recurring goes to tuning.

Leaves: The output shows what it rests on, accepting it takes a judgement where a wrong answer would cost more, and the record shows whether both are working. Every mark and pause costs the actor attention.

Connections