A data platform team is considering an AI agent for incident triage. The agent would read alerts, inspect pipeline metadata, suggest a likely cause, open a ticket, and perhaps rerun a failed job. The prototype handles the demonstration cases well. Engineers can see that it might reduce repetitive work. They can also see several ways it could create a larger incident.

The team does not lack information. It has architecture diagrams, sample traces, vendor documentation, a backlog, security requirements, and opinions from experienced people. What it lacks is complete evidence about how the system will behave inside a changing production environment.

Waiting for certainty is not a serious option. Neither is converting every unknown into a confident number and calling the analysis finished.

This is the normal shape of consequential technical work. A team must choose while model behavior, user response, operating cost, data quality, and future constraints are still partly unknown. Decision quality therefore depends less on predicting the future perfectly and more on building a process that exposes what is uncertain, limits the cost of being wrong, and makes revision possible.

Start with the uncertainty contract

Before the architecture debate becomes a contest between preferred solutions, write one page that defines how the decision will work. I call it an uncertainty contract.

The word contract matters. The page is not just a record of what people believed. It states what evidence the team will seek, how much exposure it will accept, and what it promises to do when the evidence arrives.

This artifact is also intentionally narrower than a project plan. It does not contain every task, risk, stakeholder, or technical detail. The companion note on what decision checklists miss explains why any structured representation needs a challenge around its edges. The uncertainty contract has a different job: preserve the logic that connects an incomplete picture to a bounded action.

Let the cost of delay set the pace

Teams often describe decisions as either fast or careful. That is the wrong contrast. A careful decision may need to be fast because delay is expensive. A slow decision may still be careless because time passed without producing better evidence.

Use two dimensions: the impact of a wrong choice and the cost of waiting.

The incident agent belongs in the last row if it can change production systems. A wrong action could be serious, but waiting also consumes scarce operator attention. The answer is not a six-month analysis or immediate autonomy. It is staged authority: begin with read-only access, collect evidence, let operators approve actions, and expand only when the system earns a wider boundary.

This distinction prevents governance from becoming a universal brake. A naming convention, internal prototype, or reversible prompt change does not deserve the same process as a system that moves money, changes customer access, affects employment, or writes to production. More ceremony is not automatically more responsible. The process should grow with consequence, irreversibility, and reach.

Replace confidence theater with testable claims

“We are 80 percent confident” sounds disciplined, but the number is useless unless people can explain what it describes. Is it confidence that the model can classify common failures? That operators will use the workflow? That the vendor will meet a latency target? That total operating cost will remain within budget? These are different claims with different evidence.

Represent uncertainty in a form the team can act on:

• State the claim. “The agent will identify the correct pipeline owner for most common alerts.”
• Name the evidence. “It succeeded on 43 of 50 replayed incidents from two pipeline families.”
• Mark the boundary. “The sample excludes schema migrations, regional outages, and incidents with missing lineage.”
• Name what would change the view. “A higher error rate on live shadow traffic or concentrated failures for one team.”
• Choose an action that fits the gap. “Keep the agent advisory until live evidence covers the missing cases.”

Ranges are often more honest than a single forecast. Correction effort might be 20 to 90 seconds per case. Monthly inference cost might depend on whether the agent makes two tool calls or twelve. Adoption might vary sharply between an experienced on-call engineer and someone covering an unfamiliar service. The range does not excuse vague thinking. It shows which assumptions drive the spread.

The aim is not to make everyone speak like a statistician. It is to stop a belief from acquiring more authority than its evidence. When AI projects break, hidden assumptions are often the starting point; a useful decision turns the most important assumptions into claims that can fail visibly.

Buy evidence before buying commitment

Under uncertainty, a pilot is not a miniature launch. It is a purchase of information.

That changes how the team scopes it. Instead of asking, “Can we build a working agent?” ask, “Which unresolved fact is valuable enough to test before we grant more authority?” A prototype that demonstrates the happy path may create excitement without reducing the uncertainty that matters.

For the incident agent, the team could buy evidence in several small ways:

• replay historical incidents to test classification and tool choice;
• run the agent in shadow mode beside the existing process;
• compare operator time with and without the suggested evidence packet;
• inject missing metadata, stale documentation, and failed tool responses;
• let the system draft a rerun plan without permission to execute it;
• expose one pipeline family before expanding to the whole platform.

Google’s SRE guidance on canarying releases provides a concrete version of this principle. A partial, time-limited deployment can reveal whether a change is acceptable while limiting the system exposed to failure. The important design work is not simply making the canary small. The team must choose a representative population, useful measures, a comparison, and a rollback response.

In teaching technical subjects, I repeatedly see learning deepen when a learner moves beyond complete instructions and has to investigate a real failure. The same principle applies to teams. Uncertainty becomes useful only when it is turned into a test that can surprise the people who designed it.

The best next experiment is rarely the most impressive demo. It is the smallest credible action that could change the decision.

Put the exit in writing

Technical teams are good at defining entry conditions: approved budget, assigned engineers, selected vendor, completed security review, signed-off requirements. Exit conditions are often weaker.

Write the stopping rules while the project is still emotionally cheap.

For a pilot, define four possible exits:

1. Expand because the evidence supports a wider boundary.
2. Revise because the goal still matters but an assumption or design failed.
3. Pause because a dependency must change before another test is useful.
4. Stop because the likely value no longer justifies the cost or exposure.

Each needs a trigger. “We will stop if it does not work” is not a trigger. “We will not continue after four weeks if total operator effort, including verification and correction, is no better than the baseline” is much stronger. So is “write access remains disabled if any unsafe action cannot be detected and reversed through the documented recovery path.”

Stopping rules protect the team from two opposite failures. The first is attachment: every disappointing result creates a reason for one more extension. The second is impatience: an early setback kills an idea before the intended uncertainty was tested. Pre-agreed boundaries do not remove judgment, but they make it harder to rewrite the standard around the result someone wants.

When a larger initiative is already moving, the navigation review for course-correcting AI projects provides the next layer. The uncertainty contract governs the current commitment. A navigation review asks whether the destination and route still deserve it.

AI systems make one success dangerously persuasive

Traditional software decisions also contain uncertainty, but current AI systems add a particular temptation: a fluent successful run feels like stronger evidence than it is.

An agent can choose the right tool once and the wrong one on the next attempt. A retrieval system can answer ten common questions well while failing on a rare policy exception. A model upgrade can improve average quality while changing latency, cost, tone, or behavior on a small group of important cases. Tool permissions, context length, prompts, model routing, retries, and external service behavior all change the system the team is actually deciding about.

Anthropic’s 2026 guide to evaluating AI agents recommends repeated trials because agent outputs vary, balanced cases where behavior should and should not occur, and test environments that do not quietly contaminate results. That is decision guidance as much as evaluation guidance. A team should not ask only whether the system can succeed. It should ask how consistently it succeeds, under which conditions, and what happens after failure.

The uncertainty contract should therefore distinguish at least three kinds of evidence:

• Capability evidence: the system can complete the task under defined conditions.
• Reliability evidence: it completes representative tasks consistently enough for the intended consequence.
• Operating evidence: people can detect, correct, recover from, and learn from failures at an acceptable cost.

A demo usually provides the first. Production authority requires all three. The gap between them is where human approval, observability, access limits, evaluation, fallback behavior, and incident response belong.

Review the decision before you review the result

Suppose the read-only incident pilot performs well. The team may congratulate itself on a good decision. Suppose a rare alert causes an expensive delay. The same people may decide that approval was reckless.

Neither conclusion is necessarily fair.

Outcome bias appears when knowledge of success or failure changes how people judge the quality of the earlier choice. A preregistered replication of the classic outcome-bias experiment found that participants rated otherwise identical decisions more favorably when the outcomes were successful. Technical organizations are not immune. A weak process can get lucky, and a strong process can encounter an unlikely bad result.

Separate two reviews.

The decision review happens after commitment but before the outcome is known. It asks:

• Was the decision framed clearly?
• Were credible alternatives considered?
• Did the action match the consequence and reversibility?
• Were material uncertainties visible?
• Was the next evidence capable of changing the choice?
• Were decision rights and boundaries explicit?

The outcome review happens when the agreed evidence arrives. It asks:

• What occurred, and how does it compare with the expected range?
• Which assumptions survived or failed?
• Which signal was missing, misleading, or ignored?
• What should expand, change, pause, or stop?
• What should the next decision process preserve or improve?

This separation does not excuse poor outcomes. Teams remain accountable for monitoring, recovery, and learning. It does prevent luck from becoming proof of wisdom and surprise from automatically becoming proof of negligence.

Keep the record alive without turning it into paperwork

A decision record should preserve the original reasoning, not rewrite history until the choice looks inevitable.

Microsoft’s April 2026 Azure Well-Architected guidance on architecture decision records recommends capturing context, alternatives, tradeoffs, confidence, and status. It also recommends an append-only approach: when a decision changes, create a record that supersedes the earlier one instead of editing the past.

The uncertainty contract can work the same way. Keep the original page short. When evidence changes the decision, add a dated entry:

Shadow mode reduced evidence-gathering time, but operator corrections were concentrated in incidents with incomplete lineage. Read-only access continues. Expansion is deferred until lineage coverage improves and the affected cases are added to evaluation.

That note is more useful than a polished retrospective written months later. It shows what changed, why authority did not expand, and what evidence would reopen the choice.

Not every technical decision needs this treatment. Use it for choices that are expensive, consequential, disputed, difficult to reverse, or likely to be revisited. A team that records everything will stop reading. A team that records nothing will repeat arguments and forget why important boundaries exist.

A 30-minute uncertainty review

The framework can fit into one focused meeting.

Spend five minutes stating the decision and intended change. Spend five on the cost of error and cost of delay. Spend ten on the two or three uncertainties most capable of changing the choice. Spend five selecting the smallest credible evidence purchase. Use the final five to define authority, exit conditions, owner, and review date.

End with a sentence that a person outside the meeting can understand:

We are approving a four-week, read-only pilot for two pipeline families because replay testing suggests useful triage capability, but production reliability and correction effort remain uncertain. Expansion requires representative shadow-mode results, acceptable operator effort, and a tested recovery path.

If the team cannot write that sentence, it may be debating technology before it has defined the decision.

For alternatives that remain too similar, use the variation matrix for better AI decisions to change the frame, constraint, evidence, viewpoint, mechanism, or failure condition. The uncertainty contract then turns the strongest options into a bounded commitment.

The useful decision is one the team can revisit

Decision-making under uncertainty is not the art of sounding sure. It is the discipline of making the uncertainty legible.

A strong technical team can explain what it chose, what it expected, which assumptions mattered, how much exposure it accepted, what evidence it planned to buy, and what would make it change direction. Later, it can review the reasoning without pretending the outcome was knowable in advance.

That practice will not make every decision successful. It will do something more realistic: help the team move before all the evidence exists, limit the damage when an assumption fails, and keep learning available after the result arrives.