When AI Projects Break, Look for Hidden Assumptions

Many technology failures feel sudden only because the assumptions behind them were invisible for too long.

A dashboard becomes wrong after a schema change. A machine learning model performs well in a notebook but disappoints in production. A RAG assistant answers confidently from outdated documents. An AI agent works in a demo, then runs into rate limits, loops through tools, or exposes a permission problem when real users arrive. A cloud bill grows quietly until finance asks why the experiment is now an operating expense.

The visible event may arrive in one afternoon, but the real problem usually started much earlier. Someone assumed the data would stay clean. Someone assumed users would ask questions the way test users did. Someone assumed the model provider would behave consistently. Someone assumed the prompt was stable, the retrieval index was fresh, the human approval step was obvious, or the cost curve would remain small because the pilot was small.

Then reality changes, and the team calls it a crisis.

I think a better way to read many AI, data, and software problems is this: the crisis is often the moment when an old assumption can no longer hide. That is uncomfortable, but it is also useful. It gives teams a chance to stop asking only, “Who caused this incident?” and start asking, “What belief did our system depend on, and how did we forget to test it?”

That question matters more in 2026 because technical systems are becoming more capable and less transparent at the same time. AI applications now depend on models, prompts, retrieval pipelines, tool calls, orchestration frameworks, policy instructions, model routing, human review, logs, evaluations, and governance rules. More moving parts means more assumptions. More assumptions means more ways for a team to confuse a good demo with a dependable system.

The incident is rarely the beginning

When a production issue appears, the first response is usually tactical. Restore the service. Roll back the deploy. Patch the prompt. Disable the agent. Rebuild the index. Add a rate limit. Send the update to stakeholders.

That work is necessary. People need the system to recover.

But after recovery, many teams move on too quickly. They treat the incident as a one-time interruption instead of a late signal from a longer pattern. The better postmortem does not stop at the broken step. It traces the hidden chain of assumptions that made the failure possible.

In an AI support assistant, the obvious failure might be an incorrect answer. The deeper assumptions might include:

• The knowledge base is current enough to answer customer questions.
• Retrieval quality from the test set represents real user traffic.
• The model will refuse unsupported answers when context is weak.
• The answer format will stay stable after prompt edits.
• Human reviewers know which answers require escalation.
• Logs capture enough detail to understand what happened later.

Any one of these assumptions may be reasonable in a small pilot. The problem begins when the assumption graduates into production without becoming visible, owned, and tested.

This is not only an AI problem. It is a normal software problem with a new surface area. Traditional systems also depend on assumptions about APIs, schemas, latency, permissions, user behavior, and team ownership. AI adds more probabilistic behavior, more dependency on context, and more temptation to trust fluent output.

The failure looks sudden because the system crossed a threshold. The risk was already there.

AI makes hidden assumptions easier to miss

Generative AI can compress time between idea and prototype. That is genuinely valuable. A team can test a use case, build a workflow, generate code, summarize documents, classify text, or connect a model to tools much faster than before.

Speed, however, can hide unfinished thinking.

A prototype often answers the questions that are easiest to ask: Can this model do the task once? Can we connect to the API? Can we retrieve a relevant document? Can an agent call the tool? Can a user see something impressive on the screen?

Production asks different questions: Does it work across messy inputs? Does it fail safely? Can we measure quality? Can we explain cost? Can we trace decisions? Can we revoke access? Can we reproduce failures? Can we change the prompt without breaking old cases? Can we decide when normal software is better than an LLM?

That gap is where hidden assumptions live.

Datadog’s 2026 State of AI Engineering describes modern production AI as model fleets, tool calls, long prompts, retries, capacity limits, cost control, and distributed-service debugging. It also reports that more than 70 percent of organizations in its telemetry use three or more models. That is a very different world from a single prompt in a notebook.

LangChain’s 2026 State of Agent Engineering shows the same shift from curiosity to operating reality. Many respondents reported agents in production, while quality remained one of the top barriers. Observability adoption was high because teams need to see what agents are doing, not just admire the final answer.

These reports point to the same lesson: AI projects are no longer only about whether a model is impressive. They are about whether the surrounding system can carry the assumptions the team is making.

A good team writes down what must stay true

One practical habit can prevent many avoidable surprises: write down what must stay true for the system to be useful and safe.

This sounds simple, but many teams skip it because it feels slower than building. In reality, it saves time. An assumption that remains informal cannot be monitored, tested, challenged, or assigned to an owner. It becomes part of the atmosphere. Everyone breathes it in, but nobody inspects it.

For an AI document assistant, the assumption list might include:

• Source documents are approved for the user’s role.
• Documents are refreshed within a defined time window.
• Each answer includes evidence from retrieved context.
• The model returns structured output for downstream systems.
• Unsupported questions are routed to a human or answered with a refusal.
• Token usage and latency stay within agreed limits.
• Prompt and model changes run through regression tests.

For a text-to-SQL assistant, the assumptions are different:

• The model can only query approved tables or views.
• Generated SQL is checked before execution.
• Queries are read-only unless a human explicitly approves a write action.
• Metrics have documented definitions.
• User intent can be mapped to known business concepts.
• Query failures are logged with enough context to improve the system.

For an internal coding agent, the assumptions change again:

• The agent can only access repositories and tools needed for the task.
• Tests are meaningful enough to catch common failures.
• Generated code is reviewed according to risk.
• Secrets cannot be read or printed.
• Changes to customer-facing or regulated systems require stronger human oversight.
• The team can reconstruct the agent’s actions after the fact.

The exact list depends on the workflow. The habit is the same: make the dependency visible before it becomes a surprise.

This is also a career skill. In How to build practical AI skills for today’s tech job market, I argued that practical AI skill is not just knowing the vocabulary. It is being able to build, test, explain, and improve a system. Surfacing assumptions is part of that work. It shows that you understand the difference between a demo and an operated product.

Evaluation turns belief into evidence

Many teams say a system “works” when they mean it worked on a few examples. That is not a crime; every project starts somewhere. The risk begins when a small impression becomes a production belief.

Evaluation is the discipline that keeps belief from hardening too early.

For LLM applications, evaluation should not be one vague score. It should match the failure modes that matter. A RAG system may need separate checks for retrieval relevance, answer faithfulness, citation quality, refusal behavior, and latency. An extraction system may need field-level accuracy, schema validation, and examples of partial failure. An agentic workflow may need traces, step counts, tool-call success rates, cost per completed task, and clear definitions of when the agent should stop.

The goal is not to create perfect metrics. The goal is to make the important assumptions testable.

If the team assumes retrieval quality is good, create a small dataset of realistic questions and expected evidence. If the team assumes a prompt change improved behavior, run old cases before celebrating. If the team assumes a cheaper model is good enough for one step, compare quality, latency, and cost on the task that step actually performs. If the team assumes users understand the output, watch real users work with it.

McKinsey’s 2025 State of AI survey reported that enterprise-wide bottom-line impact from AI remained limited for many organizations, even while adoption was widespread. One reason is that using AI and redesigning work around measurable value are not the same thing. The second requires evidence, workflow change, and operating discipline.

Evaluation is not bureaucracy. It is how a team earns the right to trust a system a little more.

Governance should protect learning, not freeze it

Some teams hear “governance” and imagine a committee that says no to everything. That version is not useful. If governance only slows teams down, people will route around it, especially when AI tools are easy to access with a credit card and a browser.

Good governance is different. It helps teams learn without pretending that every experiment is harmless.

The National Institute of Standards and Technology describes its AI Risk Management Framework as a way to improve how organizations incorporate trustworthiness considerations into the design, development, use, and evaluation of AI systems. That phrasing matters because it places risk management inside the work, not after the work.

In practice, governance should help answer questions like:

• What data is this system allowed to use?
• Which users can access which tools and outputs?
• What decisions require human approval?
• What evidence do we keep for audit, debugging, or user support?
• What happens when the model provider changes behavior?
• Which use cases are low-risk experiments, and which need formal review?

This does not mean every small prototype needs the same process as a regulated production system. The point is proportional control. A private summarizer over public documents does not need the same review as an agent that can modify customer records. A tool that drafts an email does not carry the same risk as a tool that approves a loan, changes code in production, or recommends medical action.

The hidden assumption to avoid is that all AI work has the same risk profile. It does not. Teams need enough structure to tell the difference.

Cost and latency are assumptions too

Technical teams often discuss quality and security first, but cost and latency can also break trust.

An AI workflow that feels useful with ten users may become too slow or too expensive with a thousand. A long system prompt may be acceptable in a prototype and wasteful in production. A multi-step agent may look intelligent in a demo and become unpredictable under real concurrency. A retry strategy may improve success rates during testing but create more load when a provider starts returning rate-limit errors.

These are not edge details. They shape the user experience and the business case.

Datadog’s report notes that production LLM systems now involve context engineering, prompt caching, model routing, rate limits, and backpressure. That is classic operational work. It also means teams should treat cost and latency assumptions as first-class requirements, not unpleasant surprises after launch.

A practical AI plan should define limits early:

• Maximum acceptable response time for the workflow.
• Maximum cost per completed task or per user session.
• Token budgets for prompts, retrieved context, and agent loops.
• Fallback behavior when a model is unavailable or too slow.
• Caching strategy for stable instructions or repeated context.
• Monitoring for spend, error rates, and unusual usage.

The right numbers depend on the product. A customer support assistant, internal research tool, code review helper, and document extraction pipeline will not share the same budget. But each one needs a budget. Without limits, the team is not managing a system; it is hoping the invoice stays reasonable.

The uncomfortable questions are usually the useful ones

The best way to prevent hidden assumptions is to make room for uncomfortable questions before launch.

Some of those questions are technical:

• What are the top five ways this system can be confidently wrong?
• Which dependency can change without our code changing?
• What will we log when a user says the answer was bad?
• How do we know whether retrieval failed or generation failed?
• What test would prove this prompt change made something worse?

Some are organizational:

• Who owns the system after the pilot?
• Who approves access to sensitive data?
• Who answers user complaints?
• Who pays for increased model usage?
• Who decides whether to pause the feature?

Some are product questions:

• What decision does this output support?
• What should the user do when confidence is low?
• What part of the workflow should remain human?
• What is the smallest useful version?
• What would make us decide not to use AI here?

These questions can feel negative in a meeting where everyone wants momentum. But ignoring them does not make the risk disappear. It only moves the risk into production, where the cost of discovery is higher.

Good technical leadership is not constant skepticism. It is disciplined curiosity. It asks enough questions to keep the team honest without killing useful work.

Make assumptions visible while the system is still small

It is easier to change a project when the team is still close to the beginning. A prompt can be rewritten. A database view can be narrowed. A permissions model can be designed. A test set can be created. A workflow can be simplified. A human approval step can be added. A risky use case can be postponed.

Later, every change has more weight. Users depend on the system. Stakeholders expect results. Data has accumulated. Integrations exist. Dashboards report numbers. People have built habits around the tool. The old assumption becomes part of the organization.

That is why early clarity matters.

Before an AI pilot moves toward production, I would want the team to answer a few plain questions:

• What are we assuming about the data?
• What are we assuming about users?
• What are we assuming about model behavior?
• What are we assuming about cost, latency, and scale?
• What are we assuming about human review?
• What are we assuming about legal, security, and privacy risk?
• What are we assuming about ownership after launch?

Then I would want each important assumption connected to evidence, monitoring, or a decision. Some assumptions can be tested with a dataset. Some need logs. Some need policy. Some need a human owner. Some need a clear reason why the team is accepting the risk for now.

The point is not to remove every uncertainty. That is impossible. The point is to avoid being surprised by assumptions the team could have named.

A calmer way to handle technical trouble

When something breaks, the easiest story is personal blame. Someone approved the wrong thing. Someone missed a warning. Someone changed a prompt. Someone trusted a model. Sometimes accountability is necessary, especially when people ignore known risks. But blame alone rarely improves the system.

The more useful response is to ask what the organization made easy and what it made hard.

Did it make it easy to ship demos and hard to fund evaluation? Did it make it easy to add models and hard to retire old ones? Did it make it easy to use sensitive data and hard to track permissions? Did it make it easy to launch agents and hard to observe their tool calls? Did it make it easy to promise value and hard to measure whether value appeared?

Those are system questions. They lead to better fixes.

A broken AI project may need a patch, but it may also need clearer data ownership, a smaller scope, better evals, stricter access controls, cost dashboards, prompt regression tests, human approval rules, or a decision to use normal software for parts of the workflow. Sometimes the best fix is not a smarter model. It is a better operating model.

That is the practical lesson I take from many technical failures: trouble often arrives when reality catches up with what a team has been assuming. The answer is not to become fearful or slow. The answer is to keep assumptions visible, test the ones that matter, and update the system before the gap becomes painful.

AI will keep changing. Models will improve. Frameworks will come and go. Context windows will grow. Agents will become more capable. But the basic engineering discipline does not disappear. Teams still need clear ownership, good data, explicit constraints, useful tests, logs, budgets, security boundaries, and honest conversations about risk.

When a project breaks, do not only look at the last change. Look for the belief that was allowed to become invisible.

That is where the real work begins.