How to Spot AI Vendor Omissions Before You Buy

The most dangerous AI buying decision is not always based on a false statement.

Often, the statement is technically accurate. The product can summarize documents. The agent can call tools. The platform has a permissions model. The model has been evaluated. The roadmap includes audit features. The vendor has enterprise customers. The pilot looks promising.

Each sentence may be true. The problem is what the sentence leaves untested.

AI due diligence has to deal with this kind of partial truth. A buyer can ask a direct question and receive an answer that sounds complete while the important gap remains outside the frame. The demo shows the best workflow, not the awkward one. The accuracy number covers the vendor’s examples, not your documents. The roadmap says a control is coming, not that it exists today. The security answer describes the platform, not the connector your team will actually use.

This is why serious AI buying should inspect silence as carefully as claims.

I do not mean that every vendor is trying to deceive buyers. Most vendors are trying to present their product well, which is normal. Internal teams do the same thing when they pitch a pilot to leadership. The responsibility of a buyer, technical leader, or governance team is to turn a persuasive story into a testable decision.

The useful question is not only, “Is this claim true?” It is also, “What would we need to know for this claim to matter in our environment?”

The Silence Map

Before a team signs a meaningful AI contract or expands a pilot, I would want a simple artifact on the table: a silence map.

A silence map is a list of areas where the story is thinner than the decision requires. It does not accuse anyone of lying. It points to missing evidence.

This table is not a substitute for legal, security, procurement, or architecture review. It is a way to make those reviews sharper. Instead of asking broad questions like “Is the tool secure?” or “Does it work?”, the team can ask what evidence is missing from the specific buying story.

That matters because AI products often touch more than one layer of the organization at once: data, identity, workflow, model behavior, cost, compliance, user trust, and support. If one layer is underexplained, the purchase may still look good in a meeting and become difficult after launch.

Availability is not the same as capability

One of the easiest places for omission to hide is the difference between what exists now and what is expected soon.

This shows up constantly in AI conversations. A vendor may say audit logs are supported, but the logs may cover user prompts and outputs, not retrieved documents, tool calls, model versions, or approval decisions. A product may support human review, but only through a manual export that does not fit the workflow. A platform may have a connector for a system you use, but the connector may not preserve your permission model. An agent may support tool calling, but not the step limits, rollback controls, or approval gates your risk level requires.

These are not small details. They decide whether the product can be governed.

The same problem appears with internal AI pilots. A team says the assistant “has citations” because the interface shows links. But do those citations support the answer? Are they permissioned correctly? Are they versioned? Can a reviewer reconstruct what the model saw when it answered? If the answer is unclear, the citation feature may be more decorative than evidentiary.

The FTC’s 2024 announcement on deceptive AI claims and schemes and the SEC’s 2024 action against firms making false or misleading AI statements are reminders that AI marketing language is no longer a harmless layer around the product. Claims about AI capability, testing, replacement of professional work, or use of advanced models need support.

For buyers, the practical lesson is simple: separate four states.

• Available now: usable in your environment without special engineering.
• Available with configuration: usable after defined setup, data access, and testing.
• Available through services: possible, but dependent on custom implementation.
• Planned: not part of the buying decision unless the contract makes it real.

This discipline protects both sides. A good vendor should prefer a buyer who understands what is being purchased. A good buyer should prefer a vendor who can say, “That part is not ready yet,” before the contract turns optimism into conflict.

Ask the same question in different rooms

AI systems cut across teams, so one answer is rarely enough.

The product team may understand the user interface. The security team may understand data boundaries. The solutions engineer may understand integrations. The account executive may understand the commercial promise. The customer success team may know what happens after rollout. A reference customer may know which parts were harder than expected.

If every question is answered in one polished meeting, the buyer may only hear the aligned version of the story.

This does not mean turning due diligence into an interrogation. It means changing the structure of the conversation. Ask product questions with product people. Ask implementation questions with implementation people. Ask data and security questions with technical people. Ask support questions with the team that handles support. Ask existing customers what surprised them after purchase.

For an AI document assistant, ask several people the same practical question: “What happens when the answer is unsupported by the retrieved documents?”

The product answer may mention citations or refusal behavior. The engineering answer may describe retrieval thresholds, prompt instructions, or reranking. The support answer may describe how users report bad answers. The security answer may explain whether logs contain enough context for investigation. The customer answer may reveal whether users actually trust the refusal path.

Conflicting answers are not always proof of a bad product. They are evidence that the operating model needs more work before the buyer depends on it.

This is where the existing DataTweets note on testing AI vendor claims before buying connects naturally. Testing a claim is not only a technical exercise. It is an organizational exercise. The buyer needs to learn whether the vendor’s story survives contact with the people who will implement, govern, support, and use the system.

A perfect story should make you slow down

Every serious technology decision has tradeoffs.

If the AI buying story has none, something is probably missing. The system is accurate, secure, inexpensive, easy to integrate, fully governed, loved by users, flexible for every workflow, and ready to scale. That may sound comforting, but it gives a buyer nothing to inspect.

Real products have boundaries. Real organizations have constraints. Real AI systems have failure modes.

This is especially true for generative AI. NIST’s AI Risk Management Framework frames trustworthy AI as something organizations incorporate into design, development, use, and evaluation. NIST’s Generative AI Profile is even more direct about lifecycle risk: generative AI risks can appear during design, deployment, operation, and decommissioning. In buying language, that means the decision is not finished when the demo works.

A vendor with a mature product should be able to discuss limits without losing credibility:

• Which tasks are poor fits for the product?
• Which data conditions reduce quality?
• Which integrations usually take longer than buyers expect?
• Which controls are strong today, and which are still maturing?
• Which use cases require human approval?
• Which customer profiles are not a good match?

These answers are valuable. They help the buyer place the product in the right workflow, budget for the hidden work, and avoid using a tool where it should not be trusted.

The same principle applies internally. If a team pitching an AI pilot cannot name the risks, the pilot is not ready for expansion. The risk list does not need to be dramatic. It needs to be honest enough to guide decisions.

The omissions are different for agents

Agentic systems raise the cost of weak due diligence because they do not only produce text. They may plan steps, call tools, retrieve documents, query databases, open tickets, draft code, update records, or trigger workflows.

That changes the due-diligence questions.

For a chatbot, a wrong answer may mislead a user. For an agent, a wrong action may change a system. The more authority the agent has, the more important it becomes to ask what the vendor has not yet explained.

OWASP’s Top 10 for LLM and GenAI Applications includes risks such as prompt injection, sensitive information disclosure, improper output handling, excessive agency, vector and embedding weaknesses, misinformation, and unbounded consumption. Those categories are useful during buying because they expose the places where a polished demo can stay quiet.

For an agent product, silence around these areas should trigger follow-up:

• Tool authority: Which tools can the agent call, and with what permissions?
• Stopping rules: What prevents repeated tool calls, loops, or runaway usage?
• Approval gates: Which actions require a human before execution?
• Traceability: Can the buyer see the agent’s steps, tool inputs, tool outputs, and reasons?
• Prompt injection: How does the system treat untrusted content from web pages, documents, tickets, emails, or repositories?
• Rollback: What happens if the agent takes an action that should not have happened?
• Version changes: How are prompt, model, connector, and tool-schema changes tested before release?

These questions are not only security questions. They are product, legal, reliability, and operations questions. A vendor may have a strong model and still have weak boundaries around action.

The EU’s AI Act overview also points in this direction. It describes risk-based rules for developers and deployers, strict obligations for high-risk systems, logging and documentation expectations, human oversight, robustness, cybersecurity, and transparency duties. Even outside Europe, the direction is useful for buyers: the more consequential the system, the more evidence the decision requires.

Due diligence should produce a decision record

The output of AI due diligence should not be a vague feeling that the vendor seemed strong. It should be a decision record.

That record can be short. It should answer:

• What workflow are we buying for?
• What claim did we test ourselves?
• What evidence supports the decision?
• What remains unproven?
• Which controls are required before broader rollout?
• What costs are included and excluded?
• Which assumptions should be revisited after 30, 60, or 90 days?
• What would make us pause, renegotiate, or stop?

The “unproven” section is the important part. It keeps partial truth from turning into invisible risk.

For example, a team might decide to buy an AI support assistant after a narrow pilot. The record could say that the tool improved first-draft speed on policy-heavy tickets, but quality still depends on document freshness, escalation design, and reviewer training. It could say the tool should remain read-only for the first phase. It could say that expansion requires citation-quality monitoring, permission testing, and a cost-per-resolution review at real volume.

That is a much stronger decision than “the pilot was successful.”

It also helps with future lock-in. In Avoid AI Platform Lock-In Before It Becomes Policy, I argued that teams need to preserve a plan for change. A decision record supports that habit. It explains why the team chose the tool, which assumptions were accepted, and what should happen when those assumptions change.

Internal buyers need enough technical judgment to hear silence

A buyer cannot notice every omission if nobody on the buyer’s side understands the system.

This does not mean every executive needs to become an AI engineer. It does mean the organization needs enough internal capability to ask grounded questions. Someone should understand how retrieval can fail. Someone should understand what tool permissions mean. Someone should understand why a single accuracy score is not enough. Someone should know that a roadmap item is not the same as an operational control.

This is a management problem as much as a technical one. If the business team evaluates only the workflow pain, the vendor’s technical story may go untested. If the technical team evaluates only architecture, the business value may remain vague. If security joins only at the end, important controls may be treated as delays instead of requirements.

Good AI due diligence brings these perspectives together early:

• Business owns the problem and success metric.
• Technical teams test feasibility, integration, evaluation, and failure modes.
• Security and legal inspect data, access, retention, auditability, and regulated use.
• Finance checks the cost model against realistic adoption.
• Users test whether the workflow actually improves.
• Leadership decides which uncertainty is acceptable.

The goal is not consensus for its own sake. The goal is a decision that can survive after the meeting, after the pilot, and after the first unexpected failure.

This is closely related to when AI projects break, look for hidden assumptions. Vendor omissions often become internal assumptions. If nobody writes them down, they become part of the architecture by accident.

The goal is disciplined trust

It is easy to turn this topic into suspicion. That would be a weak ending.

The point of due diligence is not to assume every AI vendor is hiding something. The point is to respect the fact that every buying story is incomplete until it is tested against your data, your users, your controls, your cost structure, and your risk tolerance.

AI makes omission more expensive because the surface area is larger. A product may affect knowledge access, employee decisions, customer communication, software delivery, analytics, compliance, and operational workflows. A small missing detail in a demo can become a large operating problem later.

Disciplined trust has a different posture. It does not reject the product because the story has gaps. It asks for the evidence needed to close the gaps. It does not punish a vendor for naming limits. It rewards the vendor that can explain them clearly. It does not treat governance as paperwork after the decision. It uses governance to decide what evidence is required before trust expands.

Good AI buying is not about catching someone in a lie. It is about refusing to make a serious decision from partial evidence.

Ask what is available now. Ask what has been tested on your data. Ask which controls are real, which are manual, and which are planned. Ask where the system fails. Ask what the vendor would not recommend using it for. Ask what changes if the workflow becomes higher volume, higher risk, or more regulated. Ask who owns the system after the signature.

Then write down the answers and the silences.

The silences are not always deal breakers. Sometimes they are normal uncertainties for a pilot. Sometimes they are implementation work. Sometimes they are contract issues. Sometimes they are signs that the product is not ready for the job you want it to do.

The value of due diligence is knowing the difference before the organization is committed.