Should I Trust AI?
The useful answer is never yes or no about the category. It is whether this system fits this job under checks you can actually run.
The retail operations lead pointed at the vendor slide. One tile said “AI demand forecast.” The next said “AI customer assistant.” Same logo, same demo smile. Her question was the one every steering pack eventually asks: can we trust the AI?
The immature answer treats that as a single verdict on a product family. The mature one notices that a forecast wrong by twelve percent and an assistant inventing a return window are different failures with different owners. The noun on the slide does not distinguish them.
“Should I trust AI?” is the wrong unit of analysis. That gap is where this piece lives.
Essential Problems, Without the Scare Clock
The first problem is not philosophy. It is whether the system does the job it was commissioned for under the conditions you will actually meet. In plain terms: did anyone confirm it against the intended use, and does it keep behaving under the conditions you named? Standards language calls those validity for intended use and reliability over time. If those words are new in this context, the one-sentence versions worth keeping are: validity asks “right for this job?”; reliability asks “still right next month?” Fluent output is not that confirmation.
Language models make the gap vivid. They can produce confident answers that are not true. Recent research argues this is not only messy training data. Common accuracy-style evaluations reward guessing when uncertain more than they reward an honest abstention.
Hallucinations persist partly because scoreboards still pay for bluffing. A model that sounds sure is not, by itself, a model that checked.
The second problem sits on the human side. People favour automated recommendations over their own judgement. Practitioners call that automation bias: vigilance drops, and the review that would catch a wrong cue never quite happens. Omission errors miss what the system never flagged. Commission errors act on advice that was simply wrong.
High accuracy can deepen the trap. Trust rises, vigilance falls, and a fluent paragraph is harder to challenge than a red warning banner.
Oversight is not a checkbox. “We have a human in the loop” fails when verification is more expensive than accepting the draft. The loop becomes a signature, not a check.
Agency is the third problem. When a model can call tools, write to systems, or sit in a pipeline that publishes without a named stop, the blast radius is no longer “a wrong sentence in a chat.” Application security catalogs separate prompt injection, excessive agency, and misinformation: untrusted input steering behaviour, too much autonomy without controls, and plausible falsehoods consumed as fact. Excessive agency, in one line, is granting the model room to act beyond what a named person is ready to own. You do not need a cinematic breach. A silent wrong purchase order or a published policy paragraph is enough.
Underneath all three sits accountability. Someone must be able to say what the system was for, what it did, and who owns the outcome when it is wrong. If that seat is empty, “we used AI” is a shrug, not a decision.
The Umbrella Problem — “AI” Is Too Broad to Trust
Official definitions already refuse the single-agent fantasy. An AI system, in the OECD framing, is a machine-based system that infers from input how to generate predictions, content, recommendations, or decisions that can influence environments, with autonomy and adaptiveness that vary after deployment. ISO language is aligned: an engineered system that generates content, forecasts, recommendations, or decisions for human-defined objectives.
That is an umbrella, not a colleague. Predictions and drafted content are different jobs. A recommendation that ranks SKUs is not a decision that fires a warehouse pick. Collapsing them into one trust question is a category error dressed as prudence.
Walk the retail pair under one procurement package. The demand forecast sits behind purchase planning. A bad week means overstock in the back room or empty shelves on the floor. Verification is historical error, seasonality holdouts, and a named planner who will not rubber-stamp the chart because the dashboard looked confident. The customer assistant drafts return emails. A bad week means the wrong window promised in writing: thirty days when policy says fourteen. Verification is a policy lookup and a human send, not a vibe check on tone.
Same vendor label. Different stakes, different checks, different owners. The forecast can be “good enough to propose” while the assistant must never auto-send. Those are compatible truths under one logo and incompatible answers to “do we trust the AI?”
Risk-based regulation has already made a similar cut in another vocabulary. Obligation intensity tracks use (hiring tools, credit, safety components), not whether a brochure said “AI.” Delivery leads can borrow the instinct without becoming compliance officers: name the use before you name the trust.
How to Use It Without Pretending
Trustworthiness characteristics do not arrive as a complete badge set. Valid and reliable, safe, secure, accountable and transparent, explainable, privacy-enhanced, fair with harmful bias managed. They trade off, and some matter more in a given context than others. The practical move is not to recite the list. It is to decide whether this system is appropriate for this purpose under a verification path you can fund.
Four slots are enough for most delivery conversations.
- Job — Name the output in verbs and nouns the business already uses: “weekly demand forecast for category X,” not “AI insights.” If you cannot name the job, you are not ready to trust the tool.
- Stakes — What breaks if the output is confidently wrong? Money, customer promise, safety, reputation, or only an internal draft. Stakes set how expensive verification must be.
- Verification — The check a competent person can finish in bounded time: reconcile to source policy, compare to last quarter’s actuals, run the golden cases. If verification is harder than doing the work, you have automation theatre.
- Owner — Who signs when it is wrong? A named role, not “the model” and not “the team” in the abstract. Tool access without an owner is how chat answers become system actions.
Run the retail pair through the same slots.
Forecast: job is purchase input; stakes are inventory cash and service level; verification is error against actuals on a schedule the planner believes; owner is planning. Assistant: job is draft email; stakes are a customer promise in writing; verification is policy match before send; owner is customer service. One may be trustworthy enough to use daily as a proposal engine. The other may be a drafting aid that never touches “send” without a person. Both can sit under the same logo without sharing a trust answer.
Granted, some demos really are impressive. The concession does not licence a blank cheque. Use the assistant to draft; keep the send human. Use the forecast to propose; keep the purchase order human until the error band is known. Abstention and escalation are features when the evaluation culture outside your walls still rewards guessing.
A small honesty check helps. If the only metric on the pilot slide is “time saved,” ask what error rate you are buying with that time. Speed without a verification budget is how commission errors get industrialised.
The shorter companion on this site is the RADAR field note Know what AI is actually good at — calibrated trust for a single task, not a verdict on the category.
What Trust Looks Like After the Demo
You do not owe the vendor a category yes. You owe the work a scoped decision: this job, these stakes, this check, this owner.
The forecast earns trust when its error is measured and someone will still open the spreadsheet. The assistant earns trust when it never ships a promise the policy cannot keep. The label on the slide earns nothing on its own.
Trust the use you can verify. Distrust the umbrella that pretends one verdict covers both. Keep the owner in the chair when the model is wrong.