Nobody posts about the incidents. They are the most useful part. Here are three patterns I have lived through, described generically so the lesson survives.
1. The confident wrong move
What happened: an agent proposed a large change on thin data, with a perfectly reasonable explanation. The explanation was convincing. The data was not.
What we changed: step limits (lesson 3) so the move was capped, and a data-sufficiency check before the model ran. Fewer than N signals means the agent may suggest but not act.
2. The drift nobody noticed
What happened: after an upstream data source changed its format, the agent's decisions slowly got worse over two weeks. No single decision looked wrong.
What we changed: guardrail-hit rate on a dashboard with an alert. Drift shows up as rising hits long before it shows up in revenue. Plus the eval suite (lesson 4) runs nightly against fresh cases, not just at release.
3. The tool it should not have had
What happened: an agent used a tool for a purpose nobody intended, because the tool was available and the task was ambiguous.
What we changed: minimum tool scope per agent (lesson 2), and a tool-use log reviewed weekly. Every tool call is a decision too.
The pattern
In all three, the model was not the problem. The missing piece was always a boring control around it. That is the whole playbook in one sentence: production agentic AI is about 20% model and 80% trust design.
What next
If you want this applied to your company, that is what I do: AI safety reviews, AI strategy, and mentoring the engineers who have to build it. Book a call.