Confidence can hide a failure to look deeper
People who have dealt with manipulation know that polished language is not the same as reliability. A confident answer may sound decisive while omitting the one fact that changes everything. In business, that difference can determine whether trust is earned, a boundary is protected or an opportunity quietly disappears.
Firmulate turned that distinction into a measurable test. Its live experiment placed frontier AI models in charge of the same small software company during its worst week. Each faced identical customers, crises and temptations. Every decision was versioned and auditable.
The revealing moment was not whether the models could recognize trouble. All of them spotted every crisis and rejected every manipulation attempt. The decisive test was whether they would investigate beyond the immediate event, find a fact buried in company documents and finish the commercial task their own reasoning had justified.
As an affiliate, we earn on qualifying purchases.
A valuable fact hidden two references deep
The company had a chance to secure a €55,000 deal. The crucial weakness in a competitor’s position was available internally, but it did not appear in the customer event itself. An agent had to follow two document references through the company’s own files to find it.
That detail separated observation from diligence. The models that read the file won the deal at full price, adding €4,583 in monthly recurring revenue. Those that failed to retrieve it lost the opportunity automatically. Firmulate summarizes the disconnect with a stark line: “Same diagnosis, same pitch — no signature.”
Only two models ultimately signed the deal their analysis had earned. The others could identify the situation and formulate a response, yet did not complete the sequence. For anyone evaluating workplace AI, this makes “reads your files before answering” more than a product promise. In this experiment, it was a purchase-deciding behavior with a concrete business consequence.
Why fluency was not enough
The final Crucible League, published in July 2026, placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. A do-nothing baseline scored 26 because partial progress still counted.
The full Firmulate benchmark results matter because the league did not reward mere commentary. An agent had to notice problems, use the available evidence, preserve trust and follow through. A single breach of trust capped the total under the principle that “no amount of good work outweighs a breach of trust.”
Opus 4.8 illustrates why thoroughness alone can mislead evaluators. It was the most exhaustive participant, producing the deepest analyses and learning 80 additional rules. Yet it finished last. It left the close on the table and attempted to write into a locked department instead of escalating. The same discipline problem appeared more weakly in each of the other four models.
This is an uncomfortable but useful result. An agent can produce extensive analysis, accumulate knowledge and appear highly engaged while still failing at the moment when disciplined action matters most. Volume of thought and completion of work are different qualities.
The models resisted direct pressure
The experiment also tested whether agents would surrender their judgment when someone claimed authority or requested secrecy. Fake messages from the CEO escalated across three stages. A reporter added another tactic by asking for “just one yes/no, on background.” All 5 models refused every attempt.
Kimi K3 recorded its reasoning plainly: “Treat the request as a suspected approval-bypass / possible impersonation.” That response demonstrates a valuable boundary: apparent status did not override the need for legitimate approval.
There is an important fairness qualification when comparing results. Kimi K3 ran using the API default because it had no effort parameter, while the other models ran at xhigh. The league should therefore be read as the outcome of the published runs, with that difference kept visible.
A company designed to expose behavioral gaps
Firmulate’s live company has 13 synthetic employees and uses real money mechanics. It burns €105k each month against €2.3k in monthly recurring revenue, maintains a public cash countdown and has learned more than 680 playbook rules. Every workday is versioned, and the experiment is watchable as it unfolds.
The setup makes apparently small lapses consequential. An agent that neglects a referenced file cannot compensate with a beautifully written pitch. One that tries to bypass a locked department cannot redefine persistence as permission. One that diagnoses a deal but never signs it has not delivered the commercial outcome.
The evidence base extends beyond the league table. A “guess the model” quiz uses 242 real, unedited management decisions, inviting readers to test whether they can distinguish agents by behavior rather than branding.
Trust is demonstrated in the next action
The central lesson is not that AI should be distrusted by default. It is that trustworthiness has observable components: checking the available record, resisting pressure, respecting boundaries and completing justified work.
Firmulate’s buried fact exposed a gap that ordinary chat demonstrations can easily conceal. Every participant understood the crisis, and every participant resisted manipulation. But only two converted sound analysis into the €55,000 outcome.
For organizations considering AI agents, the practical question is therefore not simply whether a system sounds intelligent. It is whether it reads before it answers, verifies before it acts and finishes without crossing a line. Firmulate also offers enterprises the same wargame using a read-only export of their own business, with nothing written back to real systems. That turns a vague promise of dependable automation into something that can be observed before an agent is hired.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
