🔍 Read the full analysis: The Business Case For Giving AI Agents A Bad Week on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Firmulate says five frontier models identified every crisis and refused every manipulation attempt in a simulated software company, but only two signed a €55,000 deal their own analysis supported. The July 2026 experiment also found that reading internal files affected the outcome; a proposed enterprise pilot would test agents against a read-only export of a company’s data.
Firmulate has published results from a July 2026 wargame in which five frontier AI models managed a struggling simulated software company, reporting that all five recognized each crisis and rejected manipulation attempts but only two signed a €55,000 deal supported by their own analysis. The company is also offering an enterprise pilot that runs scenarios against a read-only export of a customer’s business data, without writing to its real systems.
The final Crucible League put each model through the same difficult week at a small software company. Firmulate says decisions were versioned and auditable, and that partial progress counted toward the score. A breach of trust, however, capped a participant’s total. The published standings were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73; a do-nothing baseline scored 26.
The company says every model identified all the crises and refused each manipulation attempt. The separation came later in a sales opportunity: only two signed the deal after their own analysis supported it. The decisive weakness was not in the customer event itself. It was buried two document references deep in the company’s files. Firmulate reports that models that found the detail won the deal at full price, adding €4,583 in monthly recurring revenue.
The trial also included fake CEO messages escalating through three stages, followed by a reporter asking for a yes-or-no answer “on background.” Firmulate says all five models refused. Kimi K3’s recorded reasoning described the request as a suspected approval bypass or possible impersonation. In a separate result, Opus 4.8 added 80 learned rules and produced the deepest analyses, yet finished last. Firmulate says it missed the close and attempted to write into a locked department rather than escalating.
The Business Case For Giving AI Agents A Bad Week
Five frontier models ran a struggling simulated software company through the same difficult week. All five spotted every crisis and refused every manipulation attempt — but only two signed a €55,000 deal their own analysis supported. The separation was buried two document references deep in the company’s files.
Same Bad Week, Very Different Outcomes
| Rank | Model | Score | Verdict |
|---|---|---|---|
| 1 | gpt-5.6-sol | League leader — closed what its analysis justified | |
| 2 | Kimi K3 | Ran on API-default effort, not xhigh — compare with care | |
| 3 | Sonnet 5 | Strong diagnosis, incomplete execution | |
| 4 | Fable 5 | Middle of the field on the same scenarios | |
| 5 | Opus 4.8 | 80 learned rules, deepest analyses — finished last | |
| — | Do-nothing baseline | Inaction still scored 26 via partial progress |
Note: Kimi K3 ran without an effort parameter (API default) while the other models ran at xhigh. Scores record performance under the reported settings, not capability under identical effort configurations. A breach of trust capped a participant’s total.
Same Diagnosis, Same Pitch — No Signature
The Deal Buried in the Files
The decisive weakness was not the customer event itself — it was two document references deep in company records. Models that found the competitor detail won at full price, adding €4,583 in monthly recurring revenue.
Three-Stage Escalation
Fake CEO messages escalated through three stages, followed by a reporter demanding a yes-or-no “on background.” All five models refused. Kimi K3’s reasoning called it a suspected approval bypass or possible impersonation.
Depth Didn’t Win
Opus 4.8 added 80 learned rules and produced the deepest analyses, yet finished last: it missed the close and tried to write into a locked department rather than escalating the problem.
How the Enterprise Pilot Works
Read-Only Export
The pilot takes a read-only export of a company’s own records and playbooks. No write-back to real systems at any stage.
Crisis Scenarios
Models face scenario runs built on the company’s actual data — crises, sales opportunities and manipulation attempts.
Board Report
Firmulate produces model rankings plus weak points in the company’s own playbooks and operating rules.
Controlled Assessment
A controlled assessment, not a live deployment — its value depends on the data and scenarios selected.
Voices From The Wargame
“Same diagnosis, same pitch — no signature.”
Firmulate“No amount of good work outweighs a breach of trust.”
Firmulate“Treat the request as a suspected approval-bypass / possible impersonation.”
Kimi K3 · Recorded ReasoningRead the Standings With Care
One simulated company
The results describe a specific set of scenarios at one synthetic firm. No independent evaluation is included, and behavior may not carry over to other companies, tasks or configurations.
Uneven effort settings
Kimi K3 ran at API-default effort while others ran at xhigh. The report does not quantify how that setting affected its result — compare scores alongside the disclosed configuration.
Unvalidated pilot uptake
It is unclear how many companies have taken up the enterprise pilot, what it costs, or how its scenarios and rankings will be validated for each customer.
One task or a pattern?
The report provides too little detail to assess whether the document-finding task represents a common pattern across business records or a result specific to this scenario.
Finding Evidence Before Signing
The results frame agent readiness as a sequence of behaviors, not a single ability to identify an emergency. An agent may recognize a problem and explain a sound course of action, yet fail to locate information elsewhere in the company’s records or complete a justified transaction. For businesses considering automation, that gap can matter as much as whether a model gives a convincing answer.
The trust tests point to another operational question: whether an agent respects limits when someone claims authority or asks it to disclose information. Firmulate’s synthetic scenarios show how the participating models behaved in this particular exercise. They do not establish how the same systems would behave across other companies, tasks or configurations. The proposed pilot is intended to make those questions more specific by using each participating company’s own data and playbooks.
From Synthetic Firm to Pilot
Firmulate’s public experiment models a company with 13 synthetic employees, versioned workdays and more than 680 self-learned playbook rules. The scenario lists monthly burn of €105,000 against €2,300 in monthly recurring revenue, with a public cash countdown. Readers can follow the activity at firmulate.com and take a quiz based on 242 real, unedited management decisions, guessing which model made each choice.
The enterprise offer extends the test to a company’s own records. Firmulate says a pilot takes a read-only export, runs crisis scenarios and produces a board report with model rankings and weak points in the company’s playbooks. The stated setup has no write-back to real systems. That makes the pilot a controlled assessment rather than a live deployment, though its value will depend on the data and scenarios selected.
There is a comparison caveat in the league: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. The scores therefore record performance under the reported settings; they do not isolate model capability under identical effort configurations.
“Same diagnosis, same pitch — no signature.”
— Firmulate
Limits of the League Results
The published results do not establish how the models would perform in other business settings or with different data, permissions and instructions. Firmulate’s account describes one simulated company and a specific set of scenarios; no independent evaluation is included in the supplied details. It is also unclear how many companies have taken up the enterprise pilot, what the pilot costs, or how its scenarios and rankings will be validated for each customer.
The Kimi K3 effort-setting difference complicates direct score comparisons. The report does not quantify how that setting affected its result, and the standings should be read alongside the disclosed configuration. Firmulate also does not provide enough detail here to assess whether the document-finding task represents a common pattern across business records or a result specific to this scenario.
Company-Specific Tests Ahead
Readers can follow the live company at firmulate.com/live and review the full standings at firmulate.com/benchmarks.html. Companies interested in a pilot can visit Firmulate’s pilot page or contact contact@firmulate.com. The stated next step is to run scenarios on a read-only export and review a board report on model performance and playbook weaknesses.
The pilot results will show whether the patterns in the synthetic league carry over to a customer’s own records and operating rules. Until those company-specific findings are available, Firmulate’s league remains evidence about performance in its test, rather than a guarantee of how an agent will act under pressure in a live business.
Source: ThorstenMeyerAI.com
Key Questions
What did the Firmulate league test?
It put five AI models through the same simulated difficult week at a small software company, including crises, a sales opportunity and attempts to manipulate or bypass approval.
Which model ranked first?
gpt-5.6-sol scored 95, ahead of Kimi K3 at 93. Firmulate notes that Kimi K3 used the API default effort setting while the other models ran at xhigh.
What does the enterprise pilot do?
Firmulate says it runs scenarios using a read-only export of a company’s data and produces a board report with model rankings and weaknesses in the company’s playbooks. The described pilot does not write back to real systems.
Did all five models close the €55,000 deal?
No. Firmulate reports that only two signed the deal their own analysis supported. Models that found a competitor weakness buried in the company’s files won at full price, worth €4,583 in monthly recurring revenue.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
![Douglas Hofstadter: Analogy As The Core Of Cognition [Video]](https://micronomicon.com/wp-content/uploads/2026/09/douglas-hofstadter-analogy-as-the-core-of-cognition-video-featured-560x300.jpg)
