Imagine an AI that meticulously follows over 80 learned rules, analyzes every detail, and refuses every underhanded tactic — yet still loses a crucial deal. In today’s fast-paced business world, sheer diligence isn’t everything. Sometimes, less is more, and prioritization beats volume. Welcome to the real-world insights from the Firmulate experiment, where four advanced AI models faced a simulated crisis-ridden week, revealing surprising truths about performance and trust.
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The Experiment: Putting AI to the Test
At the heart of this groundbreaking test was a simulated small software company, complete with 13 synthetic employees, real money mechanics, and a public cash countdown. Over a typical week, each of four AI models was tasked with managing the company, making critical decisions amid customers, crises, and temptations to cheat. Every decision was carefully recorded and auditable, ensuring transparency and replicability.
Products Worth Considering
Bose TV Speaker (2nd Gen) Compact Soundbar with Bluetooth and HDMI
Shark Rocket Pet Plus Corded Stick Vacuum, Lightweight, Bordeaux, HV322
Philips Hue Festavia 130-Foot Indoor/Outdoor Smart String Lights, 500 LEDs
As an affiliate, we earn on qualifying purchases.
Key Findings: Diligence Doesn’t Guarantee Success
All four models demonstrated remarkable vigilance: they identified every crisis and refused every manipulation attempt. They faced social engineering tricks, including fake CEO messages and reporter tricks, and refused them all. The Kimi K3 model notably provided the clearest on-record reasoning, citing suspicion of impersonation for refusing a questionable request.
However, despite their diligence, only two of the four models actually closed a lucrative deal, worth €55,000. The other two, despite excellent analysis, left the deal on the table. Why? Because they failed to escalate critical information stored deep within the company’s files, information that was only accessible through deeper document referencing.
The Hidden Weakness: Focus on the Critical
Interestingly, the decisive weakness wasn’t in the obvious crises but buried two document references deep in the company’s files. When the models that read these files effectively accessed the critical background, they secured the full-deal at a full €4,583 monthly recurring revenue (MRR). This underscores a fundamental lesson: thoroughness alone isn’t enough; prioritization and focus on what truly matters are key.
Implications for AI and Business
This experiment highlights a crucial insight for businesses considering AI: volume of effort doesn’t equate to impact. Diligence must be paired with strategic prioritization. An AI that follows every rule meticulously but neglects to focus on the decisive details risks leaving significant value unclaimed.
Real-World Testing: The Live Company
Firmulate’s live company, running in real time with 13 synthetic employees and real money mechanics, demonstrates how these insights matter practically. With a burn rate of €105k/month against €2.3k MRR, every decision counts. Every workday, models are versioned and tested, providing a clear view of what works and what doesn’t, accessible at firmulate.com/live.
Learning from the Models: What Matters Most
The most thorough participant in the experiment, Opus 4.8, with over 80 learned rules and deep analyses, scored the lowest in closing the deal. The same weakness appeared across all four models, weaker in some than others: a tendency to leave critical insights unexploited, often writing attempts into a locked department rather than escalating them properly. This reveals that exhaustive rule-following can be a distraction if it doesn’t align with strategic decision-making.
What Business Leaders Should Take Away
In deploying AI, your focus shouldn’t merely be on how much effort it exerts, but rather on how effectively it prioritizes and acts on the most impactful information. The experiment at Firmulate shows that even the best models — those that diligently spot crises and refuse manipulations — can stumble at closing deals if they overlook crucial background data or fail to escalate appropriately.
Next Steps: Wargaming Your AI Workforce
To prepare your AI systems for real-world challenges, consider running validation exercises similar to this experiment. Firmulate offers a platform to simulate your operational week in a safe, controlled environment, ensuring your AI can navigate crises, resist manipulations, and focus on what truly closes deals. Visit firmulate.com/pilot.html to learn more.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
