Artificial Intelligence already writes code, summarizes documents, answers emails, and even negotiates contracts.
But can it actually run a company from scratch for nearly a year and a half, making hundreds of strategic decisions that only show results weeks later?
That question, which sounds simple enough, hides one of the biggest challenges facing modern AI models: maintaining a coherent strategy over time, with incomplete information, a shifting market, and cash always on a razor’s edge. 💸
That’s exactly what researchers set out to test when they created CEO-Bench, a benchmark that puts AI models in the shoes of a real CEO, running a fictional startup called NovaMind for 500 simulated days.
The result?
Out of 14 models tested, most went bankrupt before reaching the finish line. Only three managed to end up with more money than they started with, and an unexpected contender nearly left everyone in the dust without using a single language model.
Let’s break down how this simulation works, who survived, and what it tells us about the current state of AI. 🤖
What is CEO-Bench and how does the simulation work
CEO-Bench was created to evaluate the ability of AI models to make long-term strategic decisions in complex, dynamic environments. The core idea seems simple on the surface, but it’s brutally difficult in practice: put an AI in charge of managing a fictional subscription software startup for 500 simulated days.
The company, NovaMind, starts with zero customers and one million dollars in the bank. Performance is measured by the cash remaining at the end of the simulation. And here’s the brutal part: if the balance drops below zero even once, the company goes under and the simulation ends right there. No second chances. 😬
To run the company, the agent doesn’t get a set of ready-made buttons. It controls everything through a Python API with 34 tools and a database with 19 tables. Instead of just issuing individual commands, the AI needs to write its own code, query the database with SQL, and build custom workflows from the results. In other words, it faces the same challenges a human CEO would, according to the researchers.
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
And there’s no shortage of things to decide: pricing and plans, ad spending across multiple channels, product quality and R&D investment, infrastructure capacity, customer support, and even multi-round negotiations with enterprise clients. On top of all that, there’s a simulated social network where the agent can read complaints, competitor news, economic trends, and even post its own messages.
The weight of time and uncertainty
What makes the task so difficult is precisely the combination of time and uncertainty. Decisions play out on realistic business timelines: revenue only comes in on billing dates, R&D projects take days to weeks to complete, and mistakes often don’t show up until much later, through cancellations or a damaged reputation. But costs hit immediately. The agent has to spend money whose return might take weeks to materialize.
A big chunk of the company’s state is hidden from the agent. It can’t directly see customer satisfaction, willingness to pay, or minimum quality expectations. All of that has to be inferred from noisy signals like cancellations, support tickets, or social media reactions. The simulation models 26 customer segments and individual customers, each with their own budget, price sensitivity, and expectations.
And the world doesn’t stand still. Competitors periodically raise quality expectations, preferences shift over time, and a simulated economic cycle affects demand and willingness to pay. Bottom line: the agent has to keep adjusting constantly, never letting its guard down.
One smart design choice by the researchers is worth highlighting. They opted for fixed, transparent rules instead of using a language model as the simulation’s judge. The goal was to avoid a weakness seen in similar tests, like Vending-Bench, where an AI-simulated supplier could reward the agent for unrealistic verbal promises. Here, what counts is concrete results, not smooth talk.
Who survived and who went under
Of the 14 AI models put to the test, the vast majority didn’t make it to day 500 with their books in the black. Almost all of them could generate valid commands and database queries, but none managed to maintain a coherent strategy over time. Many went bankrupt well before the simulation ended.
Only three models finished their best run above the initial capital of one million dollars:
- Claude Fable 5: an impressive $47.15 million 🥇
- Claude Opus 4.8: $27.8 million
- GPT-5.5: $21.3 million
Claude Fable 5 was the only model to finish above the starting capital in more than one run. But there’s an important detail: one of Fable 5’s runs was aborted because the model itself refused to continue, and in the other two, some requests fell back to Opus 4.8. Meanwhile, GPT-5.5 went bankrupt in two of its three attempts. So even the podium wasn’t free from stumbles. 😅
The surprise of the simple rule-based agent
But the most revealing finding from the entire study didn’t come from any of the big language models. An agent based on a simple heuristic, one that never calls a language model even once, came absurdly close to beating everyone.
This agent sets fixed prices, quotas, and plans, concentrates advertising and development on a small set of customer segments, and adjusts capacity based on recent usage. Simple as that. And yet it reached $15.76 million, beating every AI model except Fable 5, Opus 4.8, and GPT-5.5.
This result shook the community because it raises an uncomfortable question: for long-term strategic management tasks, do well-calibrated heuristics still beat complex language models? Or have LLMs simply not been optimized yet for this kind of chained temporal reasoning? The honest answer is: probably both. 🤔
To put everything in perspective, the researchers estimated that the theoretical ceiling for final cash on hand is around $2.2 billion. Even the best agents fell way, way short of that. The test, according to the authors, is far from being conquered.
Exploration beats caution
When analyzing decision trajectories, the researchers found clear behavioral differences between models. GPT-5.5 and Claude Opus 4.8 keep testing new strategies as conditions change, whether by ramping up customer acquisition, adjusting plans, or redistributing support and R&D budgets.
Claude Opus 4.7, on the other hand, reacted to problems mainly by cutting costs and preserving cash. This more passive approach allowed the model to survive until the end, but prevented it from generating real profit. It’s the classic difference between playing not to lose and playing to win.
A curious detail: Opus 4.8 and GPT-5.5 reached similar results through completely different paths. Opus 4.8 wins more customers early on but ends up zeroing out its customer base midway through the simulation, while GPT-5.5 maintains its customer base from start to finish. Both wrote surprisingly sophisticated code. Opus 4.8 built its own internal simulation, modeling customer groups to forecast future cash flow. GPT-5.5 dug through the negotiation history in the database to uncover hidden customer preferences. Pure cleverness. 💡
The researchers measured four capabilities that correlate with success:
- Discovering hidden information, like which ad channel works best for each customer segment;
- Predicting the future, measured by the error in four-week cash forecasts;
- Adapting quickly to changes, measured by how fast the model notices a competitor’s move;
- Planning ahead, measured in part by how often if-then scenarios show up in the agent’s notes.
On all four points, both Opus 4.8 and GPT-5.5 scored above the average of the other models. Not coincidentally, they were the ones that finished in the green.
The tooling environment also makes a difference
Another interesting finding has to do with the software environment agents use to take action. The researchers tested Claude Opus 4.7 with Claude Code and GPT-5.5 with Codex, two popular coding assistants. In both cases, the agents acted far less frequently and performed worse.
The suspicion is that the system prompts in these tools, tuned for software development, get in the way of management-oriented reasoning. In other words: the right tool for one task can be the wrong tool for another, even when using the same model under the hood.
And shortening the timeline doesn’t help either. When the simulation was compressed to just 50 days, only GPT-5.5 managed to finish with a profit. Most models, the researchers conclude, remain weak at coordinating decisions even when the goal is short-term.
The study’s limitations and what comes next
The authors are upfront about the experiment’s limitations. NovaMind’s product is represented by a single quality score because they couldn’t find a reliable way to evaluate more detailed qualitative changes. Issues like compliance, security, and fundraising were left out to keep each run economically feasible. Even so, CEO-Bench exposes a real gap: the distance between current models’ local competence at using tools and their ability to connect actions across extended time horizons into a coherent strategy.
What the simulation revealed is that there’s a chasm between what models do well on isolated tasks and what they can sustain in continuous decision-making environments. Most modern LLMs were trained to answer questions, summarize text, generate code, and complete short-term tasks with impressive quality. But strategic management isn’t a short-term task. It’s a long sequence of interdependent micro-decisions where accumulated context matters just as much as the immediate choice.
Another point the benchmark highlights really well is the issue of consistency under uncertainty. In the real world, CEOs make decisions with incomplete information all the time. They develop a mental model of the business, update that model as new data comes in, and adjust course when needed, but without abandoning their core strategy at every bump in the road. The AI models tested tended to either swing too much or, at the other extreme, cling to an initial approach even when it was already clear it wasn’t working.
Why this benchmark matters for the future of AI
Benchmarks like CEO-Bench are essential because they create a shared vocabulary for measuring capabilities that didn’t have a clear metric before. For years, the AI community measured intelligence through language tests, math reasoning, code generation, and text comprehension. Those criteria are important, but they leave out a huge slice of what it means to make good decisions in the real world.
The agentic AI market — systems that execute tasks autonomously and sequentially — is growing fast. Companies are investing heavily in agents that can operate with minimal oversight in real business contexts. CEO-Bench works as a stress test for those ambitions. If a model can’t keep a fictional startup alive for 500 days in a controlled environment, what real guarantees are there that it’ll perform well managing critical processes at actual companies?
The surprise of the simple rule-based agent also opens up an important conversation about hybrid system design. Maybe the answer isn’t a pure LLM making all the decisions, but an architecture where a language model works alongside layers of deterministic logic, structured memory, and long-term planning mechanisms. This kind of composite approach is already being explored in several labs, and results like those from CEO-Bench are likely to accelerate that direction.
At the end of the day, the benchmark doesn’t close any doors. It clearly points to where the next doors need to be opened. AI still isn’t ready to sit in the CEO’s chair on its own, but the path to getting there just got a whole lot clearer. 🔑🚀
