How Well Can AI Do Strategy? Empirical Benchmarking Using Strategy Simulations.

This article focuses on benchmarking large language models' (LLMs) capabilities in strategic decision making using the Back Bay Battery (BBB) simulation, a widely used business strategy exercise that captures key elements of real-world strategy such as uncertainty, complexity, irreversible multiperiod decisions, and delayed feedback. The study evaluates 21 proprietary and 13 open-source LLMs on their ability to balance short-term profitability with long-term investment in emerging technologies, comparing their performance to historical data from 249 MBA students. Results show that while LLMs have generally improved over time, models from late 2024 to early 2025 (e.g., OpenAI's o3-mini, Claude Sonnet 4, Gemini 2.0 Flash) outperformed both earlier models and the MBA average, the most recent frontier models from mid-2025 (e.g., GPT-5, Gemini 2.5 Pro) surprisingly underperformed, exhibiting a bias toward exploiting core businesses at the expense of future growth. The paper argues for the importance of strategy-specific benchmarks like BBB to accurately assess and guide AI development in strategic contexts, highlighting current LLM limitations in managing strategic uncertainty despite advances in other domains.