890: The “State of AI” Report 2025
🎯 Summary
Summary of Super Data Science Podcast Episode 890: State of AI in 2025
This episode of the Super Data Science Podcast, hosted by John Crone, provides a rapid-fire analysis of the five most critical takeaways from the recently published 2025 edition of the Stanford University Institute for Human-Centered AI’s AI Index Report. The discussion highlights dramatic technical advancements, significant cost reductions, emerging agent capabilities, soaring business adoption, and shifting global investment landscapes.
Key Takeaways for Technology Professionals:
1. Extreme Efficiency Gains in Model Size: The most striking technical trend is the massive reduction in model size required to achieve equivalent performance. The benchmark used is the MMLU score (above 60%). In 2022, Google’s PaLM (540B parameters) was needed; by 2024, Microsoft’s Phi-3 Mini achieved the same result with only 3.8B parameters. This represents a 142-fold reduction in model size over two years, meaning capabilities are now accessible with models 1% the size of their predecessors.
2. Dramatic Cost Deflation in LLM Inference: The operational cost of running large language models has plummeted. The cost to query a model with GPT-3.5 equivalent performance (65% MMLU accuracy) dropped from approximately $20 per million tokens in late 2022 to just $0.7 per million tokens by late 2024 (citing Gemini 1.5 Flash 8B). This 280-fold cost reduction in under two years makes high-volume AI deployment significantly more feasible for businesses.
3. Maturation and Superiority of AI Agents: AI agents are demonstrating powerful capabilities, particularly in short-horizon tasks. Using the new REBench benchmark, top AI systems now score four times better than human experts on complex tasks completed within two hours or less. While humans still lead in tasks requiring longer timeframes (e.g., 32 hours), AI agents are already matching or exceeding human expertise in specific complex areas like certain types of code generation, delivering results faster.
4. Explosive Organizational AI Adoption: The combination of better, cheaper models has fueled rapid enterprise integration. Organizational AI adoption across companies rose from 55% in 2023 to 78% in 2024. Furthermore, the use of Generative AI specifically within business functions surged from 38% to 71% in the same period, indicating a shift from experimentation to widespread functional deployment.
5. Shifting Global Private Investment Landscape: Private investment in AI reached new peaks globally in 2024, driven heavily by the US. The US recorded $109 billion in private AI investment, surpassing the previous 2021 peak. Europe also saw record investment ($19 billion), though it remains less than a fifth of US levels. Crucially, China bucked the trend, with its private AI investment decreasing annually since its 2021 peak, falling to just $9 billion in 2024—less than half of European investment.
Strategic Implications and Context:
This conversation underscores a critical inflection point: AI is moving from a research curiosity to a highly efficient, cost-effective, and deeply integrated enterprise utility. For technology professionals, the key implication is the necessity of optimizing for smaller, highly capable models (efficiency) and rapidly integrating autonomous agents into workflows to capture productivity gains. The investment data suggests a significant concentration of AI innovation and capital in the US market, while the divergence in Chinese investment warrants strategic attention regarding global technology supply chains and competition.
🏢 Companies Mentioned
💬 Key Insights
"Interestingly, China bucked the trend seen in the US and Europe since peaking at around $25 billion of private investment in 2021, Chinese investment in AI has actually decreased every single year since, now coming in at just $9 billion..."
"The cost fell from about $20 per million tokens in November 2022 to just $0.7 per million tokens by October 2024. That's using Gemini 1.5 Flash 8B from Google."
"That represents a 142-fold reduction in model size over two years. That's crazy. We're down to 1% of the model size to get the same results in a two-year period."
"in 2022, PaLM, a model from Google, which had 540 billion model parameters, was the smallest model scoring above 60% on a very common, very important benchmark for LLMs called MMLU. By 2024, two years later, Microsoft's Phi-3 Mini achieved the same performance threshold on only 3.8 billion parameters."
"In the US alone, there was $109 billion of private investment in AI in 2024, topping the previous peak in 2021, which was about $20 billion lower, so absolutely crushing with a new high benchmark for private investment in 2024."
"the percentage of participants indicating generative AI use in business functions saw a dramatic increase, climbing from just 38% in 2023 to a remarkable 71% by the following year."