Scaling Laws
Also written: Scaling, Scaling trend, Scaling pre-training, Scaling hypothesis
Scaling laws show how AI model performance predictably improves with increases in size, data, and compute.
Scaling laws are observed patterns in how AI models get better as they get bigger. Think of it like building with LEGOs. The more bricks you have (data) and the larger your structure (model size), the more complex and capable your creation can become. This improvement isn't random; it follows predictable trends.
For example, researchers found that if you double the amount of data and computing power used to train a language model, its performance on tasks like answering questions might improve by a predictable factor, not just a little bit, but significantly. This predictability is crucial for planning how to build more powerful AI systems.
Scaling laws describe empirical relationships between model performance and key resources: model size (parameters), dataset size, and compute (FLOPs). They demonstrate that performance, often measured by loss, improves as a power law of these resources. This predictability allows researchers to forecast the capabilities of larger models before training them, guiding resource allocation and research direction.
While performance on metrics like loss is predictable, the emergent capabilities and specific behaviors of scaled models can be less predictable, as noted in Anthropic's research. Tradeoffs involve the significant computational cost and data requirements for achieving these gains.
Scaling laws are central to current AI development as frontier labs like Google Gemini and OpenAI continue to push the boundaries of model size and capability. Recent research, such as Google's ATLAS, focuses on making these laws more practical for multilingual models, while Anthropic highlights the tension between predictable scaling and unpredictable emergent behaviors.
- Measuring the persuasiveness of language modelsAnthropic · Anthropic Research ·
- AI for Atoms: How Periodic Labs is Revolutionizing Materials Engineering with Co-Founder Liam FedusOpenAI · No Priors ·
- Dario Amodei — "We are near the end of the exponential"Anthropic · Dwarkesh Podcast ·
- State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490Anthropic · Lex Fridman ·
- Ep 81: Ex-OpenAI Researcher On Why He Left, His Honest AGI Timeline, & The Limits of Scaling RLOpenAI · Unsupervised Learning ·
- ATLAS: Practical scaling laws for multilingual modelsGoogle · Google Research Blog ·
- DeepMind Gemini 3 Lead: What Comes After "Infinite Data"Google · The MAD Podcast ·
- Predictability and Surprise in Large Generative ModelsAnthropic · Anthropic Research ·
- Amazon’s AI Resurgence: AWS & Anthropic’s Multi-Gigawatt Trainium ExpansionAnthropic · SemiAnalysis ·
- OpenAI Scholars 2021: Final projectsOpenAI · OpenAI News ·
- Anthropic Claude
- Anthropic research has explored the predictability and surprise inherent in scaling laws, noting that while performance scales predictably, emergent behaviors can be unpredictable.
- Google Gemini
- Google's ATLAS project focuses on developing practical scaling laws specifically for building effective multilingual AI models.
- OpenAI ChatGPT
- OpenAI's work, including discussions on Lex Fridman's podcast, frequently touches upon scaling laws as a fundamental principle driving progress in large language models.
- Generalization
- Generalization refers to how well a model performs on unseen dat
- Overfitting
- Overfitting occurs when a model learns the training data too well a
- Hyperparameter Tuning
- Hyperparameter tuning involves finding optimal settings f
The concept of scaling laws in machine learning gained prominence with early work on neural networks and was significantly popularized by research from OpenAI and DeepMind in the late 2010s and early 2020s. Source ↗
Ask a frontier AI model like ChatGPT, Claude, or Gemini to write a short story about a specific topic, then ask it to write a longer, more detailed version. Observe how the increased prompt length and detail might lead to a more elaborate output, reflecting the principle of scaling up complexity.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- Do scaling laws guarantee better AI?
- Scaling laws show performance improves predictably with more resources, but they don't guarantee specific emergent abilities or perfect behavior. Unpredictable behaviors can still arise.
- Are scaling laws the only way to improve AI?
- No, while scaling is a major driver, improvements also come from better algorithms, architectures, data quality, and training techniques.
First seen on Subvolts May 10, 2021. Updated Oct 12, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.