Subvolts take AI's next leap requires new architectures beyond transformers for continual learning, not just more scale. Jul 29, 2026 - Developers
- Learn about potential architectural shifts beyond transformers and the challenges of continual learning.
- Business
- Understand emerging research that could redefine the future capabilities and development cycles of AI systems.
This interview presents a significant, contrarian viewpoint on AI architecture from experienced researchers at a new venture.
Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil
Audio streams from Sequoia Capital. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Sequoia Capital's Training Data podcast features Jerry Tworek and Rohan Anil of Core Automation. They argue the transformer architecture has reached its limits for AI progress. The pair propose continual learning as the next frontier, suggesting current models inefficiently use computation and cannot adapt at test time.
Two AI experts believe current AI models, like transformers, have hit a wall. They suggest future AI needs to learn continuously, like humans, adapting in real time instead of needing constant retraining. This could unlock smarter, more flexible AI.
This episode offers a contrarian view on AI architecture, challenging the dominance of transformers. It's for researchers and developers questioning current scaling approaches. The discussion connects to frontier labs' focus on agents and coding, highlighting a potential bottleneck beyond just scale. Tworek and Anil's work on continual learning could represent a significant shift if successful, impacting how future AI systems are built and trained.
- HostSequoia Capital
- GuestJerry Tworekled reasoning, OpenAI
- GuestRohan Anilco-led Gemini pre-training, Google
- Guests: Jerry Tworek and Rohan Anil
- Show: Training Data (OpenAI)
- Length: 49 minutes
- Core Automation's Premise: The transformer has carried us as far as it can; the bottleneck is architecture, not scale.
- Missing Capability: Continual learning: models that adapt at test time.
Jerry Tworek led reasoning at OpenAI, convinced that scaling reinforcement learning was the path to AGI. Rohan Anil co-led Gemini pre-training and built the Shampoo optimizer. Now they've teamed up at Core Automation on a contrarian premise: the transformer has carried us as far as it can, and the bottleneck to smarter systems is no longer scale — it's the architecture itself. The missing capability is continual learning, models that adapt at test time, which transformers can't do. In-context learning taps out fast (Codex needs compacting after ~20 minutes) and fine-tuning invites catastrophic forgetting. Rohan argues pre-training and RL should be optimized end-to-end, and that transformers spend computation inefficiently. They lay out why the largest labs won't chase alternatives while locked in the coding-agent race, and why building the world's most automated lab starts with automating kernel generation—the one place frontier models still lose to a high-taste human.
Hosted by Sonya Huang and Pat Grady, Sequoia Capital
- transformer
- Rohan Anil
- A type of AI model architecture that has been very successful, especially for language tasks.
- reinforcement learning
- Jerry Tworek
- A way for AI to learn by trial and error, getting rewards for good actions and penalties for bad ones.
- continual learning
- Rohan Anil
- AI models that can learn new information over time without forgetting what they already know.
- catastrophic forgetting
- Rohan Anil
- When an AI model learns new information and completely forgets previously learned knowledge.
1 h 08David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience
WithDavid Silver, Lasse Espeholt
1 h 041 h 0552 minMaking Cities Awesome: Peregrine’s Nick Noone & Ben Rudolph
WithNick Noone, Ben Rudolph
Same guests, other shows
1 h 03
1 h 16How GPT-5 Thinks — OpenAI VP of Research Jerry Tworek
WithJerry Tworek
Questions people ask
- What is Core Automation's main argument about AI architecture?
- Core Automation argues that the transformer architecture has reached its limits and is no longer the primary bottleneck for AI progress. They believe continual learning is the missing capability.
- Why do transformers struggle with continual learning?
- Transformers cannot adapt at test time. In-context learning is limited, and fine-tuning leads to catastrophic forgetting, preventing models from learning new information without losing old knowledge.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Sequoia Capital.