Subvolts take AI's future hinges on moving beyond current models, says a transformer pioneer. Jun 3, 2026 - Developers
- Learn about potential architectural shifts beyond transformers and how AI agents impact research workflows.
- Business
- Understand the fundamental limits of current AI and what future breakthroughs might mean for AI adoption.
- Everyday users
- Gain perspective on the core technology behind AI tools and the ongoing debate about AI's true capabilities.
Features a foundational researcher discussing the core limitations and future of AI architectures.
Ep 89: AI Research Legend’s Honest Assessment of Where We Are
Audio streams from by Redpoint Ventures. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
This "Unsupervised Learning" podcast episode features AI researcher Lukasz Kaiser discussing the limits of current AI architectures, particularly transformers. Kaiser, a co-author of the "Attention Is All You Need" paper, shares his dual perspective: belief in current progress, evidenced by his own productivity gains, alongside humility about AI's ability to generalize like humans. The conversation covers data efficiency, future architectures, and the impact of AI agents.
This podcast talks with an AI pioneer about the computer "brains" we use today, called transformers. He explains their strengths and weaknesses, and wonders if we need entirely new kinds of brains to make AI truly smart like people, especially for understanding the real world.
This episode offers deep insights into the foundational technology behind modern AI from a key architect. Listeners will learn about the potential limitations of transformer models and what might come next. It is for AI researchers, developers, and enthusiasts curious about the future of AI architectures. The discussion on architectural leaps and generalization connects directly to ongoing research at OpenAI, Google Gemini, and Anthropic as they push the boundaries of AI capabilities.
- HostJacob Effron
- HostJordan Segall
- GuestLukasz KaiserAI Researcher
- Guest: Lukasz Kaiser
- Episode Title: Ep 89: AI Research Legend’s Honest Assessment of Where We Are
- Show: Unsupervised Learning
- Length: 74 minutes
- Guest's Productivity Gain: 10x productivity gains using Codex
This episode with Lukasz Kaiser, co-author of the seminal "Attention Is All You Need" transformer paper and former researcher at both Google Brain and OpenAI, is a wide-ranging conversation about the fundamental limits of current AI architectures and whether transformers will continue to dominate or eventually give way to something new. Lukasz brings a rare dual perspective: deep belief in how far the current paradigm has taken us (he's an enthusiastic daily Codex user who's seen 10x productivity gains in his own research), while maintaining genuine intellectual humility about whether transformers can truly generalize the way humans do. The episode weaves together questions about data efficiency, the non-verifiable RL frontier, the coding agent revolution, the open vs. closed source gap, and what the next architectural leap might look like: all filtered through the lens of someone who helped build the foundation the entire field is standing on.
(0:00) Intro
(1:12) Transformers vs. Human Learning
(8:37) How Do We Get Physical World Generalization?
(10:52) What Comes After Transformers
(13:59) How Much Have Agents Improved Lukasz's AI Research Productivity?
(17:21) How Close Is an AI Research Intern?
(26:06) RL Beyond Verifiable Tasks
(35:38) App Companies: Build Models or Lean on Labs?
(46:21) Multimodal Is Still Missing Something
(49:46) OpenAI's Bet on Reasoning
(55:26) The AI Coding Wars
(59:26) Focus vs. Keeping Embers Burning
(1:02:09) Open Source vs. Closed Source Gap
(1:05:15) Quickfire
Jacob is an AI investor at Redpoint Ventures. He's led Redpoint's investments in companies like Abridge, Physical Intelligence & Legora. Follow Jacob on Twitter (@jacobeffron).
On Unsupervised Learning we probe the sharpest minds in AI in search for the truth about what's real today, what will be real in the future and what it all means for businesses and the world. If you're a builder, researcher or investor navigating the AI world, this podcast will help you deconstruct and understand the most important breakthroughs and see a clearer picture of reality. Subscribe to this show to stay up to date on our latest episodes.
- transformers
- Lukasz Kaiser
- A type of AI architecture that powers many modern AI systems, like ChatGPT.
- generalize
- Lukasz Kaiser
- The ability of an AI to apply what it learned in one situation to new, different situations.
- data efficiency
- Lukasz Kaiser
- How well an AI can learn from a small amount of data, rather than needing vast amounts.
- RL
- Lukasz Kaiser
- Reinforcement Learning, a way AI learns by trial and error, getting rewards for good actions.
1 h 46
52 min
Questions people ask
- Who is Lukasz Kaiser?
- Lukasz Kaiser is a former researcher at Google Brain and OpenAI and a co-author of the "Attention Is All You Need" transformer paper.
- What are the main topics discussed?
- The episode covers fundamental limits of AI architectures, the future beyond transformers, AI agent productivity gains, and the open vs. closed source debate.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to by Redpoint Ventures.