Subvolts take Transformer co-inventor argues current AI is stuck; Sakana AI proposes a biology-inspired fix. Nov 23, 2025 - Developers
- Learn about potential limitations of current AI architectures and a novel, biology-inspired alternative for future model development.
- Business
- Understand emerging research that could redefine AI capabilities, potentially impacting future AI product development and competitive landscapes.
A co-inventor of the foundational Transformer architecture presents a critique and a new research direction.
He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI]
Audio streams from Machine Learning Street Talk (MLST). Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Llion Jones, co-inventor of the Transformer architecture, and Luke Darlow from Sakana AI discuss limitations in current AI models. They argue that architectures like the Transformer might trap AI in "localized ruts," preventing true reasoning. The episode introduces their proposed solution, the Continuous Thought Machine (CTM), a biology-inspired model aiming to overcome these issues.
Imagine AI today draws spirals by making many tiny straight lines, faking understanding. Llion Jones, who helped invent the AI tech behind ChatGPT, says this limits AI. He and his colleagues at Sakana AI introduce a new idea, the Continuous Thought Machine, inspired by biology, to help AI truly understand concepts.
This episode challenges the foundational Transformer architecture powering much of today's AI. It's for researchers and developers questioning current AI limitations and seeking new paradigms. The discussion on "continuous thought" and biology-inspired models offers a potential path forward, contrasting with the incremental improvements seen in frontier labs like OpenAI, Google Gemini, and Anthropic. It highlights a fundamental debate about AI's future direction and reasoning capabilities.
- HostTim Scarfe
- GuestLlion JonesCo-inventor of Transformer, Sakana AI
- GuestLuke DarlowSakana AI
- Guest: Llion Jones and Luke Darlow (Sakana AI)
- Show: Machine Learning Street Talk
- Length: 73 minutes
- Core Argument: Transformer architecture may trap AI in localized ruts, preventing true reasoning.
- Proposed Solution: Continuous Thought Machine (CTM), a biology-inspired model.
The Transformer architecture (which powers ChatGPT and nearly all modern AI) might be trapping the industry in a localized rut, preventing us from finding true intelligent reasoning, according to the person who co-invented it. Llion Jones and Luke Darlow, key figures at the research lab Sakana AI, join the show to make this provocative argument, and also introduce new research which might lead the way forwards.
**SPONSOR MESSAGES START**
—
Build your ideas with AI Studio from Google - http://ai.studio/build
—
Tufa AI Labs is hiring ML Research Engineers https://tufalabs.ai/
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
**END**
The "Spiral" Problem – Llion uses a striking visual analogy to explain what current AI is missing. If you ask a standard neural network to understand a spiral shape, it solves it by drawing tiny straight lines that just happen to look like a spiral. It "fakes" the shape without understanding the concept of spiraling.
Introducing the Continuous Thought Machine (CTM) Luke Darlow deep dives into their solution: a biology-inspired model that fundamentally changes how AI processes information.
The Maze Analogy: Luke explains that standard AI tries to solve a maze by staring at the whole image and guessing the entire path instantly. Their new machine "walks" through the maze step-by-step.
Thinking Time: This allows the AI to "ponder." If a problem is hard, the model can naturally spend more time thinking about it before answering, effectively allowing it to correct its own mistakes and backtrack—something current Language Models struggle to do genuinely.
https://sakana.ai/
https://x.com/YesThisIsLion
https://x.com/LearningLukeD
TRANSCRIPT:
https://app.rescript.info/public/share/crjzQ-Jo2FQsJc97xsBdfzfOIeMONpg0TFBuCgV2Fu8
TOC:
00:00:00 - Stepping Back from Transformers
00:00:43 - Introduction to Continuous Thought Machines (CTM)
00:01:09 - The Changing Atmosphere of AI Research
00:04:13 - Sakana’s Philosophy: Research Freedom
00:07:45 - The Local Minimum of Large Language Models
00:18:30 - Representation Problems: The Spiral Example
00:29:12 - Technical Deep Dive: CTM Architecture
00:36:00 - Adaptive Computation & Maze Solving
00:47:15 - Model Calibration & Uncertainty
01:00:43 - Sudoku Bench: Measuring True Reasoning
REFS:
Why Greatness Cannot be planned [Kenneth Stanley]
https://www.amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237
https://www.youtube.com/watch?v=lhYGXYeMq_E
The Hardware Lottery [Sara Hooker]
https://arxiv.org/abs/2009.06489
https://www.youtube.com/watch?v=sQFxbQ7ade0
Continuous Thought Machines [Luke Darlow et al / Sakana]
https://arxiv.org/abs/2505.05522
https://sakana.ai/ctm/
LSTM: The Comeback Story? [Prof. Sepp Hochreiter]
https://www.youtube.com/watch?v=8u2pW2zZLCs
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley]
https://arxiv.org/pdf/2505.11581
A Spline Theory of Deep Networks [Randall Balestriero]
https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf
https://www.youtube.com/watch?v=86ib0sfdFtw
https://www.youtube.com/watch?v=l3O2J3LMxqI
On the Biology of a Large Language Model [Anthropic, Jack Lindsey et al]
https://transformer-
- Transformer architecture
- Llion Jones
- The core technology behind most modern AI like ChatGPT, which processes information in a specific way.
- Localized rut
- Llion Jones
- A state where AI gets stuck in a limited way of solving problems, preventing deeper understanding.
- Continuous Thought Machine (CTM)
- Luke Darlow
- A new AI model concept inspired by biology, designed to process information more deeply.
48 minWhat Most People Get Wrong About Evolution | Akarsh Kumar
WithAkarsh Kumar
1 h 101 h 14Who Checks a Proof No Human Can Read? — Leo de Moura
WithLeo de Moura
44 min
Questions people ask
- What is the "Spiral Problem" in AI?
- The Spiral Problem describes how current AI might fake understanding a spiral by drawing many tiny straight lines, rather than grasping the concept of spiraling itself.
- What is the Continuous Thought Machine (CTM)?
- CTM is a new, biology-inspired AI model proposed by Sakana AI that aims to fundamentally change how AI processes information, moving beyond the limitations of current architectures.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Machine Learning Street Talk (MLST).