Subvolts take Category Theory offers a scientific basis for AI, moving beyond LLM pattern matching to true algorithmic understanding. Dec 22, 2025 - Developers
- Learn how Category Theory might offer a principled way to build more robust AI systems, moving beyond current deep learning limitations.
- Business
- Understand the potential for more reliable AI by grasping the scientific underpinnings discussed, which could lead to more predictable AI performance.
This episode features a deep dive with multiple researchers into a foundational mathematical concept that could significantly advance AI science.
Making deep learning perform real algorithms with Category Theory (Andrew Dudzik, Petar Velichkovich, Taco Cohen, Bruno Gavranović, Paul Lessard)
Audio streams from Machine Learning Street Talk (MLST). Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Machine Learning Street Talk features Andrew Dudzik, Petar Velichkovich, Taco Cohen, Bruno Gavranović, and Paul Lessard discussing how Category Theory could provide a scientific foundation for deep learning. They explain LLMs struggle with basic logic, like addition, because their architecture lacks true algorithmic understanding. The episode proposes Category Theory as a framework to move AI from "alchemy" to rigorous engineering, offering a "Periodic Table" for neural networks.
Current AI models like ChatGPT can seem smart, but they struggle with simple math because they just recognize patterns, not true logic. This episode explores how a complex math idea called Category Theory could give AI a solid foundation, like a "Periodic Table," to build more reliable and understandable systems.
This episode is for AI researchers and engineers seeking to understand the fundamental limitations of current deep learning models. It connects to frontier labs like Google Gemini and OpenAI, which are pushing the boundaries of LLM capabilities. By exploring Category Theory, the discussion offers a potential path to more robust and predictable AI systems, moving beyond pattern matching to genuine reasoning. This research could inform future model architectures across the AI landscape.
- HostTim Scarfe
- GuestAndrew Dudzik
- GuestPetar Velichkovich
- GuestTaco Cohen
- GuestBruno Gavranović
- GuestPaul Lessard
- Episode Title: Making deep learning perform real algorithms with Category Theory
- Show: Machine Learning Street Talk
- Length: 44 minutes
- LLMs struggle with logic: LLMs still struggle with the logic of a second-grader and cannot reliably add large numbers.
- Category Theory as a framework: Category Theory could provide the "Periodic Table" for neural networks, turning AI "alchemy" into rigorous science.
We often think of Large Language Models (LLMs) as all-knowing, but as the team reveals, they still struggle with the logic of a second-grader. Why can’t ChatGPT reliably add large numbers? Why does it "hallucinate" the laws of physics? The answer lies in the architecture. This episode explores how *Category Theory* —an ultra-abstract branch of mathematics—could provide the "Periodic Table" for neural networks, turning the "alchemy" of modern AI into a rigorous science.
In this deep-dive exploration, *Andrew Dudzik*, *Petar Velichkovich*, *Taco Cohen*, *Bruno Gavranović*, and *Paul Lessard* join host *Tim Scarfe* to discuss the fundamental limitations of today’s AI and the radical mathematical framework that might fix them.
TRANSCRIPT:
https://app.rescript.info/public/share/LMreunA-BUpgP-2AkuEvxA7BAFuA-VJNAp2Ut4MkMWk
---
Key Insights in This Episode:
* *The "Addition" Problem:* *Andrew Dudzik* explains why LLMs don't actually "know" math—they just recognize patterns. When you change a single digit in a long string of numbers, the pattern breaks because the model lacks the internal "machinery" to perform a simple carry operation.
* *Beyond Alchemy:* deep learning is currently in its "alchemy" phase—we have powerful results, but we lack a unifying theory. Category Theory is proposed as the framework to move AI from trial-and-error to principled engineering. [00:13:49]
* *Algebra with Colors:* To make Category Theory accessible, the guests use brilliant analogies—like thinking of matrices as *magnets with colors* that only snap together when the types match. This "partial compositionality" is the secret to building more complex internal reasoning. [00:09:17]
* *Synthetic vs. Analytic Math:* *Paul Lessard* breaks down the philosophical shift needed in AI research: moving from "Analytic" math (what things are made of) to "Synthetic" math [00:23:41]
---
Why This Matters for AGI
If we want AI to solve the world's hardest scientific problems, it can't just be a "stochastic parrot." It needs to internalize the rules of logic and computation. By imbuing neural networks with categorical priors, researchers are attempting to build a future where AI doesn't just predict the next word—it understands the underlying structure of the universe.
---
TIMESTAMPS:
00:00:00 The Failure of LLM Addition & Physics
00:01:26 Tool Use vs Intrinsic Model Quality
00:03:07 Efficiency Gains via Internalization
00:04:28 Geometric Deep Learning & Equivariance
00:07:05 Limitations of Group Theory
00:09:17 Category Theory: Algebra with Colors
00:11:25 The Systematic Guide of Lego-like Math
00:13:49 The Alchemy Analogy & Unifying Theory
00:15:33 Information Destruction & Reasoning
00:18:00 Pathfinding & Monoids in Computation
00:20:15 System 2 Reasoning & Error Awareness
00:23:31 Analytic vs Synthetic Mathematics
00:25:52 Morphisms & Weight Tying Basics
00:26:48 2-Categories & Weight Sharing Theory
00:28:55 Higher Categories & Emergence
00:31:41 Compositionality & Recursive Folds
00:34:05 Syntax vs Semantics in Network Design
00:36:14 Homomorphisms & Multi-Sorted Syntax
00:39:30 The Carrying Problem & Hopf Fibrations
Petar Veličković (GDM)
https://petar-v.com/
Paul Lessard
https://www.linkedin.com/in/paul-roy-lessard/
Bruno Gavranović
https://www.brunogavranovic.com/
Andrew Dudzik (GDM)
https://www.linkedin.com/in/andrew-dudzik-222789142/
---
REFERENCES:
Model:
[00:01:05] Veo
https://deepmind.google/models/veo/
[00:01:10] Genie
https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/
Paper:
<- Category Theory
- Andrew Dudzik, Petar Velichkovich, Taco Cohen, Bruno Gavranović, Paul Lessard
- An ultra-abstract branch of mathematics that studies abstract structures and their relationships, potentially offering a unifying framework for AI.
- LLMs
- Andrew Dudzik, Petar Velichkovich, Taco Cohen, Bruno Gavranović, Paul Lessard
- Large Language Models, AI systems trained on vast amounts of text data to understand and generate human-like language.
- hallucinate
- Andrew Dudzik, Petar Velichkovich, Taco Cohen, Bruno Gavranović, Paul Lessard
- When an AI model generates false or nonsensical information that is not based on its training data.
48 minWhat Most People Get Wrong About Evolution | Akarsh Kumar
WithAkarsh Kumar
1 h 101 h 14Who Checks a Proof No Human Can Read? — Leo de Moura
WithLeo de Moura
44 min
1 h 16Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show
WithEmmett Shear, Séb Krier
24 min
Questions people ask
- Why do LLMs struggle with basic math like addition?
- LLMs recognize patterns, not true logic. They lack the internal machinery for operations like carryovers, so changing a digit breaks the pattern they've learned.
- What is Category Theory's role in AI?
- Category Theory could act as a "Periodic Table" for neural networks, providing a unifying scientific framework to move AI development from trial-and-error to principled engineering.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Machine Learning Street Talk (MLST).