Subvolts take Rethink AI model complexity: simple rules might be holding back progress. Sep 19, 2025 - Developers
- Learn why traditional approaches to model complexity might be limiting and explore alternative perspectives for building more effective AI.
- Business
- Understand how challenging core AI assumptions could lead to more robust and capable AI systems for your business applications.
A solid interview with a university professor discussing fundamental AI concepts and challenging established ideas.
Deep Learning is Not So Mysterious or Different - Prof. Andrew Gordon Wilson (NYU)
Audio streams from Machine Learning Street Talk (MLST). Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Professor Andrew Gordon Wilson of NYU discusses how common sense in AI, particularly regarding model complexity and overfitting, might be flawed. He challenges the long-held belief that simpler models are always better, suggesting a reevaluation of the bias-variance trade-off. The episode explores alternative perspectives on building effective machine learning models.
This episode questions a common AI idea: that complex computer models are bad. Professor Wilson suggests that maybe simpler models are not always the best. Understanding this could help build smarter AI that works better.
This episode challenges fundamental machine learning assumptions. Listeners will learn why the traditional fear of complex models might be misguided and how this impacts AI development. It is for AI researchers and practitioners interested in the theoretical underpinnings of model building. This discussion is relevant to frontier labs like Google Gemini and OpenAI as they push the boundaries of model scale and architecture, potentially leading to new approaches in model design and training.
- HostTim Scarfe
- GuestAndrew Gordon WilsonProfessor, NYU
- Guest: Professor Andrew Gordon Wilson (NYU)
- Show: Machine Learning Street Talk
- Length: 124 minutes
- Core Idea: Common sense ideas in AI, like fearing model complexity, might be wrong.
Professor Andrew Wilson from NYU explains why many common-sense ideas in artificial intelligence might be wrong. For decades, the rule of thumb in machine learning has been to fear complexity. The thinking goes: if your model has too many parameters (is "too complex") for the amount of data you have, it will "overfit" by essentially memorizing the data instead of learning the underlying patterns. This leads to poor performance on new, unseen data. This is known as the classic "bias-variance trade-off" i.e. a balancing act between a model that's too simple and one that's too complex.
**SPONSOR MESSAGES**
—
Tufa AI Labs is an AI research lab based in Zurich. **They are hiring ML research engineers!**
This is a once in a lifetime opportunity to work with one of the best labs in Europe
Contact Benjamin Crouzier - https://tufalabs.ai/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Description Continued:
Professor Wilson challenges this fundamental belief (fearing complexity). He makes a few surprising points:
**Bigger Can Be Better**: massive models don't just get more flexible; they also develop a stronger "simplicity bias". So, if your model is overfitting, the solution might paradoxically be to make it even bigger.
**The "Bias-Variance Trade-off" is a Misnomer**: Wilson claims you don't actually have to trade one for the other. You can have a model that is incredibly expressive and flexible while also being strongly biased toward simple solutions. He points to the "double descent" phenomenon, where performance first gets worse as models get more complex, but then surprisingly starts getting better again.
**Honest Beliefs and Bayesian Thinking**: His core philosophy is that we should build models that honestly represent our beliefs about the world. We believe the world is complex, so our models should be expressive. But we also believe in Occam's razor—that the simplest explanation is often the best. He champions Bayesian methods, which naturally balance these two ideas through a process called marginalization, which he describes as an automatic Occam's razor.
TOC:
[00:00:00] Introduction and Thesis
[00:04:19] Challenging Conventional Wisdom
[00:11:17] The Philosophy of a Scientist-Engineer
[00:16:47] Expressiveness, Overfitting, and Bias
[00:28:15] Understanding, Compression, and Kolmogorov Complexity
[01:05:06] The Surprising Power of Generalization
[01:13:21] The Elegance of Bayesian Inference
[01:33:02] The Geometry of Learning
[01:46:28] Practical Advice and The Future of AI
Prof. Andrew Gordon Wilson:
https://x.com/andrewgwils
https://cims.nyu.edu/~andrewgw/
https://scholar.google.com/citations?user=twWX2LIAAAAJ&hl=en
https://www.youtube.com/watch?v=Aja0kZeWRy4
https://www.youtube.com/watch?v=HEp4TOrkwV4
TRANSCRIPT:
https://app.rescript.info/public/share/H4Io1Y7Rr54MM05FuZgAv4yphoukCfkqokyzSYJwCK8
Hosts:
Dr. Tim Scarfe / Dr. Keith Duggar (MIT Ph.D)
REFS:
Deep Learning is Not So Myste
- overfit
- Professor Andrew Gordon Wilson
- When a computer model learns the training data too well, including its noise, and performs poorly on new data.
- bias-variance trade-off
- Professor Andrew Gordon Wilson
- The balance between a model that is too simple and misses patterns (bias) and one that is too complex and learns noise (variance).
48 minWhat Most People Get Wrong About Evolution | Akarsh Kumar
WithAkarsh Kumar
1 h 101 h 14Who Checks a Proof No Human Can Read? — Leo de Moura
WithLeo de Moura
44 minQuestions people ask
- What is the traditional view on model complexity in machine learning?
- The traditional view is to fear complexity. If a model has too many parameters for the data, it can overfit by memorizing data instead of learning patterns.
- What is the bias-variance trade-off?
- It is a balancing act between a model that is too simple (high bias) and one that is too complex (high variance).
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Machine Learning Street Talk (MLST).