Subvolts take Explore new LLM training methods to overcome creative limits and catastrophic overtraining. Sep 16, 2025 - Developers
- Learn novel techniques to enhance LLM creativity and address overtraining issues in your models.
- Business
- Understand emerging research that could lead to more adaptable and reliable AI applications.
This interview features a researcher discussing fundamental limitations and novel solutions for LLM creativity and training.
Is It Time to Rethink LLM Pre-Training? with Aditi Raghunathan - #747
Audio streams from Sam Charrington. Your place is remembered on this device.
From Subvolts
Summary and analysis by Subvolts editorial, AI-assisted.
Aditi Raghunathan joins The TWIML AI Podcast to discuss limitations in large language models (LLMs) and how to build more adaptable, creative AI. The episode explores her research on "Roll the dice & look before you leap," which aims to improve LLM novelty, and the issue of "catastrophic overtraining." They also cover creating more controllable models and targeted unlearning.
This episode discusses how current AI models, like those used by major tech companies, struggle to generate truly new ideas. Researchers are exploring ways to make AI more creative and prevent it from forgetting how to learn new tasks after training.
This episode offers insights into fundamental LLM limitations beyond next-token prediction. Listeners learn about novel approaches to enhance AI creativity and adaptability, and the challenges of overtraining. This is for AI researchers and developers working on improving LLM capabilities. It connects to frontier labs' ongoing efforts to push LLM performance and control, addressing issues like emergent behaviors and fine-tuning effectiveness.
- HostSam Charrington
- GuestAditi Raghunathanassistant professor, Carnegie Mellon University
- Guest: Aditi Raghunathan, assistant professor at Carnegie Mellon University
- Episode Title: Is It Time to Rethink LLM Pre-Training? with Aditi Raghunathan - #747
- Show: The TWIML AI Podcast
- Length: 58 minutes
- Research Focus: "Roll the dice & look before you leap" for LLM creativity
- Key Concept: Catastrophic overtraining
Today, we're joined by Aditi Raghunathan, assistant professor at Carnegie Mellon University, to discuss the limitations of LLMs and how we can build more adaptable and creative models. We dig into her ICML 2025 Outstanding Paper Award winner, “Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction,” which examines why LLMs struggle with generating truly novel ideas. We dig into the "Roll the dice" approach, which encourages structured exploration by injecting randomness at the start of generation, and the "Look before you leap" concept, which trains models to take "leaps of thought" using alternative objectives to create more diverse and structured outputs. We also discuss Aditi’s papers exploring the counterintuitive phenomenon of "catastrophic overtraining," where training models on more data improves benchmark performance but degrades their ability to be fine-tuned for new tasks, and dig into her lab's work on creating more controllable and reliable models, including the concept of "memorization sinks," an architectural approach to isolate and enable the targeted unlearning of specific information.
The complete show notes for this episode can be found at https://twimlai.com/go/747.
- LLM
- Large Language Model
- AI models that understand and generate human-like text
- next-token prediction
- next-token prediction
- AI guessing the next word in a sentence
- fine-tuned
- fine-tuned
- AI trained for a specific task
53 min1 h 30Why Jev Is Changing How We Build With AI with Diogo Almeida - #779
WithDiogo Almeida
1 h 081 h 05From Voice Agents to AI Avatars with Alexander Smola - #777
WithAlexander Smola
Questions people ask
- What is the 'Roll the dice & look before you leap' approach?
- It is a method that injects randomness at the start of generation to encourage structured exploration and improve LLM creativity.
- What is 'catastrophic overtraining' in LLMs?
- It is a phenomenon where training models on more data improves benchmark performance but degrades their ability to be fine-tuned for new tasks.
Page generated Oct 19, 2026. Summaries are Subvolts' own; the audio, artwork and notes belong to Sam Charrington.