Decoder frozen
Part of an AI model's structure is locked, so its learned parameters do not change during training or updates.
When a part of an AI model is "frozen," it means that the specific settings or knowledge within that part are locked in place. They will not be changed or updated, even when the rest of the model is being trained on new information.
This is often done to reuse a part of a model that already performs well, like a component that is good at understanding images. Instead of retraining it from scratch, developers freeze it and only train the new parts of the model. This can save time and computing power, and it helps ensure that the frozen part keeps its original capabilities. For example, a developer might freeze the image recognition part of a model and only train the part that writes descriptions of the images.
Freezing in neural networks refers to preventing the weights and biases of specific layers or components from being updated during the training process. This is commonly employed in transfer learning, where a pre-trained model's features are adapted for a new task. By freezing earlier layers, which often learn general features, and only training later layers, which learn task-specific features, training time and data requirements can be significantly reduced. It preserves the learned representations in the frozen layers, preventing them from being distorted by new, potentially less relevant, training data.
Meta AI's Llama Cookbook mentions a "decoder frozen" update, indicating this technique is actively being used and documented by major AI labs. This suggests ongoing exploration of efficient training and adaptation methods for large language models.
- Llama Cookbook: decoder frozenMeta · Llama Cookbook ·
- Llama Cookbook: decoder frozenMeta · Llama Cookbook ·
- Meta Meta AI
- Meta AI's Llama Cookbook references 'decoder frozen' updates, suggesting its use in their model development and adaptation processes.
- Fine-tuning
- Fine-tuning involves updating all or a subset of model weights on a
- Quantization
- Quantization reduces the precision of model weights to save memory
- Pruning
- Pruning removes unnecessary weights or neurons from a model to reduce i
The concept of freezing weights originates from the early days of deep learning research, particularly in the context of transfer learning. It became a standard technique for adapting large, pre-trained models to new tasks without requiring massive datasets or computational resources. Source ↗
You can explore this concept by looking at tutorials for fine-tuning models like Stable Diffusion or BERT. Many such tutorials show how to freeze certain layers of a pre-trained model before training on a new dataset. This allows you to see how only specific parts of the model learn new information while others remain unchanged.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- Why would you freeze part of a model?
- Freezing parts of a model saves training time and resources, especially when adapting a large pre-trained model to a new, similar task. It also helps preserve valuable learned features.
- Does freezing affect the model's performance?
- It can. Freezing prevents a part of the model from learning from new data, which might be necessary for optimal performance on a very different task. However, for related tasks, it often improves efficiency and prevents overfitting.
First seen on Subvolts Sep 17, 2025. Updated Sep 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.