Compressing long contexts into fixed-size states
Avoid compressing very long sequences into fixed-size states, as this loses rich information needed for tasks like document understanding.
Employ architectures like Titans that combine speed with the ability to handle massive contexts by actively updating memory, or use methods that retain richer information.
Compressing extremely long contexts into a fixed-size state, as done by some efficient RNNs and SSMs.
Fixed-size compression cannot adequately capture the rich information present in very long sequences, limiting the model's understanding and accuracy for complex tasks.
Avoid compressing long sequences into fixed-size states, as this leads to information loss. Instead, use architectures that can actively update and manage memory, preserving the rich details necessary for tasks like full-document understanding or genomic analysis.
- Titans + MIRAS: Helping AI have long-term memory ↗Google Research · Google Research Blog · Dec 4, 2025 · paraphrased
First seen on Subvolts Dec 4, 2025. Updated Sep 11, 2026. Spotted a problem? Tell us.
- Titans + MIRAS: Helping AI have long-term memoryGoogle · Google Research Blog · Dec 4, 2025
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- HeuristicMitigate alignment tax with mixed data fine-tuningMix original pre-training data into RL fine-tuning to reduce performance drops on academic tasks. · 10 min
- HeuristicCombine statistical covariates with foundation model embeddingsSynergistic value from fusing structured statistical data with latent foundation model representations. · 5 min
- HeuristicUse Constitutional Classifiers to block AI jailbreaksDefend AI models against jailbreaks using input/output classifiers trained on principles. · 10 min
- Anti-patternAssume LLM state policy answers are correct without verificationDo not trust LLM answers on state policy without cross-checking, as they may substitute data from other states. · 5 min
- HeuristicDiscovery-Execution framework predicts mathematical reasoning scalingUse a Discovery-Execution framework to predict how AI mathematical reasoning scales with compute. · 15 min
- HeuristicMeasure conditional execution for longer reasoning predictionsShort-budget success rates don't always capture longer reasoning potential; measure execution. · 10 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with in the sources.
- Where did this come from?
- Google Research (Google Research Blog). Subvolts collected, normalized and credited it; the link above opens the original.