Evaluate LLM forecasting performance with structured effects
Measure how much structured future information improves LLM time series forecasting accuracy.
Compare the Mean Absolute Scaled Error (MASE) of LLM predictions using history-only prompts versus prompts that include structured effects (Macro and Micro effects) derived from realized future information.
Lower MASE indicates better performance. The percentage reduction in aggregate relative MASE quantifies the improvement from using structured effects.
GPT-5.6 Sol showed a 40.7% reduction in aggregate relative MASE when using structured effects compared to history-only prompting.
Evaluate LLM forecasting performance by comparing MASE with and without structured future effects. This method quantifies the benefit of contextual information for time series prediction tasks.
It directly measures the impact of incorporating verified future information on prediction accuracy, isolating the contribution of contextual data.
When the goal is not to evaluate forecasting models or when future information is unavailable or unreliable.
- GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation ↗arXiv · arXiv cs.AI · Oct 13, 2026 · paraphrased
First seen on Subvolts Oct 13, 2026. Updated Oct 15, 2026. Spotted a problem? Tell us.
- GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model EvaluationAnthropic · arXiv cs.AI · Oct 13, 2026
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- TemplatePrompt Library for Life Sciences TasksA collection of pre-written prompts designed for optimal results on common life sciences tasks. · 2 min
- ChecklistPre-computation Checklist for AI in Life SciencesItems to consider before using AI for complex life sciences tasks. · 5 min
- HeuristicAuxiliary environment signals improve LLM authorization accuracyProviding LLMs with auxiliary environment signals can substantially improve their accuracy in authorization decisions. · 30 min
- Case studyUsing LLMs to build tools for iOS location data analysisResearchers used an LLM to build a tool that decoded iOS location data, overcoming LLM analysis limitations. · 60 min
- ToolExtractor tool to recover protobuf schemas from Apple binariesPython tool to reliably recover protobuf schemas from Apple binaries for analysis. · 30 min
- Case studyUsing Screwtape to find confirmed failures of AI monitorsDemonstrates how a red-teamer can find confirmed failures of an AI monitor without developer infrastructure access. · 120 min
Questions people ask
- Which AI tools does this work with?
- Any current model. It was seen with Claude in the sources.
- Where did this come from?
- arXiv (arXiv cs.AI). Subvolts collected, normalized and credited it; the link above opens the original.