FutureX is a dynamic real-time future prediction evaluation benchmark designed for LLM agents jointly released by research teams from ByteDance, Fudan University, Stanford University, and Princeton University. Future event questions are collected in real time from 195 high-quality websites through a semi-automated pipeline, and real results are automatically obtained for scoring after the event is resolved, effectively avoiding data pollution.

FutureX’s in-depth review: a touchstone of LLM agent’s future prediction capabilities

Core parameters and statistics

Project	Details
Product Name	FutureX
Product Type	LLM Agent Evaluation Benchmark (Research Benchmark)
Delivery form	arXiv paper + automated evaluation pipeline
Supported languages	English
Target users	AI researcher LLM developer, agent platform
Data source	195 high-quality websites
Issue areas	Politics, economics, finance, sports, entertainment
Question type	Single choice, multiple choice, open ranking, numerical prediction
Difficulty level	Level 4 (Level 1-4)
Assessment frequency	Daily automatic updates
Output size	~500 events per week

FutureX is not a commercial SaaS tool in the traditional sense, but a dynamic real-time evaluation benchmark for LLM agents. Its core contribution is to solve the long-standing "data pollution" problem in LLM evaluation - by focusing on future events that have not yet occurred, ensuring that the answers to the model have not yet been made public when making predictions, thereby obtaining fair and uncontaminated evaluation results.

User and market recognition

As an academic research project, FutureX has been recognized by research teams from several top institutions:

• Joint publisher: Researchers from four institutions including ByteDance, Fudan University, Stanford University, and Princeton University participated.
• Model Coverage: In the paper, 25 LLM/Agent models were systematically evaluated, including mainstream models such as Grok-4, Gemini-2.5-flash Deep Research, and GPT series DouBao-Seed1.6-Thinking.
• arXiv included: The paper is published on arXiv (2508.11987) and has been updated to version v3 as of September 2025.

Unlike traditional static benchmarks (such as MMLU, GSM8K), the dynamic nature of FutureX enables it to continuously track the evolution of model capabilities without worrying about the "exam-oriented education" effect caused by test set leaks.

Cost advantage

Cost Dimension	Description
C-side/Researcher	Totally free. Papers, evaluation data, and construction methodologies are all open source.
API / Developer	No API fees. There is no usage fee for the benchmark itself, and researchers can build their own evaluation pipelines by referring to its methodology.
Enterprise / Privatization	Enterprises can refer to FutureX’s methodology to build an internal evaluation system without paying license fees.

• Hidden Cost: Deploying and running the complete FutureX evaluation pipeline requires a certain amount of engineering investment - including configuring crawling of 195 data sources, event template generation, model prediction scheduling, etc. For teams that lack infrastructure, this part of the labor cost cannot be ignored.
• Hidden benefits: The evaluation results that avoid data pollution can significantly reduce the trial and error cost of model selection decisions, which is especially suitable for fields such as finance and political analysis that have high requirements for real-time prediction capabilities.

Main functions

• Dynamic Real-Time Assessment: Future incident questions are collected in real time from 195 high-quality websites through a semi-automated pipeline, and real results are automatically obtained for scoring after the incident is resolved. Unlike static benchmarks, which are updated annually, FutureX's problem sets are dynamically updated daily.
• No Data Contamination Guarantee: Since the evaluation questions are all based on events that have not happened yet, the model cannot "cheat" by memorizing training data. This fundamentally solves the most difficult test-set contamination problem in LLM evaluation.
• Multi-type task support: Covers four question types: single-choice, multiple-choice, open ranking and numerical prediction, and comprehensively evaluates the performance of the model on different prediction tasks. From "the rise or fall of a certain stock next week" to "the probability of the election result of a certain country", it covers a variety of prediction scenarios.
• Four levels of difficulty grading: From Level 1 (simple fact-based multiple choice questions) to Level 4 (super agent level - open questions that require in-depth research and multi-source information synthesis), the system evaluates the model's progressive reasoning capabilities.
• Large-scale cross-sector coverage: Collecting questions from 195 high-quality websites, covering multiple fields such as politics, economics, finance, sports and entertainment, generating approximately 500 events per week, it is currently the largest and most diverse real-time future prediction benchmark.

Model and version evolution

FutureX is released as an arXiv paper and currently has three versions:

• v1 (2025-08-16): Initial version, defining the overall architecture and evaluation methodology of the FutureX benchmark, and conducting preliminary evaluations of 25 LLM/Agent models.
• v2 (2025-08-19): Optimized the evaluation process and data presentation, and corrected some experimental data.
• v3 (2025-09-05): The latest version, updated with evaluation results of more models (including Grok-4, Deep Research, etc.), and more in-depth analysis of failure modes and performance bottlenecks.

The core architecture of the three versions remains consistent, with the main difference being the gradual expansion of model coverage and experimental depth. The official version management page is not provided independently from the paper. Subsequent updates shall be subject to the arXiv page.

Technical advantages

Architecture Link: FutureX’s core evaluation process is as follows:

LLM Agent → Event acquisition → Information collection (search/browse) → Prediction output → Waiting for event resolution → Comparison of real results → Score
                ↑ |
                └──────── Daily automated pipeline: 195 website crawling + template generation ───────────────┘

• Automated Pipeline: Use AIME agent to automatically collect a large number of relevant website URLs, filter out 195 high-quality sites through LLM + manual review, generate event templates, and achieve daily automatic planning and evaluation.
• Anti-Taint Design: The biggest problem with traditional benchmarks is that the model may have seen test questions in the training data. FutureX essentially circumvents this flaw by focusing on future events—the model cannot predict what has not happened yet.
• Real World Challenge Simulation: FutureX not only evaluates the model's memory ability, but also tests its advanced cognitive skills such as information collection, data synthesis, probability trade-offs, and causal reasoning, placing the model in the real-world information flow.
• Fine-grained failure analysis: Provides in-depth analysis of model failure modes, including vulnerability to fake websites, time validity judgment, etc., providing clear optimization directions for model improvement.

How to use

FutureX, as a research benchmark, is used differently than commercial tools:

Purpose	Description
Paper reading	Directly access the arXiv page to read the complete technical report
Methodology reference	The paper describes the construction process of the evaluation pipeline in detail and can be reproduced
Model self-evaluation	Refer to FutureX’s problem collection and evaluation methodology to build your own evaluation system
Data citation	Paper data can be used as a reference for model performance comparison

Typical research path: Read arXiv papers → Understand the evaluation methodology → Build an evaluation system with reference to pipeline design → Connect your own model for evaluation → Compare the baseline results in the paper.

It should be noted that FutureX is not a SaaS product and does not provide a ready-to-register online evaluation platform. Its value is mainly reflected in methodological reference and comparison of experimental results.

Product Pricing

• C-side/Researcher: completely free. The full text of the paper and evaluation data are available through arXiv.
• API/Developer: No paid plans. The research team does not provide a commercial API and all content is published as academic open access.
• Enterprise / Privatization: Enterprises can use FutureX’s methodology to build internal evaluation systems free of charge. The paper adopts the CC BY-NC-SA 4.0 license agreement, which is free for non-commercial use. Commercial use needs to confirm the compliance boundaries.

Application scenarios

• Model Selection and Comparison: Conduct a fair comparison of future prediction capabilities among models such as Grok-4, Gemini-2.5-flash, GPT-4o, DouBao-Seed1.6, etc. to provide reference for purchasing decisions.
• Agent Capability Assessment: Evaluate the performance of tool-enhanced agents such as Deep Research Agent on real-world prediction tasks, rather than focusing solely on traditional QA or code generation capabilities.
• Financial forecast model verification: Evaluate the model's ability to predict financial events such as stock prices and economic indicators, and assist financial institutions in selecting high-performance analysis agents.
• Model iterative optimization: Provide clear directions for model improvement through fine-grained failure mode analysis (fake website identification, time validity judgment).
• Academic Research Reference: As a cutting-edge exploration of LLM evaluation methodology, it provides benchmark design and implementation reference for subsequent research.

Applicable people

• AI researchers and scholars: Researchers who are concerned about LLM evaluation methodology, data pollution issues, and the boundaries of agent capabilities can use FutureX as an evaluation tool or comparison benchmark.
• Model Developer: Teams who are training or fine-tuning LLM/Agent models can use FutureX's methodology to test the model's true capabilities on future prediction tasks.
• Financial/Policy Analysis Institution: Professional institutions that need to evaluate the performance of AI on real prediction tasks can refer to FutureX’s experimental conclusions to assist decision-making.
• Not suitable for crowds:
  • End users looking for ready-to-use commercial AI tools (FutureX is a research benchmark, not a SaaS product);
  • Teams who need an out-of-the-box evaluation platform (currently no online demo or UI interface);
  • Scenarios where there is no demand for real-time news predictions (FutureX focuses on future event predictions and is not suitable for traditional evaluations such as knowledge Q&A or code generation).

Summary and Outlook

FutureX represents an important direction in the field of LLM evaluation - moving from static, potentially contaminated test sets to a dynamic, real-time, unpredictable evaluation paradigm. It solves the stubborn problem of data pollution through exquisite pipeline design, and provides a more reliable yardstick for the evaluation of the real ability of intelligent agents.

Misfit Boundary: FutureX is not a general LLM benchmark, it focuses on the specific capability dimension of future prediction. For other ability dimensions such as code generation, mathematical reasoning, and long text understanding, it still needs to be comprehensively evaluated in conjunction with traditional benchmarks. In addition, FutureX currently only supports English questions, and Chinese localization has not yet been covered.

Procurement/Adoption Risk Assessment: For companies considering adopting the FutureX methodology, the following risks need to be noted: ① Operation and maintenance costs of the assessment pipeline - Continuously crawling 195 data sources and keeping template updates requires stable engineering investment; ② Field coverage bias - Current data sources are mainly Western mainstream media and financial data, with limited coverage of Asian markets and minority language fields; ③ Academic license restrictions - CC BY-NC-SA 4.0 The license has additional restrictions on commercial use, and companies need to complete a compliance review before going online. It is recommended to fully understand the methodology and data boundaries through arXiv papers, and then evaluate whether it is worth investing engineering resources to build an internal evaluation system.

Related tools: CrewAI, langchain

Version Info

• FutureX v3 (arXiv 2508.11987v3) ：The third edition has been revised to update experimental results and model evaluation data, including more evaluation results of agent models. （2025-09-05）
• FutureX v2 (arXiv 2508.11987v2) ：The second edition has been revised to optimize the assessment process and data presentation. （2025-08-19）
• FutureX v1 (arXiv 2508.11987v1) ：The initial version releases the FutureX benchmark definition, dataset construction methodology and preliminary evaluation results of 25 models. （2025-08-16）