futuresearch benchmarks
Want to run on our benchmarks? Please contact us at evals@futuresearch.ai.
FutureSearch evaluates our research agents on the following benchmarks:
Bench to the Future 3 (BTF-3)
BTF-3 is the third edition of our pastcasting benchmark: 1,007 resolved forecasting questions — 759 binary and 248 numeric — researched and forecast against a frozen web corpus. Paper and dataset to follow.
BTF-3 Leaderboard
Evaluated: June 2026
All scores are on the Brier scale; lower is better, and the best score in each column is bolded.
Agent | Pooled score(n=1,007) | Binary(Brier, n=759) | Numeric(RPS, n=248) |
|---|---|---|---|
1FutureSearch SOTA* | 0.120 [0.109–0.131] | 0.115 [0.101–0.129] | 0.137 [0.123–0.150] |
2Claude Fable 5 (high) | 0.126 [0.114–0.137] | 0.124 [0.109–0.140] | 0.130 [0.117–0.144] |
3Claude Opus 4.8 (xhigh) | 0.127 [0.116–0.138] | 0.126 [0.112–0.141] | 0.130 [0.117–0.143] |
4GPT-5.5 (agent SDK)‡ | 0.127 [0.118–0.136] | 0.129 [0.118–0.140] | 0.122 [0.109–0.135] |
5Claude Opus 4.8 (high) | 0.134 [0.123–0.145] | 0.128 [0.114–0.143] | 0.152 [0.138–0.165] |
6Claude Opus 4.8 (agent SDK)‡ | 0.135 [0.123–0.147] | 0.132 [0.117–0.147] | 0.145 [0.131–0.160] |
7Claude Opus 4.6† | 0.138 [0.127–0.150] | 0.134 [0.120–0.149] | 0.148 [0.133–0.163] |
8Claude Opus 4.7† | 0.140 [0.128–0.152] | 0.134 [0.120–0.150] | 0.157 [0.140–0.174] |
9GPT-5.5 (high) | 0.140 [0.131–0.150] | 0.140 [0.128–0.152] | 0.141 [0.127–0.154] |
10Gemini 3.5 Flash† | 0.155 [0.143–0.168] | 0.156 [0.141–0.172] | 0.154 [0.138–0.170] |
Binary questions are scored by the Brier score (mean squared error of the forecast probability), numeric questions by a normalized ranked probability score (RPS), which generalizes the Brier score to distributional forecasts. The pooled score is the mean across all questions.
Brackets are 95% confidence intervals, computed by percentile bootstrap (5,000 resamples of the question set).
* FutureSearch SOTA synthesizes forecasts from multiple FutureSearch agent runs. † Run with an earlier version of our forecasting agent. ‡ Self-driving run via the model vendor's agent SDK (Claude Agent SDK / OpenAI Agents SDK) instead of our forecasting agent. Claude Fable 5 (high) is missing 23 binary questions (n=736) and 3 numeric questions (n=245). Claude Opus 4.7 is missing one numeric question (n=247).
Pairwise comparisons
Paired bootstrap on pooled scores
Each cell is the difference in pooled score (row − column) on the questions both agents forecast; negative (green) means the row agent is more accurate. Bold, bordered cells are statistically significant (two-sided paired-bootstrap * p<.05, ** p<.01, *** p<.001); grey cells are not. Hover a cell for the 95% confidence interval, p-value, and shared question count.
Deep Research Bench (DRB)
DRB benchmarks how well LLM agents do research on the web. Each of the 0 diverse, real-world tasks provides 10-100k webpages stored offline for search and reasoning, accompanied by carefully curated answers.
DRB Leaderboard
Last updated:
Agent | Score | Cost ($) | Runtime (s) |
|---|---|---|---|
Scores averaged first per task category (radar chart), then across all tasks (table). Runtime is estimated from ReAct steps, not wall-clock time.
Papers
Bench to the Future 2 (BTF-2)
BTF-2 evaluates agents on 1,417 hard forecasting questions. Agents research and forecast offline against a frozen 15M-document corpus. Rationales and reasoning traces are evaluated for strategic reasoning.
BTF-2 Leaderboard
Last updated: 2026-04-20
Agent | Brier (accuracy) | Calibration | Refinement |
|---|---|---|---|
| FutureSearch Agent | 0.119 | 0.002 | 0.081 |
| Opus 4.6 Agent | 0.130 | 0.005 | 0.075 |
| Gemini 3.1 Pro Agent | 0.141 | 0.012 | 0.069 |
| GPT-5.4 Agent | 0.152 | 0.010 | 0.056 |
| Grok 4.20 Beta Agent | 0.165 | 0.003 | 0.039 |
Brier scores on 1,417 pastcasting questions (lower is better). The FutureSearch Agent is an ensemble significantly more accurate than any single frontier agent. Radar chart shows CHAMPS KNOW strategic emphasis (Borda scores, 8 of 10 dimensions).
Papers
Datasets
Evaluating forecasts on S&P 500 stock returns
View all 500 companies →On August 5, 2025, FutureSearch research agents forecast revenue, margin, and shareholder payout for each S&P 500 company through 2035.
To evaluate accuracy, we turned those forecasts into a dollar-neutral, industry-neutral paper portfolio, going long the most-undervalued name and short the bottom-half within each GICS industry.
The chart on the left is how this portfolio has done; NVIDIA on the right is a summary view of the forecasts on one company.
For about three weeks, from August 8 through September 2, 2025, this same Aug 5 batch was the publicly accessible Stockfisher product (top 10 companies free, the rest behind paid tiers), before being refreshed with a newer batch of forecasts.
Read more about our methodology for forecasting stock returns: can AI forecast stocks through fundamental analysis?, calculating intrinsic value from scratch, and a superforecasting approach to stock fundamentals.
Industry-neutral, dollar-neutral paper portfolio: long the most undervalued name and short the bottom half within each GICS industry (51 long bets, 231 short bets). Both baskets plotted as underlying returns; Net = Long − Short. No commissions, borrow cost, or dividends. Not investment advice.
Nvidia
(NVDA)Semiconductors & Semiconductor EquipmentForecasts as of August 6, 2025
RetroSearch
DRB and BTF-2 use RetroSearch, a system designed to serve agents a frozen, previously scraped version of the internet instead of the live pages, allowing reproducible runs even as the internet changes, and enabling forecasting tasks to be run as "pastcasting".
RetroSearch aims to emulate Google search (specifically, the Serper search API) as closely as possible, so as to minimize differences between live and "retro" agent runs. A single RetroSearch search query follows the following steps:
- Run a live Serper search for the query
- Look up pages obtained from live search in the RetroSearch database and other archive sources
- If the page is not found in the RetroSearch database, remove it from the results
- Write new snippets from a sample of page content using a simple LLM
- Return the results in the original format of the Google results
This approach ensures a search experience for agents that is consistent with real search, but backed exclusively by pages we have a frozen candidate for. The following diagram from the paper illustrates the process:
