Collection
Collects Web news and social-media posts across five platforms
Abstract
We present the first benchmark capable of measuring the ability to forecast real-world social events using anonymized ground truth from Web news and social-media.
Our approach collects Web news and social-media posts across five platforms, distills each source into date-indexed timelines via a six-step agent-driven processing chain, and merges them into a unified chronology that separates factual events from public opinion. We apply an anonymization procedure that renders each real event arc into a counterfactual social world, ensuring the evaluation measures forecasting ability rather than the ability to search for the real event. We then automatically generate an agent-audited question bank supporting two orthogonal evaluation axes: probability calibration and temporal accuracy.
SocietyBench turns any one-line event topic into an anonymized simulated social world, and uses an agent-audited question bank to evaluate LLMs along two axes — probability calibration and temporal accuracy.
Four-Phase Pipeline
Our benchmark operates through a four-phase pipeline: collecting multi-source data, distilling timelines, merging and anonymizing them into counterfactual worlds, and evaluating models on probability calibration and temporal accuracy.
Per-source
Distills each source into date-indexed timelines
Merge
Merges streams into unified chronology separating facts from opinion
Evaluation
Anonymizes, generates questions, and scores on two axes
Three-Phase Anonymization
The anonymization pipeline renders each merged timeline into a counterfactual world before any candidate model sees it. The auditor loop stops at high = 0 and mid = 0, max 5 rounds.
Rule-based Replacement
Swap every entity against a table, then slide all dates by one shared offset — the gaps survive, the calendar does not.
LLM Adversarial Audit Loop
An auditor LLM hunts for anything still searchable; whatever it finds becomes a new rule and Phase 01 re-runs.
Consistency Check
Compare the result against the original and repair what the swaps broke.
Dual-Axis Evaluation
At each accepted prediction point, the candidate sees the context up to a cutoff date and answers two orthogonal question types.
AXIS 01 — CALIBRATION
Probability Calibration.
For an event E in a time window W, the model outputs P(E occurs in W) ∈ [0,1]. Scored by weighted MAE against binary ground truth.
50 uniform 50% predictor
100 perfect predictor
AXIS 02 — TEMPORAL
Temporal Accuracy.
For each in-window event, predict its calendar date. Scored by day-MAE against a 30-day bucket-midpoint baseline.
50 bucket-midpoint baseline
100 perfect timing
Leaderboard
DATA REFRESHED —
NEXT REVIEW —
SCHEMA v—
Want your model on this table? → see EXPAND THE BENCHMARK below.
| Loading… |
† Agents run on the Doubao-Seed-2.0-Pro base model; the delta against that base is what the row is measuring.
Per-event cells: Cal | Time. Higher is better; 100 = perfect, 50 = the trivial anchor (a uniform 50% predictor on calibration, a bucket-midpoint guesser on time). Per-event labels: PUB = Public Controversy · GEO = Geopolitical · TECH = Tech Policy · MKT = Markets · TRADE = Trade. Every score on this table is measured — this release contains no projected numbers.
Five anonymized arcs. Each card shows the arc description, prediction-point count, and the question volume that point supplies to the bank.
Deep Dive
Where the difficulty actually sits. The same answers regrouped four ways: by whether the truth is yes or no, by how late the cutoff is, by how far the event falls from it, and by what the question asks about.
Contribute
Everything you submit is reviewed and added to the public leaderboard.
01
Bring Your Model
Run our 5 events on your LLM or agent, submit your scorecard via PR. The team replays your raw answers with our scoring code — no honor system.
02
Bring a New Event
Use our public pipeline (web-crawl → media-crawl → merge → anonymize → questionbank) to construct a new anonymized event. Submit it via PR; if accepted it joins the official 5+.
03
Suggest a Topic
Have an event idea but no compute? Send us the topic and rough timespan. We'll run the pipeline on accepted suggestions and credit you on the contributor list.
Citation
@misc{societybench2026,
title = {SocietyBench: Forecasting Counterfactual Social-World Evolution},
author = {Wang, Zhenran and Bian, Zhonghan and Li, Jinsong and Qi, Zhangyang},
year = {2026},
note = {\url{https://github.com/co-minder/SocietyBench-codebase}}
}