Humanity's Last Exam (Diamond)
Challenging LLMs at the frontier of human knowledge
HLE-Diamond, built with the Center for AI Safety (CAIS), is a refined subset of Humanity’s Last Exam (HLE) resulting from a year-long process of cleaning and refinement with input from research communities.
Introduction
In partnership with the Center for AI Safety, we addressed the problem of benchmark saturation by creating Humanity's Last Exam (HLE): the toughest, most subject-diverse, multi-modal questions we could assemble, designed to be the last academic exam of its kind for AI. HLE tests both depth of reasoning (eg. world-class mathematical problems) and breadth of knowledge across its subject domains, providing a precise measurement of model capability. At release in January 2025, frontier models scored in the single digits and systematically exhibited uncalibrated overconfidence in their answers.
HLE-Diamond is where that work lands now. It consists of 1,000 questions drawn from the existing HLE pool and its held-out reserve. High accuracy on HLE-Diamond would demonstrate that AI has achieved expert-level performance on closed-ended cutting-edge scientific knowledge. It would not alone suggest autonomous research capabilities or "artificial general intelligence."
See the linked full paper and dataset.
What changed in HLE-Diamond
Most of these changes were made incrementally over a year of HLE-Rolling updates. HLE-Diamond is where they land as a single release.
Size. 1,000 questions, down from 2,500. Every question comes from the existing HLE pool and its held-out reserve.
Two partitions. HLE-Diamond consists of 500 reasoning and 500 knowledge questions, measuring reasoning and expert knowledge, respectively. Final HLE-Diamond results are aggregated between the two partitions.
A frozen question set. HLE-Diamond is fixed, with stable question IDs, so a score from today is comparable to a score from next quarter.
Headroom. The strongest model we tested lands at 60.6%, well below the noise ceiling. We expect HLE-Diamond to carry signal for the next 6 to 12 months.
Main Results
| Model | Overall | Reasoning | Knowledge |
|---|---|---|---|
| GPT-6 Astra | 60.6% | 75.6% | 45.6% |
| Claude Opus 5.5 | 55.0% | 63.2% | 46.8% |
| Claude Fable 5.1 | 51.3% | 62.0% | 40.6% |
| GPT-6 Sol | 33.8% | 44.2% | 23.4% |
| Gemini 3.8 Flash | 34.3% | 38.6% | 30.0% |
| Muse Spark 1.3 | 25.4% | 31.6% | 19.2% |
| Grok 4.7 | 23.4% | 32.4% | 14.4% |
Overall, we observed a strong correlation between reasoning and knowledge partitions; models that perform better on reasoning consistently score higher on knowledge.
Metrics
We report accuracy on the reasoning and knowledge partitions, plus an overall accuracy across all 1,000 questions. Models are ranked on the leaderboard using overall accuracy.
We also use the model's own stated confidence to derive an RMS calibration error, using the implementation from Hendrycks et al., 2022 with the default hyperparameters provided. We want to emphasize calibration error as an important metric alongside accuracy.
Evaluation methodology
Evaluation is automatic. Models are prompted to give a final answer and an estimation of confidence using the system prompts (or user prompt when not configurable), following the setup from Wei et al., 2024.
Because HLE-Diamond uses closed-form solutions, we use an LLM judge as an automatic extractor and judge to compare the model response against the ground truth answer.
Evaluation with tools
HLE-Diamond questions are designed to be answerable in a closed-book setting, testing both reasoning and expert knowledge. Since HLE is also used to evaluate the capabilities of agentic systems, we outline our recommended settings for evaluating HLE-Diamond with tools here.
Acknowledgements
Humanity's Last Exam was a global collaborative effort developed in partnership with the Center for AI Safety. We extend our deepest gratitude to all participating question contributors and expert reviewers involved in creating and refining the dataset, and to the researchers whose feedback across HLE-Rolling shaped HLE-Diamond.
Performance Comparison
60.60±3.00Calib Err: 28.10
55.00±3.10Calib Err: 7.10
51.30±3.10Calib Err: 13.80
38.60±3.00Calib Err: 21.50
34.30±2.90Calib Err: 57.20
33.80±2.90Calib Err: 38.80
31.20±2.90Calib Err: 53.80
25.40±2.70Calib Err: 48.20
23.40±2.60Calib Err: 28.90
22.20±2.60Calib Err: 54.40
16.40±2.30Calib Err: 53.10
13.40±2.10Calib Err: 76.30
Rank (UB): 1 + the number of models whose lower CI bound exceeds this model’s upper CI bound.
CE: Calibration Error