GAIA Benchmark (General AI Assistants)

What is the GAIA Benchmark (General AI Assistants)?

GAIA (General AI Assistants) is a cutting-edge benchmark designed to evaluate the performance of AI systems. It pushes AI capabilities beyond mere accuracy, focusing on the ability to handle complex, multi-layered queries. GAIA presents real-world scenarios that test fundamental AI abilities including reasoning, multi-modal processing, web navigation, and general tool utilization.

Frontier Model Leaderboard

The July 2024 leaderboard features frontier models that use a scratchpad for reasoning without any additional tools or external services.

The archived revision does not identify an external source, exact evaluation date, dataset split, prompts, evaluator, or complete harness. Preserve these scores as provenance-limited historical context. This scratchpad-only model setup is methodologically different from the tool-using agent submissions below, so the tables are not directly comparable.

The GAIA dataset is publicly available on Hugging Face. The benchmark questions are stored in the metadata.jsonl file within the dataset. Some questions are accompanied by additional files, which can be located in the same directory. The corresponding file for each question, when applicable, is identified by the file_name field in the metadata.

The GAIA benchmark comprises 466 human-designed and annotated questions. These text-based queries, sometimes accompanied by files like images or spreadsheets, cover a wide range of use cases from daily tasks to scientific inquiries. While conceptually simple for humans, these questions prove challenging for even advanced AI systems. The stark performance gap is evident: human respondents achieve 92% accuracy, while GPT-4 with plugins manages only 15%.

Those figures describe GAIA's original 2023 baseline; later agent systems have closed much of that gap.

GAIA's unique approach evaluates both the correctness of answers and the methods used to arrive at them. This holistic assessment is analogous to grading a student's problem-solving process, not just their final answer. The benchmark employs a tiered question system, with increasing levels of complexity and cognitive demands. It utilizes various metrics to gauge AI proficiency, including accuracy, reasoning capability, and response time.

GAIA's tasks require systems to coordinate reasoning, browsing, multimodal inputs, and tools across multiple steps. Official scoring uses quasi-exact matching on the final answer; it does not grade the reasoning trace or the method used to produce that answer.

The benchmark's tasks simulate real-world applications, effectively testing an AI's ability to comprehend and operate within human contexts. GAIA's philosophy represents a paradigm shift in AI benchmarking, focusing on tasks grounded in real-world interactions rather than isolated, task-specific evaluations. It eschews predefined APIs, instead emphasizing direct engagement with the real world.

This approach marks a significant departure from traditional AI benchmarks, potentially revolutionizing how we assess and develop AI assistants. By moving away from siloed, task-specific evaluations, GAIA aims to push AI systems towards more generalized, human-like problem-solving capabilities. Its comprehensive and realistic testing methodology provides a robust framework for understanding and improving AI performance in complex, real-world scenarios.

Archived GAIA agent snapshot (2024)

This archived table features agent systems that used tools in conjunction with models for enhanced reasoning and performance. The later historical-results section preserves and reconciles these same legacy identities against the July 2026 official data.

GAIA Test Leaderboard Snapshot (July 15, 2026)

The official GAIA leaderboard ranks submitted agent systems on the private-answer test set. The snapshot below was retrieved from the leaderboard's public results data on July 15, 2026. Overall score is exact-match accuracy across the test set. The level columns preserve the benchmark's difficulty breakdown: Level 1 contains the most direct tasks, while Levels 2 and 3 require progressively more steps and tool coordination.

These are agent-system results, not isolated base-model scores: each submission may combine models, prompts, search, code execution, and other tools. Submission dates describe when a result entered the leaderboard, not when every component was released. Some leading entries are opaque or omit organization and implementation details, and leaderboard inclusion should not be treated as an independent audit of the full system.

Historical GAIA agent results

The earlier version of this page highlighted the following agent systems. The first four rows below reproduce values still present in the official results data. The remaining eight restore the identities and overall scores recorded by the earlier page; they are historical values whose current-data verification is qualified below.

The current official results data still contains UK AI Safety Institute Internal, FRIDAY, FRIDAY_without_learning, Ceylon, Chamomile, and AutoGPT4, but now reports overall scores of 24.9169%, 24.2525%, 21.5947%, 16.9435%, 14.6179%, and 4.9834%, respectively. It contains two DIP rows scoring 4.3189% and 15.9468%, neither of which reproduces the archived 17.06%, and it no longer contains the exact GPT4 + manually selected plugins identity. The eight restored values should therefore be read as the earlier page's historical snapshot, not current official standings.

The archived scratchpad-only comparison above used a different setup, so its scores should not be merged into the agent leaderboard or treated as directly comparable.

Example GAIA Question and Response

The benchmark consists of 466 questions across three levels of complexity, defined by the number of steps required to solve the task. The questions are conceptually simple for humans yet challenging for most advanced AIs. For instance, human respondents obtain 92% accuracy, while GPT-4 equipped with plugins only achieves 15%.

A paper about AI regulation that was originally submitted to arXiv.org in June 2022 shows a figure with three axes, where each axis has a label word at both ends. Which of these words is used to describe a type of society in a Physics and Society article submitted to arXiv.org on August 11, 2016?

The evaluation in GAIA is automated, fast, and factual. Each question calls for an answer that is either a string (one or a few words), a number, or a comma-separated list of strings or floats, unless specified otherwise. There is only one correct answer, and evaluation is done via quasi exact match between a model's answer and the ground truth.

According to github, when was Regression added to the oldest closed numpy.polynomial issue that has the Regression label in MM/DD/YY?

However, GAIA does not evaluate the trace leading to the answer. Different paths could lead to the correct answer, and there is no obvious and simple way to grade these paths.

The Metropolitan Museum of Art has a portrait in its collection with an accession number of 29.100.5. Of the consecrators and co-consecrators of this portrait's subject as a bishop, what is the name of the one who never became pope?

The GAIA benchmark is designed to test the capabilities of AI systems in a way that is closer to real-world tasks, rather than focusing on tasks that are difficult for humans. The researchers behind GAIA believe that the successful resolution of GAIA would be an important milestone towards the next generation of AI.

To maintain the integrity of the benchmark, the researchers have released the questions while retaining answers to 300 of them to power a leaderboard. The remaining 166 questions and answers were released as a development set.

As of the latest data, the leading GAIA score belongs to GPT-4 with manually selected plugins, at 30% accuracy. This performance disparity between humans and AI systems underscores the challenges that AI still faces in handling tasks that are simple for humans but complex for AI systems.

In the separately dated July 15, 2026 test snapshot above, CustomGPT.ai Research Lab v44 led submitted agent systems with a 93.3555% score. Differences in tools, prompts, implementation disclosure, and evaluation dates limit direct comparisons with the earlier plugin baseline.

The goals and structure of the GAIA

Designed to overcome the limitations of Large Language Model (LLM) evaluations, GAIA encompasses a range of use cases, from everyday tasks to scientific inquiries. The benchmark's philosophy contrasts with traditional AI benchmarks by focusing on tasks that are deceptively simple for humans but intricate for AI, aiming to push AI towards the next generation of capabilities.

The GAIA benchmark assesses AI systems against real-world tasks through 466 questions that test fundamental abilities like reasoning, multi-modality handling, web browsing, and tool-use proficiency. These tasks, while straightforward for humans, present significant challenges for AI. The benchmark's automated, swift, and precise evaluation process requires answers in the form of strings, numbers, or comma-separated lists.

To foster competition and progress, the GAIA leaderboard is powered by 300 retained questions, with the remaining 166 questions serving as a development set. Currently, GPT-4 with manually selected plugins leads with a 30% accuracy rate, highlighting the gap between human and AI task performance and the ongoing challenges in AI's evolution.

The separately dated 2026 agent-system snapshot above records later submissions under different tool and scaffold configurations.

What are the key features of the GAIA benchmark?

The GAIA (General AI Assistants) benchmark is a framework designed to evaluate AI systems, particularly their ability to function as general assistants. Here are its key features:

• Real-World Questions — GAIA includes 466 human-designed and annotated questions that are text-based and may include files like images or spreadsheets. These questions are intended to reflect real-world challenges.

• Automated and Factual Evaluation — The benchmark is structured for automated, fast, and factual evaluation. Answers are expected to be strings, numbers, or lists, with only one correct answer for each question, allowing for quasi-exact match evaluation.

• Levels of Difficulty — The benchmark is structured around three levels of difficulty, each representing a more sophisticated understanding and cognitive demand.

• Fundamental Abilities — GAIA tests for fundamental abilities such as reasoning, multi-modality handling, web browsing, and tool-use proficiency. These abilities are crucial for AI systems to navigate complex, layered queries.

• Performance Disparity — There is a significant performance gap between humans and AI on GAIA, with humans achieving 92% accuracy compared to 15% for GPT-4 equipped with plugins. This contrasts with other benchmarks where AI may outperform humans in specialized domains.

• Focus on Process — GAIA evaluates not only the correctness of the answers but also the process by which the AI arrives at those answers, akin to assessing a student's work.

• Multi-step process requirements — GAIA questions require reasoning and tool use across several steps, while official scoring evaluates only the final answer.

• Real-World Interaction — Unlike other benchmarks that may focus on an AI's ability to use specific APIs, GAIA emphasizes interactions with the real world, which is considered a more general and challenging approach.

• Milestone for AI Research — Solving GAIA is seen as a significant milestone in AI research, indicating progress towards Artificial General Intelligence (AGI).

These features make GAIA a comprehensive and challenging benchmark that aims to push the boundaries of what AI systems can achieve in terms of general assistance and real-world problem-solving.

How does the GAIA work?

The GAIA benchmark requires AI systems to demonstrate a variety of fundamental abilities to solve its questions effectively. These abilities include:

• Reasoning — The capacity to process information and make inferences or deductions based on the given data.
• Multi-modality handling — The ability to interpret and integrate information from various modalities, such as text, images, and spreadsheets.
• Web browsing — The skill to navigate the internet to find information that can help in answering questions.
• Tool-use proficiency — The general capability to utilize tools, which could include software applications or online services, to perform tasks or solve problems.

These abilities are essential because GAIA's questions are designed to be conceptually simple for humans but challenging for AI, requiring more than just retrieval of information from training data. They necessitate an understanding and operation within the human world, simulating real-world applications and interactions.

What are some examples of multi-modality handling in GAIA benchmark questions?

The GAIA benchmark includes questions that require multi-modality handling, which means the AI must be able to interpret and integrate information from various modalities, such as text, images, and spreadsheets. The GAIA benchmark is designed to be challenging for AI systems, requiring them to demonstrate a range of fundamental abilities, including multi-modality handling, to effectively answer the questions.

What is the difference between GAIA and other AI benchmarks?

The GAIA benchmark distinguishes itself from other AI benchmarks in several significant ways:

• Real-World Focus — GAIA emphasizes real-world questions that require fundamental abilities like reasoning, multi-modality handling, web browsing, and tool use. These tasks are conceptually simple for humans but challenging for advanced AI systems, unlike benchmarks that focus on tasks difficult for humans or those testing specific model capabilities.

• Holistic Evaluation — Instead of specifying APIs, GAIA relies on real-world interactions. This approach evaluates general AI capabilities rather than proficiency with specific APIs, providing a more comprehensive assessment of AI systems' real-world applicability.

• Significant Performance Gap — GAIA reveals a stark contrast between human and AI performance. Human respondents achieve 92% accuracy, while GPT-4 with plugins manages only 15%. This disparity is particularly noteworthy given the recent trend of large language models (LLMs) surpassing human performance in specialized domains like law or chemistry.

• Current leaderboard context — Later agent-system snapshots show that the historical gap has narrowed under newer tool and scaffold configurations.

• Comprehensive Cognitive Assessment — GAIA goes beyond simple task completion, assessing a broad spectrum of cognitive abilities. It employs various metrics including accuracy, reasoning quality, and response time to provide a nuanced evaluation of AI proficiency.

• Revolutionary Approach — GAIA represents a paradigm shift in AI evaluation, moving away from siloed, task-specific methods towards a more holistic assessment of AI assistants' capabilities.

By emphasizing real-world interaction tasks, GAIA departs from conventional AI benchmarks that focus on narrow, task-specific challenges, offering a more robust and realistic evaluation of AI systems.

Benefits of GAIA

GAIA (General AI Assistants) is a benchmark that evaluates AI Assistants' performance on real-world tasks and scenarios. Its key advantages include:

• Assessment of Human-like Reasoning — GAIA challenges AI systems with real-world questions that require fundamental abilities such as reasoning, multi-modal processing, web navigation, and tool utilization. This approach tests the AI's capacity to mimic human-like cognitive processes.

• Promotion of Human Values — By incorporating tasks that require empathy, creativity, and ethical judgment, GAIA encourages the development of AI systems aligned with core human values.

• Tiered Evaluation Structure — Questions are organized into progressive difficulty levels, allowing for a nuanced assessment of AI capabilities across varying cognitive demands.

• Comprehensive Performance Metrics — GAIA evaluates not only the correctness of answers but also the methodologies employed to reach those answers, providing a more holistic view of AI performance.

• Benchmark for AI Progress — Success in GAIA represents a significant milestone in AI development. Currently, the highest GAIA score is held by GPT-4 with manually selected plugins, achieving 30% accuracy. This benchmark serves as a clear indicator of the current state and future potential of AI systems.

• Current leaderboard context — The separately dated leaderboard above retains later agent-system results under their reported protocols.

What are the limitations of GAIA?

GAIA's evaluation methodology has several key limitations:

• Focus on Final Answer — GAIA assesses only the end result, disregarding the various approaches an AI might employ to reach that conclusion. This means diverse problem-solving methods yielding the same correct outcome are not differentiated or evaluated.

• Reproducibility Challenges — Models behind closed APIs may evolve over time, potentially rendering evaluations non-reproducible. For instance, ChatGPT plugins and their functionalities frequently change and are not yet accessible via the GPT Assistants API.

• Configuration Drift — The exact model version and tool integrations used to produce a given score can change or become unavailable.

• Single Correct Response Constraint — While robust against token generation randomness, GAIA's evaluation of only final answers that admit a single correct response may limit its applicability in scenarios where multiple valid solutions exist.

• API Proficiency Assessment Gap — GAIA's focus on real-world interactions, while valuable, may not comprehensively evaluate an AI system's proficiency with specific APIs.

• Temporal Relevance — The benchmark's pertinence may decrease over time due to changes in training data or the availability of online resources it relies upon.

• Significant Performance Disparity — GAIA reveals a substantial gap between human and AI performance, with human respondents achieving 92% accuracy compared to only 15% for GPT-4 with plugins. This underscores the current limitations of AI in comprehending and addressing complex, real-world tasks.

• Current leaderboard context — Later agent-system snapshots show that this historical gap has narrowed under different tool and scaffold configurations.

These limitations highlight the challenges in creating a comprehensive benchmark for evaluating AI systems' capabilities in real-world scenarios, emphasizing the need for continued refinement and development of evaluation methodologies.