If you want to receive the latest updates and exclusive content on industry-leading AI, consider joining our daily and weekly newsletters.
A new AI benchmark called GAIA has been developed to assess if chatbots like ChatGPT can exhibit human-like reasoning and handle everyday tasks proficiently. This benchmark was created by researchers from Meta, Hugging Face, AutoGPT, and GenAI. GAIA presents real-world questions requiring fundamental skills such as reasoning, handling different types of media, web browsing, and using various tools effectively, according to a paper published on arXiv.
These questions are described as simple for humans but challenging for advanced AIs. In tests, humans scored 92 percent, while GPT-4 with plugins managed just 15 percent. This gap highlights a significant discrepancy, as recent large language models have often outperformed humans in professional domains like law or chemistry.
Instead of focusing on tasks that are tough for humans, GAIA aims to evaluate whether AI systems can handle tasks with the same robustness as the average human. The researchers created 466 real-world questions with clear answers. Three hundred of these answers are kept private for powering a public GAIA leaderboard, while 166 were released as a development set.
Lead author Grégoire Mialon of Meta AI stated that solving GAIA would mark a significant milestone in AI research. So far, the highest GAIA score belongs to GPT-4 with selected plugins, achieving 30% accuracy. According to the researchers, mastering GAIA could indicate progress toward artificial general intelligence (AGI).
GAIA shifts the focus from testing AIs on complex professional exams to everyday questions, such as which city hosted the 2022 Eurovision Song Contest according to its official website or the number of images in the latest 2022 Lego Wikipedia article. The researchers believe that achieving AGI will depend on systems demonstrating robustness similar to that of an average human.
The release of GAIA could influence the future direction of AI research, emphasizing human-like competence in everyday tasks. If future AI systems can demonstrate human-level common sense, adaptability, and reasoning as measured by GAIA, this could indicate the practical achievement of AGI, accelerating the deployment of various AI services and products.
However, the authors caution that today’s chatbots still have much ground to cover in solving GAIA. Current shortcomings in reasoning, tool use, and handling diverse real-world situations highlight their limitations.
As researchers work on meeting the GAIA challenge, their progress will reveal how capable and trustworthy AI systems can become. Benchmarking efforts like GAIA also prompt reflection on designing AI systems that prioritize human values, such as empathy, creativity, and ethical judgment.
The GAIA benchmark leaderboard is available for viewing, showcasing which next-generation large language model currently performs best in this evaluation.