Sign up for our daily and weekly newsletters to stay updated on the latest in AI advancements. Google introduced its highly anticipated AI system, Gemini, on Wednesday, claiming it could rival OpenAI’s GPT-4 in reasoning abilities. However, the launch quickly faced criticism for potentially overstating Gemini’s capabilities.
During a meticulously planned video demo, Google showcased Gemini processing visual data through a camera positioned above a desk, answering questions, and solving problems while a human assistant manipulated objects. The presentation suggested that Gemini could function as a smart digital assistant, capable of advanced conversations and helping with daily tasks.
Despite the impressive demo, tech experts analyzing the technology behind the scenes doubt that Gemini can meet Google’s high expectations. Google is releasing Gemini in three versions — Gemini Pro, Gemini Light, and Gemini Ultra. However, early reviews of the mid-range Pro version revealed on Wednesday show that it struggles with tasks considered basic for a top-tier AI system.
One early tester of the Bard update, Victor de Lucca, posted on social media that Gemini Pro failed to accurately list the 2023 Oscar winners, expressing disappointment with its performance. Others noticed discrepancies between Google’s benchmark claims and the actual performance of the publicly available Pro version. Developer Nick Dobos questioned the validity of Google’s comparison to an older version of GPT-4, suggesting the benchmarks might be misleading.
Additionally, the Gemini demo video faced scrutiny after a Google spokesperson admitted it was pre-recorded and narrated later, rather than a live demonstration. This incident highlights the challenges Google faces in promoting its AI systems. While tech enthusiasts focus on benchmark data and research papers, the general public is more influenced by aspirational videos promising groundbreaking technology.
This kind of disconnect has caused problems for tech companies before, such as in 2016 when Microsoft’s Tay chatbot was removed after it learned to use hate speech from Twitter users. Google has also faced criticism before; in September, reports indicated that Google Bard was still not meeting expectations even after major updates.
Google aims to bounce back by making Gemini more accessible to developers and researchers, allowing thorough testing. However, the rocky start indicates that the company still has work to do to ensure its AI assistant meets the high expectations set by its own marketing.