Google's Project Astra: Gemini App Introduces Visual Interaction Features

At the Mobile World Congress (MWC) in Barcelona, Google unveiled exciting updates for its Gemini app, part of Project Astra. The latest version of Gemini, known as Gemini 2.0 Flash, allows users to engage in real-time conversations about their surroundings through the application on Android and iOS devices.

This updated app now supports 45 different languages, providing users with the unique ability to switch languages mid-conversation without the need to alter their device's language settings. Users can simply continue speaking in another language, and Gemini will seamlessly understand and respond.

Additionally, Google is set to roll out a live video input feature later this month, which is a pivotal aspect of Project Astra. Initially introduced at Google I/O, this feature will enable users to converse with the AI assistant while actively viewing their environment through smart glasses. However, for now, this functionality will be available on smartphones through the Gemini app. The AI will also remember previous discussions, allowing users to refer back to them later.

Alongside the live video capability, the app will include a screen sharing feature, permitting users to discuss what they are viewing on their devices with Gemini. For instance, users will be able to converse about potential purchases, like new jeans, directly through the app.

Initially, these visual AI features will only be accessible to Pixel and Samsung device users. The push to enhance AI assistants with visual capabilities is a trend among major tech companies. For example, OpenAI has developed a conversational AI agent named Operator, which can be instructed to make purchases using natural language. However, OpenAI has yet to release its visual capabilities, which are anticipated to be similar to those being introduced by Google.

Meta, on the other hand, has been focusing on visual assistance through its smart glasses, the Ray-Ban Meta Glasses, which allow users to interact with their environment and ask questions directly. However, these features are currently unavailable in the EU.