Google Brings Multimodal Search To AI Mode

Google has announced the expansion of its AI mode feature, allowing users to upload an image and ask questions about it. This feature, known as multimodal search, enables users to receive detailed, context-rich answers complete with helpful links for further exploration.

Along with the expansion, Google has released a major update to AI Mode, including multimodal search capabilities powered by Google Lens and a customized version of Gemini, Google's advanced AI model. This enhancement allows users to take or upload an image, ask a related question, and receive rich, context-aware responses that include useful links for further exploration.

Unlike traditional visual search, this feature goes beyond simply identifying objects. AI Mode now understands image context holistically—analyzing relationships between items, textures, colors, shapes, and spatial layout. For instance, if a user uploads a picture of a bookshelf, AI Mode not only recognizes each book but also curates recommendations based on the titles, complete with shopping links and detailed insights.

“AI Mode builds on our years of work on visual search and takes it a step further,” says Robby Stein, VP of product for Google Search. “With Google Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes, and arrangements.”

According to Google, the update employs a "fan-out technique," in which it issues multiple queries about the image and any objects within it in order to provide "incredibly nuanced and contextually relevant" responses. That allows it to do things like identify books that are displayed within an image, issue suggestions for similar titles with positive ratings, and answer questions to further curate recommendations.