First experience with Android XR: interacting with smart glasses and headsets in a real time, fluidly speaking to a person’s context

It is a normal Tuesday. I wore glasses in the room that looked like ordinary glasses with a group of people. One of them is speaking Spanish in front of me. I don’t speak Spanish. I can see her words being translated into English subtitles when I am in the air. I can see that she’s describing what I’m seeing in real time.

This is my first experience with Android XR — a new mixed reality OS designed for headsets and smart glasses, like the prototypes I’m wearing. A new generation of augmented reality devices that embody all our wildest dreams of what smart glasses have to offer is a big bet by GOOGLE.

“One thing I’m really confident about, something that’s not just different from before, is that Gemini is really that great,” says Kihwan Kim, EVP at Samsung Electronics, who nods furiously in agreement when I mention this. To Kim, it’s the ability to fluidly speak to Gemini and the fact that it understands a person’s individual context that opens dozens of different options for the way each person interacts with XR. “That’s why I clearly see that this headset will give more insight about what [XR] should be.”

Samsung’s headset feels like a mix between a Meta Quest 3 and the Vision Pro. The light seal is optional so you can let the world bleed in. It’s lightweight and doesn’t pinch my face too tightly. My ponytail easily slots through the top, and later, I’m thankful that I don’t have to redo my hair. At first, the resolution doesn’t feel quite as sharp as the Vision Pro — until the headset automatically calibrates to my pupillary distance.

At this point, I start to feel the same feelings as before. I’m walked through pinching to select items and how to tap the side to bring up the app launcher. It feels like a process like the Vision Pro’s. If I want, I can retreat into an immersive mode to watch YouTube and Google TV on a distant mountain. I have the ability to open apps, change them, and place them around the room. I’ve done this all before. This just happens to be similar to the search engine.

For the skeptic, it’s easy to scoff at the idea that Gemini, of all things, is what’s going to crack the augmented reality puzzle. Generative Artificial Intelligence is having a moment, but not always in a good way. Outside of conferences filled with tech evangelists, the field of artificial intelligence is viewed with suspicion and derision. But inside the Project Moohan headset or wearing a pair of prototype smart glasses? I can catch a glimpse of why Google and Samsung believe Gemini is the killer app for XR.

For me, it’s the fact that I don’t have to be specific when I ask for things. Usually, I get flustered talking to AI assistants because I have to remember the wake word, clearly phrase my request, and sometimes even specify my preferred app.

I can say, “I want to go to JYP Entertainment in Ulm” and Moohan will show me a building on the map. If my windows get cluttered, I can ask it to reorganize them. I do not need to reach for a finger. While wearing the prototype glasses, I watch and listen as Gemini summarizes a long, rambling text message to the main point: can you buy lemon, ginger, and olive oil from the store? I transitioned from speaking English to asking Japanese for the weather in New York and getting the answer in spoken and written Japanese.

I think of interactions with Gemini more than any other one. It’s how experiences can be built on top of them. I asked Gemini how to get somewhere and saw turn-by-turn text directions. The text turned into a map when I looked down. It’s very easy to imagine myself using something like that in real life.

But as cool as all that is, headsets can be a hard sell to the average person. Personally, I’m more enamored with the glasses demo, but those have no concrete timeline. (Google made the prototypes, but it’s focusing on working with other partners to bring hardware to market.) There are some cultural cues that still need to be established. There has to be a lot of apps for the average person, not just early adopters.

As I listen to Kim and Izadi talk, I want to believe. But I’m also acutely aware that all of my experiences were tightly controlled. I wasn’t given free rein to try and break things. I couldn’t take photos of the headset or glasses. At every point, I was carefully guided through preapproved demos that Google and Samsung were reasonably sure would work. I and other consumers are not able to believe until we can play with these things without being held back.

I can not deny that for an hour, even knowing that. I felt like Tony Stark and the people who created him. This example has molded our expectations for how AI and XR assistants should work. I have tried hundreds of headsets and smart glasses that promise to make what I see in movies look realistic. For the first time, I experienced something relatively close.

The choice of the term “XR” for the OS is maybe the most interesting part. There are many different words and jargons for this space, which include: virtual reality, mixed reality, extended reality, and others. The broadest of the terms is XR, which seems to be why it has been chosen by the search engine giant. “When we say extended reality or XR,” Samat said, “we’re really talking about a whole spectrum of experiences, from virtual reality to augmented reality and everything in between.”