AR Expert Shifts Focus Back to Mobile Phones After Vision Pro Experience

This is Lowpass, a newsletter exploring the dynamic intersection of technology and entertainment, delivered weekly to subscribers.

While working on the Vision Pro at Apple, Mark Drummond had a revelation that diverged from the company’s usual narrative. As the head of the Character Intelligence Team, he was involved in creating the Encounter Dinosaurs demo, which allows users to interact with lifelike prehistoric creatures through the headset.

“We collaborated with Jon Favreau,” Drummond notes, highlighting Apple’s ongoing partnership with the director of The Mandalorian. “Before we had the headset to take to Burbank, we relied on iPhones and iPads,” he recalls. Using mobile devices for demos was logical since VisionOS is essentially a variant of iPadOS. “It worked out well. We had a great experience with iPhone and iPad,” he adds.

From this experience, Drummond discovered that mobile devices were more than just temporary substitutes for the headset. “What we learned while exploring sources of surprise and delight with interactive characters in mixed reality is that the headset isn’t necessarily the best device for this,” he explains.

“I still think it’s an impressive piece of hardware,” Drummond acknowledges. However, he points out that headsets can create a sense of isolation, separating users from their surroundings and the people around them. “It can feel lonely,” he admits. In contrast, an AR app on a phone allows for shared experiences. “People can lean in over your shoulder,” he says.

After leaving Apple in 2023, Drummond turned his focus to mobile devices for augmented reality storytelling. For the past two and a half years, he has been developing an iPhone AR app called Pixi, designed for mobile-first interactive storytelling.

In essence, Pixi aims to create the AR equivalent of an email greeting card. When it launches soon, users will be able to choose an interactive character and scenario, add a personal message, and send it via iMessage or WhatsApp. Once the recipient opens the Pixi message, the character will appear in their real-world environment through the camera, engaging with them.

During a recent demo, Pixi showcased AR experiences featuring an animated cat and a robot that could tell jokes, play tic-tac-toe, or challenge users to a game of Whac-A-Mole right on their desk.

At first glance, these features might not seem revolutionary if you’ve used AR filters on Snapchat or other mobile AR apps. “If you’re not a little embarrassed by your first product, you launched too late,” Drummond remarks, hinting that the company plans to introduce more characters and advanced storylines in the future.

However, Pixi does include some intriguing interactive elements: for example, the cat comedian tells a series of dad jokes until the app detects the viewer smiling through the phone’s front camera. After that, the character thanks the viewer for their attention like a stand-up comic wrapping up a show and delivers a personalized greeting.

“To make a character feel present, it has to pay attention,” Drummond explains. This involves not only focusing on the viewer but also on their environment. “This level of attention is only achievable through on-device AI,” he adds. Pixi utilizes AI and machine learning to recognize facial expressions and can download custom ML models on the fly to identify objects and weave them into the narrative.

When mobile AR gained popularity with Snapchat filters and ARKit-based apps, developers had limited tools for interacting with the real world. Initially, apps could only detect surfaces to place virtual objects on top. Over time, AR applications improved with occlusion, allowing characters to hide behind real-world items, and some even introduced basic object detection.

Recent advancements in AI have the potential to enhance object recognition significantly, enabling apps to identify categories of objects, understand their functions, and integrate them into interactive experiences. Drummond envisions a future where Pixi’s AR greeting cards could work like this:

“Imagine I have a friend who’s a lawyer and just got promoted. I send her a Pixi featuring a golden retriever that appears on her office floor. The dog looks at her, sniffs around, and heads to a filing cabinet, fixated on a drawer. She thinks: ‘That’s just old tax returns.’ But the dog persists, looking adorable. When she opens the drawer, she finds a huge virtual dog biscuit with the message: ‘Congratulations, see you Friday.’”

Could this be achieved with a Vision Pro app? Possibly, but there are many factors holding the headset back, including its hefty $3,500 price tag. “It’s expensive,” Drummond admits, though he believes it’s worth it for the impressive technology, especially the displays.

However, the high cost has limited its market penetration, with estimates suggesting only 45,000 units were shipped during the last holiday quarter. “It doesn’t have great market penetration,” Drummond states.

Even those who own a Vision Pro may hesitate to use it for short clips or experiences. “You don’t always have it handy, and there’s the setup and teardown involved,” he explains. “We carry our phones with us all the time.”

It’s clear that the Vision Pro is just the beginning for Apple, which is reportedly developing its own smart and AR glasses. However, Drummond believes these glasses won’t replace smartphones anytime soon due to physical limitations. Phones can run complex machine learning models locally, while glasses will have much more restricted power and processing capabilities. As a result, they’ll likely serve as another display for notifications and directions, similar to the Apple Watch. “I think the Watch and Apple’s spectacles will fulfill similar roles,” he says.

This suggests that AR developers aiming to create interactive entertainment experiences may find that smartphones remain the best platform for the foreseeable future, a platform that continues to evolve and improve, thanks to advancements in AI.