How does Waymo’s technology handle extreme weather or unexpected obstacles?

An autonomous vehicle cruises along a desolate highway when suddenly, a colossal tornado looms in the distance. What happens next for this driverless ride?

This scenario is just one of many that Waymo can recreate in its newly developed “hyper realistic” virtual environment, crafted with assistance from Google’s DeepMind. The Waymo World Model utilizes Genie 3, Google’s cutting-edge AI world model capable of generating interactive virtual spaces based on text or image prompts. But Genie 3 isn’t limited to creating subpar imitations of classic games; it can also produce photorealistic, interactive 3D environments specifically designed for the challenges of driving, according to Waymo.

Simulation plays a vital role in the development of autonomous vehicles, allowing developers to test their creations in a wide range of situations, including those that are extremely rare, all without the risk of endangering passengers or pedestrians. AV companies leverage these virtual settings to conduct extensive testing, accumulating millions, if not billions, of miles in the process, all aimed at better preparing their vehicles for any potential “edge case” they might face in the real world.

What kinds of edge cases is Waymo exploring? Beyond the tornado scenario, the company can simulate a snow-covered Golden Gate Bridge, a flooded suburban street with floating furniture, a neighborhood on fire, or even an unexpected encounter with a rogue elephant. In each of these situations, the Waymo robotaxi’s lidar sensors create a 3D representation of the environment, including any obstacles in its path.

“The Waymo World Model can generate virtually any scene—from everyday driving to rare, long-tail scenarios—across multiple sensor modalities,” the company explains.

Waymo highlights that Genie 3 is particularly suited for crafting virtual worlds for its robotaxis, thanks to three distinct mechanisms: driving action control, scene layout control, and language control. Driving action control allows developers to simulate various “what if” scenarios, while scene layout control enables customization of road configurations, including traffic signals and the behavior of other road users. Language control is described as the “most flexible tool,” allowing adjustments for time of day and weather conditions, which is crucial for simulating low-light or high-glare situations that could challenge the vehicle’s sensors.

The Waymo World Model can also convert real-world dashcam footage into a simulated environment, ensuring the “highest degree of realism and factuality” in virtual testing. It can even create extended simulated scenes that run at 4X speed without compromising image quality or processing power.

“By simulating the ‘impossible,’ we proactively prepare the Waymo Driver for some of the most rare and complex scenarios,” the company states.

This isn’t the first instance of Waymo tapping into Google’s extensive AI capabilities to enhance its autonomous driving methods. The Waymo EMMA (End-to-End Multimodal Model for Autonomous Driving) training model was developed using Google’s Gemini, and Waymo is reportedly working on a Gemini-based in-car voice assistant. Additionally, DeepMind has provided solutions to help Waymo reduce the occurrence of “false positives” in its sensor data.