The new Hollywood: Runway’s AI models edge closer to simulating reality

Early AI-generated video models were full of obvious flaws. Models like the original version of OpenAI’s Sora were shockingly good, but could barely go 15 seconds before displaying some glitch that ruined the magic trick.

Newer models like Runway’s Gen-4 and Google’s Veo 3 can produce videos nearly indistinguishable from reality. Part of the realism comes from the models’ ability to predict the movement of the physical world, including the way light reflects off objects and how water behaves.

Because AI models are “black boxes,” and their inner workings are something of a mystery, there’s no way to explain exactly how they seem to replicate real life so well.

For tasks like complex engineering and traditional weather forecasting, scientists use systems known as physics models to make decisions. Those models require huge amounts of computation to make very narrow predictions.

Neural networks can be used as a kind of shortcut to approximate the work of physics models. Recently, companies have made huge progress in using neural networks to predict parts of the physical world, like the weather and the operations of machines.

But AI models used in video creation aren’t trained specifically to model physics. They do it as a byproduct of ingesting huge amounts of video.

There’s some evidence that video models are doing more than just mimicking the countless videos they’re trained on, Germanidis says.

Runway tests its models by asking them to generate videos of simple things, like a ball hanging from the ceiling, and then comparing the trajectory of that generated object to a real-world version that acts as ground truth.

The models still make mistakes and are far from being able to replace traditional physics modeling. But Germanidis said they haven’t come close to hitting a plateau in capabilities. “We still haven’t trained models on the vast majority of the observations of the world, like the outcomes of scientific experiments and the sensor data we have available to us,” he said. “The moment we do that, we’re going to build more powerful simulators of the world.”

Germanidis also described how Runway’s products are creating a new kind of human actor.

The company’s newest product, Aleph, isn’t a video generator. It’s built on top of its Gen-4 model, allowing customers to edit real-world footage with AI-generated special effects. One of the tool’s use cases is transforming the way actors look, like a digital makeup and costume department.

Today, a movie star’s appearance is an essential part of the job. Some actors might be incredibly talented but lack a certain look. But already, Germanidis said, Aleph is opening new creative roles for actors who may be particularly well-suited for an AI-generated skin. “One way of putting it is that you have a bit of unbundling of those skills. So if you’re amazing at the performance itself, you can just focus on that.”

Rather than replace actors outright, what AI might do is challenge the definition of what it means to be a good actor, like the advent of sound in 1920s films.

Hollywood might need to change the way it holds auditions so that it can isolate the underlying acting ability necessary for a specific role, rather than the right look.

It’s possible AI models will get so good that they’ll replace humans altogether. But for high-quality art, the subtle emotion conveyed in an actor’s voice, facial expressions, and movement will be difficult to replicate with AI alone.

One view is that it doesn’t matter if AI is “just a tool” for creatives. The tools themselves matter.

From a deep dive by The Hollywood Reporter’s Steven Zeitchik: “The compact of cinema for its roughly 125 years of existence is that we accept all the trickery onscreen because we know it was created by humans standing behind it — that whether the Death Star is being blown up or Chow Yun-fat and Michelle Yeoh are flying through the air in a sword fight, those born of flesh-bound mother and in possession of human brain came together, puzzled over a problem and figured out its solution to give us the art that we now see. Whatever didn’t happen involved a lot of people to make happen.”