What Microsoft’s new AI demo for Minecraft means for gaming

Microsoft has shown off an internal demo that gives users the ability to control its popular Minecraft video game by telling the game what to do, instead of manually moving characters to build and dig, according to people familiar with the matter.

The demo was made possible by cutting-edge AI, though it didn’t use Prometheus, the underlying technology that powers its recently-announced Bing chat service developed with OpenAI, the company behind ChatGPT. The advances could upend the more than $300 billion online gaming industry as companies compete for obsessive players willing to spend real money on virtual objects.

Applying natural language commands to Minecraft is particularly challenging because the game contains many variables and there are lots of steps required to build objects in it. For instance,  there are several different ways to build a car with Minecraft components, and an AI could probably come up with methods that people have not thought of.

Then it’s a small step to take the written commands and eventually allow for voice prompts, so users could verbally explain to the AI what they want to build.

People familiar with the matter said Microsoft has no immediate plans to include the feature in the public version of Minecraft.

Last week, Microsoft unveiled its new Bing search engine, which includes an artificial intelligence chatbot capable of having conversations with people and drawing vast amounts of data to answer seemingly any question.

The company has not announced concrete plans to expand the use of the technology to other areas of its business, such as gaming and services it sells to other businesses. Microsoft declined to comment on the demo.

Other companies, such as Minecraft competitor Roblox, have in the past openly discussed using so-called “generative AI” to develop video games. Roblox executive Stefano Corazza is scheduled to give a talk on that topic at the Game Developers Conference in March.

The open-ended nature of the Minecraft world makes it a fertile testing ground for computer automation, a fact not lost on artificial intelligence researchers who have used it for years to test the capabilities and limits of new models.

The game involves foraging for raw materials and then converting those resources into tools, weapons, building materials, and more. The possibilities are so complex and vast that the game has spawned a whole genre of instructional books, YouTube channels, and even fantasy novels set in the Minecraft world. (I know because my son brings them home from the library).

Until recently, attempts to train computer models to automatically navigate and build in that world have produced impressive but somewhat limited results. (Microsoft hinted that it was looking at this kind of feature at its Build conference last May.)

In a research paper in June of last year, OpenAI said it trained a computer model to play Minecraft by having it watch 70,000 hours of videos of real people playing the game. It was able to get the model to take on increasingly advanced tasks in the game, such as constructing wooden tools and crafting tables.

Recent advances in AI, though, could take those capabilities from the research lab to the real world, making the game exponentially easier and possibly more appealing to a broader audience. The software may even demonstrate new ideas that humans have yet to dream up. This would fundamentally change the way Minecraft is played.

Microsoft has put itself in an enviable position compared to other video game companies. Not only has it helped develop the latest AI models, it has optimized its Azure cloud to run them on a massive scale.

That means Microsoft could — if it wanted — roll out the technology to a staggering number of games. While the software giant is known more for its Windows operating system and Office products, it has become a video game juggernaut with blue chip properties, including Xbox, Minecraft, and Flight Simulator.

It begs the question: Did Microsoft CEO Satya Nadella have this technology in mind when he shrewdly agreed to acquire Activision in 2022 for $69 billion? (That deal faces regulatory hurdles in the U.S. and Europe).

But Microsoft still faces some difficult choices: How does it incorporate this new technology in a way that fundamentally changes its games but preserves what was fun about them in the first place?

The AI revolution in video games is still a long way off, writes Will Bedingfield in WIRED.

Not only are there major technical limitations to creating video game content with AI, there are intellectual property concerns, too, as much of what generative AI draws on is content created by people that is shared on the web.

Bedingfield quotes Julian Togelius, co-director of the NYU Game Innovation Lab: “It feels like the worst kind of crypto bros left the crypto ship as it was sinking, and then they came over here and were like, ‘Generative AI: Start the hype machine.’”