AI System Accurately Replicates Video Games Just by Watching Them

A design program developed at Georgia Tech can reverse engineer popular games after watching two minutes of gameplay.

Published on 9/12/2017 at 7:00 AM

Researchers from the Georgia Institute of Technology have developed a technique for training artificially intelligent systems to make their own video games. A dedicated AI system — or agent — watches video of an existing game being played, then reverse-engineers the code on its own.

If all goes well, the AI actually generates a clone of the game, without ever having accessed the software coding at all. From there, the agent can build out its own version of the game, creating new levels and environments on the fly. The research could help developers create new games.

For now, the AI system is limited to relatively simple games. But it's a fast learner. In a series of experiments earlier this year, the AI was able to replicate individual levels of 2D-platform style games, such as Super Mario Brothers, by watching less than two minutes of gameplay video.

Matthew Guzdial, lead researcher and a Ph.D. student in computer science at Georgia Tech said the dedicated AI agent builds its own model of the game by carefully observing the video on a frame-to-frame basis. Using predictive modeling algorithms, the agent builds a profile of the game by trying to predict what the game will do next. Amassing trial-and-error data at superhuman speed, the agent can figure out the game's virtual “physics” of how objects move, where they move, and why.

Guzdial said the project was inspired by online “let's play” videos, in which gamers record or live-stream a gaming session and narrate their progress as they go. Because of his background in game design, Guzdial is often able to recognize the specific coding that's powering the game, just by watching the video

“I noted that even in videos where I had never played the game myself, I was still learning a lot about the game's design,” Guzdial said. “It struck me you might be able to automate this process.”

The researchers initially trained the AI system on basic “speedrunner” videos where the player's avatar — Mario, say — runs directly toward the goal. This gave the agent an accurate general model of a game using only the video footage. The AI was then able to work backward from the gameplay video to create unique game level designs.

The AI system currently works with Super Mario Brothers, but the researchers have also started replicating the experiments with other popular “platformer” games like Mega Man and Sonic the Hedgehog.

Guzdial and his fellow researchers hope that the new approach will enable human game designers to work with AI agents to create entirely new games. The technique could also have education value for game design students.

“One obvious [educational application] is in teaching people to make games,” he said. “You'd need to have a system that had a lot of knowledge to do something like that, and this is a way to get that knowledge.

Guzdial said it's important to note that the research, presented last month in Melbourne, Australia at the International Joint Conference on Artificial Intelligence, is only a first step.

“The really cool stuff comes when we have multiple game engines learned in this same representation,” he said. “At that point you can start imagining what combining elements of different game engines can get you. In the future you could imagine the average reader able to generate whole games just by showing our system a few videos.”

WATCH: How Will Artificial Intelligence Change Our Lives?