Space & Innovation

A.I. Program Teaches Itself How to Play Video Games

The self-learning software could be used to control autonomous cars and improve stock market and weather forecasting.

A computer program designed by Google researchers has figured out how to play classic Atari video games without knowing ahead of time how the games work.

The authors of the report, which is being published today in the journal Nature, say their program could be used to control self-driving cars or in a smartphone app that plans your next vacation and then buys all the tickets. The group has already discussed using the program for weather forecasting and stock market predictions.

Video Game Fails: Slide Show

The so-called Deep-G Network (DQN) software agent learns by trial-and-error, getting rewarded for making the right moves and punished for failures.

The group allowed the software to "learn" on its own using two methods, reinforcement learning and so-called "deep learning," in which the program figures out the game based on the controls used and the pixels on the screen.

"It's a very impressive piece of work and results are a bit scary," said Stuart Russell, professor of computer science at the University of California, Berkeley, and author of more than 100 papers on artificial intelligence.

Video Game Evolution

"It's a situation similar to that of a newborn baby. This system has no understanding of the world. It's just opening its eyes and looking at a screen. Within a few hours it's playing 49 Atari games. If it was your child, you would think your baby was possessed by demons or something."

The researchers (formerly with London-based artificial intelligence (A.I.) firm, DeepMind Technologies, which Google purchased for $400 million in 2014) used circa-1980s Atari 2600 games because they represented just the right challenge of not too hard and not too easy for the program, according to Demis Hassabis, founder of DeepMind and an author of the paper.

"The ultimate goal is to build smart machines," Hassabis told reporters during a teleconference from London.

"We are many decades off from doing that. This is the first significant rung of the ladder that we are on. We have shown that a general learning system can work, and we can go from pixels to actions and it can work on a challenging task that even humans find difficult."

Video Game Fails: Slide Show

The DQN (deep-q network) software still doesn't have more advanced conceptual knowledge that humans possess, Hassabis said.

In fact, the software program performed significantly better than human experts in games like Breakout, where the goal is to build a wall of bricks, whereas it was a near-failure at Seaquest, a game in which players have to plan ahead to fill their submarine with oxygen. That requires conceptual thinking, something that DQN doesn't have yet.

"I don't think it's formulating the idea of what a submarine is, or do I need to get oxygen," he said. "These are abstract notions that we get from the real world. They are not building those representations."

Instead, the software takes data from the pixels on the video games, some of which are simple two- or three-dimensional images. Some experts see immediate applications for this new form of artificial intelligence.

"Perhaps the most promising application is to robotics in which a robot must sense the environment, plan a course of action to achieve a goal or reward, and act in the world," said Mark Riedl, professor of computer science at Georgia Tech. "Of course acting in the real world is many times more complex than computer games, and many times more complex than recognizing objects in videos."

Video Game Evolution

One computer scientist said the Google paper demonstrated the "chocolate-and-peanut butter effect" of two machine learning techniques baked the correct way together lead to a greater effect.

"This paper has electrified the community about what's possible with the right combination of techniques," Carnegie Mellon's Drew Bagnell said in an e-mail to Discovery News.

The Deep Q-Network agent, shown here as a brain, tracks the status of inbound retro-styled game sprites.

As Microsoft’s motion-controlled Kinect gaming device hits stores, everyone from players to industry analysts are wondering whether the device is a “game changer” (obvious pun, sorry.) The image-sensing, controller-free device combines a camera, depth sensor and microphone to track a player’s body and then accurately reflects that motion on the screen. It can also distinguish between different individuals and respond to voice commands. Of course, it’s still way too early to know Kinect’s impact on the video game industry. But history has shown us that not every new-fangled interface will succeed. So while we wait to see if Kinect will reign, let’s look at some of gaming’s biggest fails.

Mattel Intellivision’s Mystery Keyboard

Unveiled by toy manufacturer Mattel in 1979, the Intellivision got cute by combining the words "intelligent" and "television," and then claimed to have better graphics than its main competitor, the Atari 2600. But it’s what Intellivision didn’t come with that made it infamous: a keyboard. That missing component, featured in commercials but not packaged with the system, promised to increase the Intellivision’s computing capabilities, turning it from a game console into a home computer. But manufacture of the keyboard was delayed so often that even the Federal Trade Commission started fining Mattel. Eventually, 4,000 keyboards were produced, but by then the whole project was canceled and both the world and Mattel had moved on.

Atari Mindlink, 1984

Computers seemed limitless in the mid-80s: Matthew Broderick was using one to start World War 3; Anthony Michael Hall was building a girlfriend with his; and Jeff Bridges went inside one to race neon motorcycles. So of course we'd at least be able to link our brains with one to control video games, right? Enter the Atari Mindlink. The game controller supposedly worked by reading your brain’s muscle movements with infrared sensors, which then transferred those movements to games. The games never worked, people complained of headaches, and the Mindlink was quickly scrapped. Amazingly, however, technology is being used today to help people with disabilities use thoughts to control computers. Electronics firm Emotiv, for example, is developing games and brain-computer interfaces that record electrical activity produced by neurons firing in the brain.

Milton Bradley Vectrex, 1982

Costly ($430 in today's dollars) and unwieldy (unlike other consoles, it came with its own monitor), the Vectrex set itself apart by featuring vector graphics like those found in the popular arcade game “Asteroid!” It actually might not be totally fair to call the Vectrex a flop -- in many ways it was ahead of its time, with a 3-D headset, an advanced Motorola processor, and even a light pen that let players "draw" on the Vectrex monitor. But its release in late 1982 coincided with the Video Game Crash of 1983 and the console quickly faded away. Even today, though, Vectrex homebrewers are out there and the system has a cult following.

Nintendo Power Glove, 1989

Mattel built it. Nintendo licensed it. But it was a young Fred Savage in the movie The Wizard that catapulted the Power Glove into infamy and Internet meme status. The Nintendo-produced movie tried to generate awe about the glove, which was supposed to recreate hand movements in real time. Sure, it looked kind of cool in a cyborg-ish way, but mostly it was a plastic mitten with a controller attached to the forearm that is widely regarded as one of the worst video game blunders of all time.

Apple-Bandai Pippin, 1995

For every iMac and iPhone, there's a Pippin. The multimedia platform was designed by Apple and produced by Japanese firm Bandai in 1995. It was supposed to compete against the likes of the Sega Saturn, the Nintendo 64 and the Sony Playstation. But with a launch cost of $599, and very little software to support it, all the Pippin did was make a bunch of "worst ever" and "biggest tech busts" lists. Including this one. Plus, Pippin (a type of apple, naturally) is just a terrible name for a video game console.

Nintendo Virtual Boy, 1995

The 3-D Virtual Boy console was basically a set of viewfinder-like goggles that sat precariously on a plastic stand. A player would crane his or her head inside and use the wired controller to play monochromatic 3-D games like Mario’s Tennis. So basically it's like taking an eye exam. Only 22 games were released for the console, which is now considered a collector's item. Time heals all video game blunders.

Sega Activator, 1993

You could call it the great-great-grandfather of Kinect. Or you could call it an ugly, plastic octagon of awkward dancing. Instead of reading your real-time body movements and converting them on screen (like Kinect does), the Sega Activator expected players to stand inside the octagon and perform a series of coordinated steps within the device's eight quadrants, which corresponded to a button on a hand-held controller. Also, you had to avoid placing it under overhead lights, ceiling fans, mirrors or metallic ceilings. And couldn't look at it funny or call it names. If you have any doubt why it failed, check out this instructional video.

Philips CD-i, 1991

The tech graveyard is littered with the corpses of dead or failed file formats, from the Betamax to the mini-disc to the CD-i. The Compact Disk Interactive was the format of choice for the multimedia player released by Philips in 1991. Priced initially at $700 (you mean that’'s it?!), the console was first intended for educational and self-help titles, then gradually housed several video games that are known as some of the worst ever made. In the so-bad-it’s-awesome category are two CD-i Legend of Zelda titles that feature full animation cut scenes so jarringly bad that they’re frequently posted online.

Atari's E.T. the Extra-Terrestrial video game, 1982

Manufactured by Atari to capitalize on the enormous success of Steven Spielberg’s sci-fi movie about a loveable lost alien and the boy who found him, this game should have been an easy win. I mean, who wouldn’t want to play the part of Elliot, saving ET from the bad guys, riding your bike up into the sky? Bring it on! But what happened is the cautionary tale for game companies looking to cash in on movie tie-ins: Stay faithful to the movie source and don’t rush the product. Instead of a cool adventure, players got impossible-to-navigate game-play, terrible graphics, and an odd storyline. The company allegedly buried massive unsold and returned copies of the game in a New Mexico landfill.