Planet of the Apes: Motion-Capture Gets Wild
Weta/20th Century Fox
The technology uses a network of carefully calibrated monochrome cameras that track the movements of reflective markers attached to key spots on the bodies of actors.
Warner Bros. Pictures
This summer is full of sci-fi thrillers, such as "Edge of the Tomorrow," "Lucy," "Interstellar" and "The Giver," to name a few. In all of these movies, technology is used for good and evil, just like in real life. Science fiction constantly holds a mirror up to society and culture and it's one of the reasons we love that genre. Our fears about the future are often reflected in the sci-fi stories we tell ourselves. In fact, the history of science fiction movies is lousy with mechanical villains inspired by our various technophobias. Who's the scariest of them all? We can start with cybernetic assassin robots from the future in the Terminator movie series. Here, the T-800 stares us down with eyes capable of face recognition, long-range zoom, night vision and tracking two different moving targets at once. You won't go far.
Above is the robot Maria from Fritz Lang's 1927 German expressionist epic "Metropolis." The hugely influential film dealt with themes like class revolt and the dehumanizing aspects of industrialization.
No list of technological terrors would be compete without HAL 9000, the murderous computer from Stanley Kubrick's classic "2001: A Space Odyssey." HAL reflects a fear that's been bouncing around sci-fi for decades -- the amoral computer that carries out its programming regardless of human life.
The original Godzilla was a kind of technological terror himself -- a giant sea monster spawned by nuclear radiation. Godzilla was conceived as a metaphor for the devastation of the atomic bomb attacks on Japan, and in 1974, director Jun Fukuda upped the ante with the robotic Mechagodzilla.
20th Century Fox
Ridley Scott's 1979 sci-fi/horror hybrid "Alien" explored another recurring trope in science fiction -- the man who's really a machine. In the film, the calculating android Ash determines that his human colleagues are expendable in pursuit of scientific knowledge.
In his 1987 film "RoboCop," director Paul Verhoeven mixed science fiction with savage social satire. In a near-future Detroit, the robotic enforcer known as ED-209 is developed by the ruthless OmniCorp to make policing more cost effective -- even if it means the occasional bystander gets shot.
Considered by many scholars to be the first true work of science fiction, Mary Shelley's 1818 novel "Frankenstein" has inspired dozens of films, including director Kenneth Branagh's 1994 version with Robert De Niro as the monster. The Frankenstein story continues to recycle itself in pop culture as a cautionary tale about science and technology run amok.
Several recent sci-fi films appear to reflect our increasing unease with drone technology. In the 2013 post-apocalypse thriller "Oblivion," Tom Cruise must contend with an entire squadron of unruly, unreasoning drones -- or "unmanned aerial vehicles," to use the fashionable military euphemism. Who is your favorite techno-terror from the movies?
The men and women who survived a deadly virus that wiped out much of earth's human population hunker down amidst the ruins of San Francisco; meanwhile, a growing ape population has built a lovely and thriving community outside of San Francisco in Muir Woods. The tension between the two societies drives the action-packed Dawn of the Planet of the Apes,the sequel to the 2011 Rise of the Planet the Apes that starred James Franco.
As this sequel begins, Franco's character has been dead for a decade, and the apes have had plenty of time to create their version of civilization. But in real time, it's been just three years since the Rise movie. In the world ofmotion picture technology, though, that's an eternity. Long enough to create computer graphics gear robust enough to take out of the studio and deep into a real forest. And long enough that moviemakers no longer need to give a recognizable Hollywood star top billing to bring in audiences.
In fact, if you passed the leading man of Dawn — Andy Serkis — on the street, you wouldn't recognize his face at all, for you never see it on the screen. That's because his performance in the woods (actually, forests near Vancouver, not San Francisco) wasn't filmed traditionally, it was motion-captured and used as a framework for a computer-created realistic digital ape, Caesar. And, for the first time in my knowledge, it's the performances of the motion capture actors, not the regular actors portraying humans, that are getting all the good reviews from critics; there is even talk of the first best-actor Oscar nomination for a motion-capture performance.
Motion capture enables moviemakers to create realistic non-human characters, including Gollum in Lord the Rings, the Na’vis of Avatar, and the intelligent chimpanzees and orangutans of Rise of the Planet the Apes. It has also let moviemakers digitally tweak human characters, aging Brad Pitt in The Curious Case of Benjamin Button.
The technology uses a network of carefully calibrated monochrome cameras that track the movements of reflective markers attached to key spots on the bodies of actors and then use built-in processors to extract the precise coordinates of the markers. A motion capture movie set also uses a handful of regular high resolution video cameras to record the overall scene for the director and others involved in the production to use as reference. Later, animators correlate the data about the marker locations with the same points on virtual characters, like shoulders, knees, and feet.
After it is imported into the computer system, the data about the movement of the markers becomes a connect-the-dots representation of how the human actor moved that drives the digital characters. Animators later adjust the movements to better match them to ape physiology. It’s a complicated business and has historically taken place in a studio or, at the most extreme, a small and contained outdoor area, where lighting, shadows, and reflections that could impede the tracking of the markers can be carefully controlled.
But the movie Dawn of the Planet of the Apes, being released into theaters on 11 July took motion capture into the wild, with more than 85 percent of the movie shot outside the studio, in forests near Vancouver and in various outdoor locations near New Orleans.
Much of the movie action surrounds a community of 2000 apes, living in a rainforest-like environment. The technical challenges were huge, reported Joe Letteri, senior visual effects supervisor, and Dejan Momcilovic, motion capture supervisor, both from New Zealand’s Weta Digital.
“We went deep into the forest, where it was raining and wet,” says Letteri. “It was the absolutely worst conditions we could have had for getting the [motion capture] to work reliably in.”
Top: Actor Andy Serkis wears an LED-studded suit to create the performance of Caesar, the leader of the ape nation, in Dawn of Planet of the Apes. Bottom: Karin Konoval plays the orangutan Maurice. Weta/20th Century Fox
The effort used about 50 motion capture cameras and eight standard high definition cameras. So many were needed, Letteri explained, because, unlike the previous movie in the series, Rise of the Planet of the Apes, in which most of the scenes show apes individually caged, the apes in this movie are running all over the place, in a visually crowded environment. With multiple actors in action running between trees, markers are frequently blocked from any one camera’s view; 50 cameras meant that no matter what happened at least one camera would be tracking most markers.
The crew set some of the cameras on the ground, camouflaged by moss, put others on 5-meter tall aluminum towers that could each hold several cameras, put single cameras on poles attached to flat bases in places where they needed to blend into their surroundings, and, in some cases, simply tied the cameras directly to tree limbs. They camouflaged them well, said Momcilovic, and in post production carefully looked to remove any evidence of the array of cameras but, he said, it’s likely that an audience member watching the film closely will be able to spot the occasional camera lens peaking out of leaves or moss.
The set up had to be assembled for each day’s shoot and calibrated by a crewmember walking the scene carrying a rod with markers to make sure multiple cameras could spot each marker at every place an actor might go. A good gust of wind, or someone bumping into a camera pole, could disturb the position of a camera enough in order to require recalibration. And the whole setup had to be packed away at the end of the day, even if the crew was planning to come back to the same spot, it couldn’t sit out in the damp forest.
The cameras had to be protected from the weather. And they had, for the first time, to communicate with the local data server wirelessly, adding to the complexity of getting this whole setup to work reliably. The team from Weta designed the outdoor housings and the Wi-Fi add-ons themselves. And it was a crunch. The camera cases, eventually manufactured in Canada, were still in prototype form when the crew arrived on location.
In spite of the special cases, a lot of the equipment had to spend each night surrounded by silica gel in order to dry out. And keeping everything going required the efforts of a software developer, Glenn Anderson, who also knew a lot about electronic.
“He was fixing stuff full-time,” Momcilovic recalls, and along with programming changes in the software, using a custom web site that pushed out updates to the array of cameras while they were in use. (Anderson carries a certain amount of tech-cred outside the computer graphics world: he developed the Eudora Mail Internet Server back in 1993.)
Each actor wore 48 markers on his or her body; faces, filmed with head-mounted cameras, were dabbed with spots of white paint. Because of the lack of control over the lighting in an outdoor environment, the Weta team couldn’t use standard reflective markers. They had previously developed infrared LED markers for a brief outdoor scene in the previous Planet of the Apes movie; these, however, weren’t reliable enough to use for extended filming. So, Momcilovic says, they molded them strands of translucent plastic, strong enough to take a direct hit from a hammer without damage to the LEDs inside. The strands, attached to the actors with Velcro, could be controlled remotely, turned on or off or made brighter to deal with changing light conditions.
As many as 13 actors wearing these active LED markers appeared in a scene at one time. The processors in the cameras, after analyzing the video on the fly to extract the marker locations, transmitted the data to a server on site. As soon as the crew brought the server out of the forest and onto the Internet, it sent all the motion capture data to New Zealand, where Weta’s team of animators could immediately start the process of using it to create digital apes.
Though the weeks shooting in the Canadian forest presented the biggest technical challenge up front, the toughest scene for the computers to process afterwards turned out to be a fully computer-graphic industrial environment, set outside of New Orleans.
“There’s a big ending sequence,” says Letteri, “with a large number of apes on a half-built skyscraper, at night, with lots of lights throughout. Everything you see in the scene is CG — creating the fur, the muscles, and the facial simulation in that environment took an intense level of rendering,” about 550 core-processor-hours per frame. The team, of course, used banks of computers with multiple cores, typically 200 to 1000 processors would be in use at a time, though sometimes as many as 50,000 ran simultaneously. Was it worth it? I intend to find out today.
Get more from IEEE Spectrum
Ender’s Game is Already a Reality for the U.S. Military
Digital Actors Go Beyond the Uncanny Valley
Hockey Fans to Test Facial Recognition Technology
This article originally appeared on IEEE Spectrum; all rights reserved.