Mars Rover Opportunity Suffers Worrying Bouts of 'Amnesia'
Inevitable age-related issues have surfaced for Mars rover Opportunity and mission engineers are being challenged by an increasingly troubling bout of flash memory glitches.
Problems with NASA's Mars Exploration Rover Opportunity's flash memory have intensified over recent weeks, so Discovery News Space Producer Ian O'Neill spoke with NASA Project Manager John Callas about the severity of the glitches, how they're affecting Opportunity's mission and how his team hope to find a fix.
Mars Exploration Rover Opportunity has been exploring the Martian surface for over a decade - that's an amazing ten years longer than the 3-month primary mission it began in January 2004. But with its great successes, inevitable age-related issues have surfaced and mission engineers are being challenged by an increasingly troubling bout of rover "amnesia."
Opportunity utilizes two types of memory to record mission telemetry as it explores the Meridiani Planum region. Sister rover Spirit, which sadly succumbed to the Martian elements in 2010 after 6 years of exploring Mars, used the same system. The two types of memory are known as "volatile" and "non-volatile."
"The difference is non-volatile memory remembers everything even if you power off, in volatile memory everything goes away," said Mars Exploration Rover Project Manager John Callas, of NASA's Jet Propulsion Laboratory in Pasadena, Calif. "So volatile memory is like the traditional RAM you have in your computer; non-volatile memory uses flash memory technology."
Usually, all telemetry data is stored in the flash memory, so that when the rover powers down during the Martian night or reboots, the data remains stored - like when you turn off your digital camera, the photos remain saved to the camera's flash card. Any data stored in the rover's RAM, however, is lost as it shuts down.
Flash memory may be great for storing data when the rover's electronics are powered down, "but flash memory has a limitation on how many times you can read and write to it," Callas told Discovery News. "It ‘wears out' with use."
And, after a decade of continuous use, it's the rover's flash memory that mission engineers have identified as the source of lost data and unexpected reset events that are plaguing the rover's surface mission.
Old Rover Memory Glitches "The problems started off fairly benign, but now they've become more serious - much like an illness, the symptoms were mild, but now with the progression of time things have become more serious," added Callas.
"So now we're having these events we call ‘amnesia,' which is the rover trying to use the flash memory, but it wasn't able to, so instead it uses the RAM ... it stores telemetry data in that volatile memory, but when the rover goes to sleep and wakes up again, all (the data) is gone. So that's why we call it amnesia - it forgets what it has done."
Opportunity uses NASA's veteran Mars Odyssey satellite as a communications relay between the Red Planet and Earth, so whenever Odyssey makes an orbital pass, commands are sent down to the rover and telemetry is beamed back to Earth.
But should an orbital pass be unavailable until the rover has powered down and then rebooted the following day, the rover team noticed that data was being lost - the rover had been encountering the flash memory error and then saving it to RAM, avoiding the flash memory all together. As the rover powered down, the RAM was wiped and the data was gone the following day.
Christmas Blues The flash memory issue has grown into a bigger problem than losing valuable data, however.
As the rover attempts to save data to the flash memory, and is repeatedly unsuccessful, its software forces the rover to reboot. If a sequence of commands is sent to the rover, it will keep rebooting over and over again, forgetting what the previous command instructed the rover to do.
"Basically the rover stops what it was doing because it wasn't sure what caused the reset," said Callas. "So that interrupts our science mission on the surface of Mars.
"It's like you're trying to drive on a family trip - the car stalls out every 5 minutes. You don't make much progress that way!"
And now the rover team's worse nightmare has reared its ugly head - Opportunity stopped communicating with Earth over the Christmas break.
As the NASA team went into the Christmas holidays, a series of 3 sol (Mars day) plans gave the rover a sequence of commands to work on. On the first sol, the rover would operate as expected, but come the second and third sols, not only would the rover not execute the rest of the commands, it stopped talking to mission control.
Like any space mission, when Opportunity stops talking, "we get very, very worried," said Callas.
Fortunately, after sending commands to the rover, it sent a reassuring "beep" in reply and continued with its instructions.
It seems the source for all these problems lead back to one particular bank of flash memory. 7 banks are used by Opportunity and it's the 7th bank that is triggering the data loss, rover resets and communications glitches.
Now the culprit has been identified, JPL software engineers have developed a technique that will force the rover's software to ignore the 7th bank and utilize the other 6 apparently healthy banks. According to Callas, his team is probably a couple of weeks away from completing the software change so it can be uploaded to Opportunity.
Apart from this troubling turn of events, Callas is amazed at the health of the rest of the rover's hardware, but he remains realistic about Opportunity's future.
"The rover has been amazingly healthy considering how much we've used it ... we thought the mobility system would have worn out a long ago but it's in great health.
"But anything could fail at any moment," he said. "It's like you have an aging parent, that is otherwise in good health - maybe they go for a little jog every day, play tennis each day - but you never know, they could have a massive stroke right in the middle of the night. So we're always cautious that something could happen."
And the kicker, says Callas, is that Opportunity's most exciting science could be less than half a mile from the rover's current location at the rim of Endeavour Crater.
A Sprint to Marathon Valley "Perhaps the most exciting part of the mission is ahead of us... we have this valley, we call it Marathon Valley, only about 650 meters away from the rover."
Marathon Valley is so-called as the location marks the distance the rover will have exceeded a marathon on Mars should it get there. Opportunity has traversed over 26 miles and currently holds the off-world record for any rover - robotic or driven by an Apollo astronaut.
According to orbital mapping of Marathon Valley, the location contains a variety of clay minerals that could have only been formed when Mars had an abundance of pH-neutral water on its surface. It has ancient geology spanning back to the Noachian era, much older than Gale Crater - where NASA's Curiosity rover is currently exploring. Like Opportunity's previous exploration of clay-rich deposits, studies of Marathon Valley could provide invaluable data as to the ancient, potentially habitable Mars environment.
So as we keep our fingers crossed for a successful software fix for Opportunity and the huge science potential the rover still promises, it's mind-blowing to think that this rover, that had a primary mission of just 3 short months in 2004, is still doing incredible science in this alien environment, adding more pieces to the Mars habitability puzzle.
This March 22, 2014, self-portrait of NASA’s Mars Exploration Rover Opportunity shows the effects of wind events that had cleaned much of the accumulated dust off the rover’s solar panels.
On Jan. 24, 2004, Mars Exploration Rover Opportunity (MER-B) entered the Martian atmosphere and bounced onto the red planet's surface. The complex stages of entry, descent and landing operation saw the rover, surrounded with airbags, roll into a small hollow called Eagle Crater in Meridiani Planum -- a feat rover mission scientists called a "hole in one." After righting itself, the airbags were deflated and pulled into Opportunity's lander so the golf cart sized robot could see Mars for the first time -- it's new home. The rover's primary mission was planned to last just 3 months; little did NASA realize that they would still have an operational rover exploring Mars
12 years later
. Here are just a few of the breathtaking views from Opportunity during the very early days of the mission.
Shown here is the first self portrait Opportunity beamed back to Earth shortly after landing on "Sol 0" of the mission (a sol is a Martian day and the length of any Mars surface mission is measured in sols). Its shiny solar array glints in the sunlight on top of its landing platform. Note the deflated airbags surrounding the rover and the rippled impressions the airbag made in the Mars dust.
Opportunity was the second of the twin Mars Exploration Rovers to touch down on Mars regolith in January 2004. A little over 2 weeks earlier, NASA had celebrated the successful landing of Spirit on the other side of the planet in Gusev Crater. Spirit was also highly successful, carrying out groundbreaking science in a very different landscape compared with Opportunity. Sadly, Spirit succumbed to the Martian elements in 2009 after becoming stuck in a sand trap.
In this section of a panorama captured by Opportunity's Pancam, the rover checks out its surroundings on Jan. 26, 2004. Eagle Crater's rim is approximately 10 meters from the rover, obsecuring Opportunity's view of the surrounding landscape.
This Jan. 30 view from Opportunity's rear Hazcam shows the deck of its lander and the surrounding crater before the rover "egressed" onto the Mars soil.
Evidence of the rover's dramatic entry surround the lander. These bounce marks look like they were created by a huge basketball, but they are actually imprints left by the huge airbags that were used to cushion Opportunity's landing, bounce and roll into Eagle Crater.
On Sol 7 of the mission (Jan. 31, 2004) rover drivers commanded Opportunity to roll onto the Martian surface, leaving behind the lander that it called home for the 283 million mile journey from Earth to Meridiani Planum.
The lander -- shown here in color with rover tracks in the background from Opportunity's egress -- was named the Challenger Memorial Station in memory of the final crew of the Space Shuttle Challenger who died when the shuttle suffered an in-flight breakup during launch on Jan. 28, 1986.
After exploring its little 22 meter-wide divot, Opportunity left Eagle Crater and took this panorama of its landing spot on the 33rd, 35th, and 36th sols of its mission. Surrounding the crater is the flat plains of Meridiani Planum, a region it would go on to spend 10 years and nearly 24 miles of hard roving (to date).
As the rover trundled over the loose regolith, rover drivers were able to use the wheels to carve out trenches in the dirt to see what materials lied beneath. Shown here is a trench that was opened for closer inspection by Opportunity's microscopic imager.
With the microscopic imager, Opportunity discovered the region it was exploring is littered in
. This surprise discovery has led scientists over the last decade to theorize that these spherical hematite inclusions originated from Mars' wet past or were formed through volcanic activity.
To analyze the material just below Mars rocks' surfaces, it employs the use of its rock abrasion tool that leaves a circular "lovebite" in rocks after use. This image by Opportunity's front hazcam shows one of those marks on a rock called "McKittrick," located in the "El Capitan" area of the larger outcrop near Opportunity's landing site on Sol 30 of the mission.
During exploration of Eagle Crater, Opportunity was able to spot its discarded backshell and parachute from its descent through the Martian atmosphere. This became a mission target for Opportunity to roll to and investigate.
In a rare opportunity to see a piece of the equipment that allowed the surface mission to survive the fiery entry into the Martian atmosphere a year after landing, Opportunity approached its old backshell that it last saw on Jan. 24, 2004, just before it touched down in Eagle Crater. Shown here is the shiny backshell from its heat shield that protected the rover from temperatures that soured to thousands of degrees on entry.
When Opportunity made its inspections of the heat shield debris field, engineers were excited to see that the shield had inverted, exposing the silvery inner material. Notice the large springs on the ground -- very man made objects in a very alien landscape.
In the months after landing, it became clear that Opportunity wasn't going to stop operating and its mission was extended far beyond its original 3 months. Throughout its adventures, Opportunity repeatedly took "selfies" so that engineers back on Earth could monitor dust buildup on its solar panels. Around a year and a half into the mission, the panels are seen to be virtually dust free. This, however, is not how Opportunity's solar array remained...
A decade of Mars roving caused a huge amount of dust buildup on Opportunity's solar panels. In this photograph taken in January 2014, Opportunity is almost camouflaged. Although this inevitably reduced the amount of sunlight that can be turned into energy, the rover soldiered on -- until it was given a helping hand by Mars winds and dust devil "cleaning event," blowing some of that dust clear...
This self-portrait of NASA's Mars Exploration Rover Opportunity was taken in late March 2014 (right) showing that much of the dust on the rover's solar arrays was removed since a similar portrait from January 2014 (left). A decade of Mars exploration has taken its toll -- one of the rover's six wheels has frozen up and some of its instruments are kaput -- but Opportunity rolls on, turning up valuable new science and finding new mysteries that provide answers to some of the most vexing questions in planetary science. But above all, Opportunity is a testament to the scientists and engineers at NASA and collaborating institutions who sure know how to build a rover.