As Sean Carroll points out over at Cosmic Variance, this isn't one of those pesky three-sigma results that pop up all the time in particle physics - most recently in the ongoing search for the Higgs boson - and then just as quickly disappear as more data is added to the mix, because it turned out to be a statistical fluctuation.
A five-sigma result (five standard deviations) is usually sufficient to claim a discovery. The OPERA collaborators are reporting an impressive six-sigma result - in other words, it's probably not due to a random statistical fluctuation.
So why aren't they enthusiastically claiming discovery from the highest mountaintop? They recognize that, as the saying goes, extraordinary results demand extraordinary evidence. "Whenever you touch something so fundamental, you have to be much more prudent," Ereditato told The Guardian. "A result is never a discovery until other people confirm it." That's why the team spent six months double, triple and quadruple checking their analysis. "If there is a problem, it must be a tough, nasty effect, because trivial things we are clever enough to rule out."
Don't Believe the Hype (Yet)
I'm sorry to report that, for all the hoopla, the general consensus that has emerged over the last couple of days is that (a) it's a really interesting, potentially exciting result, but (b) it probably won't hold up over time. Even the OPERA team isn't entirely convinced they're right; they're putting their work out there and basically asking their colleagues to poke holes in it and find anything they've missed. These are world-class physicists, mind you, but nobody is perfect, particularly when it comes to such tricky measurements.
So, what could be the problem? Per Carroll:
There is another looming source of possible error: a "systematic effect," i.e. some unknown miscalibration somewhere in the experiment or analysis pipeline. (If you are measuring something incorrectly, it doesn't matter that you measure it very carefully.) In particular, the mismatch between the expected and observed timing amounts to tens of nanoseconds; but any individual "event" takes the form of a pulse that is spread out over thousands of nanoseconds. Extracting the signal is a matter of using statistics over many such events - a tricky business.
A similar method was used by scientists with the MINOS collaboration, which also saw hints of neutrinos traveling slightly faster than light in 2007, although with much smaller statistical significance - so much so that Fermilab physicist Joseph Lykken described the result as "inconclusive."
"It's a pretty messy way to try to test a fundamental property," Lykken told Discovery News. "You have a proton beam at CERN that makes the neutrinos, but you don't know which proton made which neutrino. This makes it tough to claim nanosecond timing of the neutrinos. OPERA says they can do this on a statistical basis. Maybe so, but normally in experiments you use something well understood to measure something messy, not the other way around."
Another objection: "In a way, this experiment has been done," according to Marc Sher, a particle physicist with William & Mary College. We can look to the neutrinos detected from Supernova 1987A, which arrived roughly three hours before the light from the exploding star reached the detectors. But that's not because neutrinos traveled faster than light. Rather, they were able to pass right through all the material forming an envelope around the dying star, whereas photons would have to work their way through.
Physicists did the calculations and expected a three-hour delay, and that's exactly what they observed with the neutrinos from SN1987A. However, as Sher (and many others) have pointed out, if the OPERA result is real, those neutrinos should have traveled much faster, so much so that they would have arrived even sooner - say, in 1984. I think physicists probably would have noticed.
"Supernova neutrinos are known, experimentally, to travel at the same speed as light to better than a part in a trillion," Sher emphasized. "The OPERA claim is that they are traveling faster than light by a part in 30,000." And, well, that's problematic.
John Beacom of Ohio State University told Discovery News that the comparison to SN1987A neutrinos might not be the best one to make: "It's meaningless without knowing how the speed might vary with neutrino energy, distance, etc." If you want a possible good cross-check of the OPERA results, he suggests a closer analog would be to look to experiments like IceCube, which searches for high-energy neutrinos associated with gamma ray bursts.