Particle Physics, Statistics, Overanalysing Ponies, and Dead Fish · 7:17pm Mar 25th, 2016
Anyone connected with the world of High Energy Particle Physics, or anyone who reads lots of popular science articles, may have picked up the news that something interesting has been seen at CERN. Maybe.
To be more specific, after sifting through the huge amount of data recorded by the two big experiments at the Large Hadron Collider, something odd showed up: smashing protons together was producing more light than expected. To be even more specific: they observed an excess of events producing two high energy photons. This shouldn’t happen. At least according to the Standard Model of Particle Physics.
This could be big. It could be something really new and exciting. A new particle! The first evidence for a whole new world of Physics Beyond the Standard Model. This is exactly the sort of unexpected discovery we’ve been dreaming of, which could let us decipher the puzzles in current theories. Just what we need to renew our enthusiasm, which was getting a bit low due to the lack of any exciting discoveries after the Higgs Boson in 2012. It might even provide enough public interest to create the political will to get funding approved for the International Linear Collider.
Then again, it might just be a statistical fluctuation.
In the language of particle physics, the CMS experiment has seen a 3.2σ excess. For the ATLAS experiment it was 3.6σ (see: Diphoton Update)
Sigma (σ) is the symbol for standard deviation, a measure of how dispersed a data set is. For a normal distribution of random events, 68% of events lie within 1σ of the mean, 95% within 2σ, 99.7% lie within 3σ. So, in a simplified interpretation, a 3σ excess means there is only a 0.3% chance it is a random fluctuation. This makes it sound rather convincing, especially considering that it was seen by both experiments.
The trouble is that 0.3% probability events actually occur rather often. If you look a thousand such searches, you would expect to see one. The ATLAS and CMS teams have spent the last few years analysing the incoming data looking for every possible signal of new physics. Among all the hundreds of null results for searches for new particles and interactions you would expect to find a few false positives. There have been 3σ excesses seen before, which just disappeared once more data was available. This is sometimes called the ‘look elsewhere’ effect – the problem being that if you don’t find anything exciting in your first analysis, you just keep looking through the data until something pops up.
So we don’t yet know if this is real or not. The convention in particle physics is that a “discovery” must be a 5σ signal – where there is only a 1 in 3.5 million probability it is by chance.
This is a problem in many areas of science, especially research which involves running a lot of software analyses on very large volumes of data. Neuroscience (brain scanning) is a field which has unfortunately earned a bit of a reputation here. Possibly because as technology has developed, researchers have had access to new brain scanning techniques such as fMRI, but not all of them had the experience to know how to interpret the data. The idea sounds simple: you scan a lot of brains, while their owners perform particular tasks, or look at particular images, then run a program to look for correlations in the brain activity. Thus you can identify the regions of the brain associated with different functions. But be careful—when searching for statistical correlations in high resolution images, you can expect to see quite a few random coincidences. If you don’t properly take account for this, it’s very easy to fool yourself into thinking there’s a real effect there. This was most impressively demonstrated by Ig-Nobel-prize-winning research by Craig Bennet, who carried out repeated fMRI scans of a dead salmon while showing the fishy subject images of human individuals in social situations and asking it what emotion the individual was experiencing. The salmon did not give any answers, but the instrumental noise was sufficient to create a number of false positives, until an appropriate correction was included in the analysis procedure.
The same thing can happen when you watch too many episodes of My Little Pony. Consider the anomaly I mentioned on my blog a few weeks ago: a total of 7 episodes include a mention of “Tuesday”, but the average number of episodes with a pony mentioning a given day of the week is only 2. Is this a sign of a secret Tuesday-worshiping cult at Hasbro? Do the writers have a subconscious bias towards this day? Perhaps they meet to discuss scripts on Wednesdays and so tend to write the first drafts on Tuesday evenings? Or, as mylittleeconomy hypothesized, is Tuesday preferred for spoken dialogue as the only day that starts with a hard consonant?
Let’s first consider the null hypothesis that this is just down to a random fluctuation. If there is an equal probability that a script writer will choose any day of the week, then we can show there is a 1 in 86 chance of getting seven episodes mentioning the same day. But consider that I have been overanalysing pony episodes since 2012. So it is actually not unexpected that I would stumble on something like this. If it hadn’t been an excess of Tuesdays, it would have been something else. Because I don’t examine pony scripts in a systematic way, it is difficult to make a quantitative estimate of how likely this is. (Incidentally I spotted this excess of Tuesday only because when writing Time on Their Hooves, I was interested in how the show references calendrical terms.)
Of course this doesn’t necessarily mean it is just a random fluctuation. It could be a real effect. There is only one way to find out, both for particle physics and pony analysis.
We need more data.
This is why particle physics is the realest of the real. Most other disciplines are far less rigorous (though usually not by choice). It may depend on the amount of data one can efficiently and economically collect.
The Standard Model has been so rock-solid for so long, that I'm betting it is a statistical fluctuation. But I'm really hoping it isn't.
As for the Tuesday Effect, many comedians claim that there are some words that are inherently funnier than others. Noodle, for instance.
Delving into my memory of old comics, I immediately recall, "I will gladly pay you Tuesday, for a hamburger today." And, "Tuesday after lunch is the most cosmic time in the universe."
So... hurrah for confirmation bias! Or is there really something inherently funny about the word?
Definitely looking forward to further information on both Equestria and the diphoton. 2016 is going to be a very interesting year.
3826445
As far as I can tell, the 5 sigma rule is just a convention, and I think many statisticians would think it a rather simplistic way to define a discovery. How you go about searching for something does make a big difference. If you set out to build an experiment to measure one thing and get a 3 sigma result, it's rather more convincing than if you spot such an effect as one of many experiments done with a general purpose apparatus.
3826458
The Standard Model has been amazingly successful, but no one in the field considers it the final theory, as there are still so many unanswered questions: what is dark matter? What is the origin of the asymmetry between matter and antimatter? How do we combine gravity with quantum mechanics and why is it so much weaker than other forces? But it's quite possible that there will be no new physics at the scale which can be reached by current experiments.
At any moment, I expect somebody to say "Oops, we didn't get that detector plugged in quite right. Try it again."
I seem to remember something similar happening with a potential Tacheon discovery...
The trouble with doing ten billion measurements, is that million to one chances occur 9 times out of 10
Dont forget the birthday effect as well. Pick a class of 30 and teres an even chance that two people will have the same birthday. I have a 3DS with a birthday checker for passig people with the same setup, and after 260 days ticked off, but the total number passed since activation not recorded, there are empty spaces. September only has 4 days unmarked, and yet February 29th is marked.
There are things Ive seen in the simple explanations which puzzle me, like there being a difference between Minkowski and Euclidean spaces, when looking up Feynman diagrams, and something about the dangling of a spring and the throwing of a ball being parabolic. Id have to look again at teh spring, but the dangling of a rope is a catenery, hyperbolic function, and the path of a ball is an ellipse, orbital about teh centre of mass of the planet. Rotating the framework from one to the other, converting reals to imaginarys, now makes more sence.
One I havent seen yet is the framework where time is the scalar, and space is the imgaginary vector, because I suspect, as one physisist put it, imaginary space does not exist and so he is trying to rewrite the whole of maths to avoid the value i. while writing a description that is identical to that totally arbitary symbol.
If you take c as the limit, then it makes sence that ds^2=(c^2)dt^2-dx^2-dy^2-dz^2 for a spacetime four vector, and that s is a value given by the speed of light. This means it can also undergo maths that leads to a variable speed of light, or simultaneous dual value speed of light. Which simplified models dont allow.
Also, the instant Feynman allowed even the idea of FTL in his particles, it exploded. Apply relativistic functions to teh quantum articles and you get auto renormalisation.
3826626
I remember that one. Exciting times. Of course no one ever seriously thought it was true... But it would have been so so so cool if it was... The technical difference is that that was a systematic error rather than a statistical uncertainty. They definitely saw a real effect - I think it was 6 sigma certainty. But was it caused by faster-than-light travel, or by some technical issue? We were all imagining some subtle effect involving general relativity... A little embarrassing for the OPERA collaboration that it turned out to be an optical fibre not properly plugged in.