Friday 4 December 2009

Fiddling the numbers the CRU way

I thought I would have some fun this week by fiddling with a set of statistics to show how graphs can be manipulated to say anything you want. I spent a few minutes considering what sort of data set to use, and I settled on the ages of our Federal MPs - you can look up their dates of birth at the Australian Parliament House website.

The first graph shows the number of MPs born in each year. They range from 1935 to 1981. I then fitted a lovely, smoothed curve to it, which resembles a bell curve.

However, no matter how I fiddled with the data, I couldn't produce a hockey stick. So I reached into my bag of CRU "tricks" and simply deleted all MPs born after 1967. Even though that removed the "tail" of low results, a 3 order polynomial curve of best fit refused to produce a hockey stick.

So I changed to a 4 order polynomial - and look! The signs of a hockey stick start to appear!

So I cranked it right up to a 6 order polynomial and - wooowor! - the hockey stick just takes off!

I then discovered that Excel 2007 has this nifty "prediction" button when producing graphs like this, so I asked it to add a prediction for 10 years into the future.

Taking this to a Tim Flannery type extreme, I asked it to predict out to 100 years. According to my scientific survey, in 2109, there will be 800,000 Federal MP's who were born in 2067. The weight of those MPs alone will cause Canberra to sink, and the entire city to be drowned under Lake Burley Griffin. Oops, I mean water levels will rise, leading to the catastrophic destruction of Canberra. We could stop that happening by opening the weir downstream and letting some water out, but in typical fashion, Labor and the Greens would insist on taxing something in order to stop this - like neck ties.

Now it was quite unfair of me to exclude all those MPs born in 1968 or later, but let's assume that we didn't have their exact DOBs, and we had to estimate their ages by proxy. Here is the list of excluded members, and their actual DOBs.

Scott John Morrison 1968
Jason Peter Wood 1968
Sen Hon Penny Wong 1968
Sophie Mirabella 1968
Hon Tanya Plibersek 1969
Damien Francis Hale 1969
Kirsten Fiona Livermore 1969
Hon Anthony Stephen Burke 1969
Sen Cory Bernardi 1969
Stuart Rowland Robert 1970
Sen Scott Ludlam 1970
Michael Andrew Johnson 1970
Sen Michaelia Clare Cash 1970
Yvette Maree D'Ath 1970
Mark Christopher Butler 1970
Hon Peter Craig Dutton 1970
Sen Mathias Hubert Paul Cormann 1970
Julie Maree Collins 1971
Sen Mark Victor Arbib 1971
Sen Louise Clare Pratt 1972
Nicholas David Champion 1972
Jodie Louise Campbell 1972
Michael Fayat Keenan 1972
Jason Dean Clare 1972
Hon Christopher Eyles Bowen 1973
Sen Scott Michael Ryan 1973
Sen Simon John Birmingham 1974
Steven Michele Ciobo 1974
David John Bradbury 1976
Darren Leicester Cheeseman 1976
Alexander George Hawke 1977
Hon Katherine (Kate) Margaret Ellis 1977
Amanda Louise Rishworth 1978
Sen Sarah Coral Hanson-Young 1981

Now let's revisit the APH website and try to guess their ages from their photos (in a scientific manner).

Here's Senator Scott Ryan. Looks like a very serious bloke. Bit chubby round the jowls. Must be 45 at least (actual age - 36)

Senator Pwong. Always looks haggard when I see her in the paper. She dresses like a state school teacher. Again, must be 49 at least (actual age 41).

A youthful looking Senator Birmingham. I'd say he's about 42. Erp. He's 35.

And we finish with Senator Arbib. With a dome like that, I'd guess he's 48. Nope, he's 38.

So I went back to my spreadsheet and entered my "estimated" values for the ages of those born in 1968 or after - and look at this - suddenly even a flat trend line is taking off. Because my estimates were completely crap, it boosted the numbers of those apparently born in the 1960s. However, I will defend to the death my "scientific" method for estimating their ages, and if you try and FOI my data, I'll just delete it. And you have no right to my data, even though the taxpayer paid to generate it, because it's mine. Mine, mine, mine. So there.

There is another way to estimate the ages of our MPs, and that is by using the infill method beloved of climate forgers. If you have temperature records from one weather station, but the one next door is on the blink, then you fit the records of the working station to those of the dud one and "complete" your data set that way.

So let's imagine that we didn't have Pwong's age. We could infill it by taking the age of the Senator that she sits next to and then "adjusting" it using some sort of fudge factor. As I couldn't find a photo of Pwong sitting in the Senate, I have substituted one of her with Australia's favourite window licker.

We know the wax muncher was born in 1958. His data of birth is probably the only thing he hasn't lied about, so I will begrudgingly accept it. If we use the method used by the GISS to infill missing data, I'd have to say that Pwong was born in 1960, and use that record in my graph.

Oops. She was of course born in 1968, but what's 8 years between friends?

And thus is data twisted, manipulated and plain made up in order to serve a cause.



Jon Peltier said...

Why do you want to produce a hockey stick? There isn't any such behavior in your data.

It also makes little sense to try to fit the data any closer than in your first chart. There is too much inherent variability in the data. This fit actually makes the most physical sense: people born in the earlier years have largely retired or passed on, people born in later years have not yet gotten into politics, so people in the middle have the greatest representation.

The crazy predictions you show are typical of fitting a polynomial, then extrapolating beyond the extent of the data. Polynomial fits are only good for interpolation within a well characterized data range. Poly fits do not help discerning the underlying mechanism behind the data, and they are outrageously bad at forecasting.

alethiophile said...

Yes, that's the whole point. The CRU 'hockey stick' graph is just as unfounded.

Boy on a bike said...

Hooray! Tom gets it!