How to compare apples with oranges

The Silly is at it again - this time fluffing the results of state schools.

THEY receive more than twice the funding of their public school counterparts but students from independent schools perform no better on literacy and numeracy tests, a new analysis reveals.

A retired high school principal and education consultant, Bernie Shepherd, examined data from the federal government's MySchool website for the NSW Secondary Principals' Council.

He looked at NAPLAN results, financing and students' social background for more than 8200 primary and secondary schools, scrutinising the relationship between funding and results.

I'm glad he got his hands on all the stats, because I sure as hell  haven't been able to get the whole lot in one go.

Here comes the apples and oranges:

Independent schools at the top end of the advantage scale were found to have an average annual income of more than $19,000 per student, compared with an income of about $8200 for each student at a government school.

There are a lot of Independent schools in Australia. There are over 900 in NSW. Only the top 43 charge anything like $19,000 per student (in fact PLC charges $21,492). Notice how he's compared the top independent schools (the top 5%) with the average of all state schools. It would be fairer to compare the most lavishly funded state schools with that top 5%, or all independent schools with all state schools.

Because I've had to extract my numbers laboriously by hand from the MySchool website, my sample is 1/100th the size of Bernie Shepherd's. My sample only has 7 independent schools, but their average fees amount to $12,400. The average of 24 Catholic schools is $9,079. The average of 52 state schools is $9,273.

But when their NAPLAN scores were averaged and converted into a scaled index, government schools on 567 edged out independent schools on 566.

Ha ha - he's done exactly what I did back in March. There are 5 NAPLAN scores - I simply added them together to give a total. His "averaged and scaled index" is simply my total divided by 5.

As I said before, I don't have access to the full range of data, but in my sample of primary schools only, the Micks had an index score of 448 and the state schools 420 - and these are schools from the same geographic areas. I didn't have enough independent schools to give a meaningful result.

The independent schools attracted about $3300 in state and federal government funding, with the bulk of their remaining income provided by fees paid by parents. Ninety per cent of the public schools' funding came from government.

90% of the public schools' funding came from government? Is he looking at the same data as me? A state school that raises more than 5% of its funds from fees, charges and parent contributions is doing pretty well.

The most advantaged Catholic schools received an average annual income of about $13,000 a student - more than $6000 of it from government - and recorded a NAPLAN index of 565.

Again, he's looking at the "most advantaged" and not the average. As I mentioned earlier, the average is down around $9,000 - and it's actually lower than what state schools get on average. It's amazing what results you can get when you cherry pick the data.

The schools in the sample had an Index of Community Socio-Educational Advantage (ICSEA) range of about 1200.

1200! My sample includes the leafy (and expensive) lower north shore. I have only two schools in my sample with an ICSEA over 1200. In order to get an ICSEA of 1200, you're talking about a very small number of very expensive suburbs. Mosman has an ICSEA of 1206. Cranbrook, one of Australia's most expensive schools is located in Bellevue Hill - it has an ICSEA of 1200. The Scots College is also there - it has an ICSEA of 1188.

''In terms of the cost benefit, if we were looking strictly at educational outcomes as measured by NAPLAN, you would have to think government schools were better value for money,'' he said.

Well, on that note, I'll leave you with this graph showing how much it costs to achieve each point of a NAPLAN score. Schools on the left provide a cost effective education. Schools on the right provide an expensive (and often ineffective) education.

I'll use this matrix again - if Bernie Shepherd is right, then you'd expect to find expensive Independent and Catholic schools over in the right hand quadrants. But funnily enough, you don't. The worst performing schools from both a cost and results perspective (the bottom right hand quadrant) and all state schools.

Frankly, I'd love to get my hands on the full data set for MySchool to see if the results hold true when all the numbers are crunched. I think Bernie's analysis is interesting, but he's engaged in cherry picking and comparing like with not-like.