An article in the Australian, Ratting on the Ratings, gives an excellent example of the GIGO Fallacy. David Woodhouse is the executive director of the Australian Universities Quality Agency, and he discusses the flaws of the University ranking system employed by Shanghai University:
Now, this is not to say such a system in inherently useless. It gives us information which we can use to inform judgments about universities. However, it is to say one needs to be aware of the limitations of the system and that one should not take its results as “Gospel”.
THE central criticism of whole-of-institution rankings relates to the methodology that addresses quality in a superficial way but projects a complex image. Most rankings rely on two types of data: information from institutions that may not be validated and data obtained from opinion polls in the name of "expert opinion". With both components providing shaky foundations, the use of complex formulas with weights and indicators helps to project a pseudo-scientific image to outcomes that may be statistically irrelevant.Put simply: GIGO. That is, even though a complex calculation is applied to the data to come up with a rank order of universities, given the input data may not be validated and is based on opinion; the output should be treated with skepticism. It does not matter how powerful the analytic tools being applied are, if the raw data being supplied is suspect, any conclusion based on this data is suspect too. Woodhouse goes on:
Rankings rely on quantification, indicators and weights, assigning weights to indicators and using the weighted scores to rank the institutions. This forces the multidimensional quality aspect into a linear scale. In this process, aspects of the institution that cannot be measured with weightings and numbers get distorted… Furthermore, they are sensitive to relatively small changes in the weightings used. Small changes in the weightings of indicators alter the results from year to year without any tangible change between institutions.Not only is the input data suspect, the data is weighted and it only takes a minor change in the weighting to alter the results. This adds to the GIGO.
Now, this is not to say such a system in inherently useless. It gives us information which we can use to inform judgments about universities. However, it is to say one needs to be aware of the limitations of the system and that one should not take its results as “Gospel”.
