Greg Sheridan in Public opinion is putty in academic hands,
points out a case of Observational Selection, in this case – advocacy research – the researcher may consciously or unconsciously seek to confirm his or her view. He is talking about the Lowy Institute's recent survey on public opinion and foreign policy.
…[The Lowy Institute] will reinforce the sadly quite narrow range of opinions held among professional academic and quasi-academic foreign policy commentators. Two conclusions of this survey demonstrate the point. They are that Australians have a deep commitment to international law, and would never support the US militarily to protect Taiwan from China. You could not find two more perfect expressions of foreign policy orthodoxy. How did the poll produce such results? The answer lies in the questions.
On international law, respondents were asked to choose between these alternatives: "Australia should rely on international law even though decisions may go against us OR Australia should do whatever benefits us the most in any given situation regardless of what international law says."
Not surprisingly, the first alternative gets the majority vote. But what would the answer be to a question phrased: If a group of officials from non-democratic countries with appalling human rights records operating in a UN committee directed Australia to do something the majority of its people thought was wrong, should Australia follow international law even though it involves doing wrong or should it do what it believes is right?
In reality, that is much more how questions of breaching international law would present themselves to Australian opinion. Even to ask a question about so loaded a concept as international law without giving some idea of what you mean is inherently dishonest. The pollsters' question on Taiwan is even more loaded. Respondents were asked to agree or disagree with the proposition: "Australia should act in accordance with our security alliance with the US even if it means following them to war with China over the independence of Taiwan."
Not surprisingly, a majority would not sign a blank cheque for a hypothetical war. A more realistic question would have been: Do you think China is justified in mounting a military invasion of Taiwan, even if it causes tens of thousands dead, in order to reunify it with mainland China?
Sheridan's examples demonstrate the problem with not knowing the questions behind statistics – they are essentially meaningless without this information. This is exacerbated when the statistics are based on completely hypothetical situations without any context given to the interviewees. Sheridan himself, it could be argued, is just as biased with his replacement questions. No doubt they would give the answers he is looking for. It's also worth noting his replacement questions are significantly different to the originals.
points out a case of Observational Selection, in this case – advocacy research – the researcher may consciously or unconsciously seek to confirm his or her view. He is talking about the Lowy Institute's recent survey on public opinion and foreign policy.
…[The Lowy Institute] will reinforce the sadly quite narrow range of opinions held among professional academic and quasi-academic foreign policy commentators. Two conclusions of this survey demonstrate the point. They are that Australians have a deep commitment to international law, and would never support the US militarily to protect Taiwan from China. You could not find two more perfect expressions of foreign policy orthodoxy. How did the poll produce such results? The answer lies in the questions.
On international law, respondents were asked to choose between these alternatives: "Australia should rely on international law even though decisions may go against us OR Australia should do whatever benefits us the most in any given situation regardless of what international law says."
Not surprisingly, the first alternative gets the majority vote. But what would the answer be to a question phrased: If a group of officials from non-democratic countries with appalling human rights records operating in a UN committee directed Australia to do something the majority of its people thought was wrong, should Australia follow international law even though it involves doing wrong or should it do what it believes is right?
In reality, that is much more how questions of breaching international law would present themselves to Australian opinion. Even to ask a question about so loaded a concept as international law without giving some idea of what you mean is inherently dishonest. The pollsters' question on Taiwan is even more loaded. Respondents were asked to agree or disagree with the proposition: "Australia should act in accordance with our security alliance with the US even if it means following them to war with China over the independence of Taiwan."
Not surprisingly, a majority would not sign a blank cheque for a hypothetical war. A more realistic question would have been: Do you think China is justified in mounting a military invasion of Taiwan, even if it causes tens of thousands dead, in order to reunify it with mainland China?
Sheridan's examples demonstrate the problem with not knowing the questions behind statistics – they are essentially meaningless without this information. This is exacerbated when the statistics are based on completely hypothetical situations without any context given to the interviewees. Sheridan himself, it could be argued, is just as biased with his replacement questions. No doubt they would give the answers he is looking for. It's also worth noting his replacement questions are significantly different to the originals.
