How You Ask a Question Matters

Looks like Gavin Newsom is going to comfortably avoid being recalled. But for a while there the polls indicated a close race. One poll especially caused panic among the Governor’s backers: a SurveyUSA poll released on August 4th that showed the Yes vote leading 51-40%. Some other polls at the time showed a close result, but none had the pro-recall forces in the lead (let alone by 11 points). The tightening race lit a fire under the Democrats to take the recall seriously. Six weeks of concerted effort seems to have paid off with a victory for the No side.

But turns out that a slightly-misworded question in that August poll may explain its outlying result. All voting polls have models to determine how likely a respondent is to actually vote. They ask questions about voting history (i.e., “Did you vote in the last election?”). They also ask about enthusiasm for the upcoming election. The August 4th survey tried to gauge this by asking voters how certain they were to vote or not vote “in the recall election to remove the Governor.” See the problem with that question? If you were opposed to the recall, you might misidentify yourself as unlikely to vote, even if you were going to vote. Thinking, “I’m not going to vote to ‘remove the Governor’, so I’ll answer no to that question”. Some percentage of likely “No” voters were thus excluded from the survey results.

SurveyUSA acknowledged this mistake and reworded the question in future polls, asking participants how certain they were to vote or not vote “in the recall election”. The results of these future polls showed Newsom with a comfortable lead.

We’ll never know how much this seemingly-innocuous question affected the results of the August 4th poll. But given just how much of an outlier it was, I suspect it had a big impact. And it just goes to show a couple of things. First, even professional, non-biased pollsters make mistakes. And second, how you ask a question can make a big difference in the answers you get.

Leave a Reply