Random sampling, election polling and lunch
I’ve just started reading a book called Canuckology by pollsters Darrel Bricker and John Wright. It’s intended to be an account of “what Canadians think and why,” based on meticulous market research. Among other things, its introduction contains a rather eloquent summary of what polling actually is.
“Imagine that you have 100 people in a room, and you want to figure out what to order them for lunch. You don’t have a clue what to order, but you know that you don’t have the luxury of asking them each individually what they would like to eat. And, most importantly, you want to make as many people happy as possible. So you decide to ask a few people in the room to give you input.
“You ask everybody to stand up and count off by ones to 10, until each person has a number. Then you write out 10 cards, numbered one to 10, and put them in a hat. You have someone pull a card out of the hat and read you the number. It’s three. You then ask all of the 10 people who said ‘three’ what they would like to eat for lunch. You find out that four people want Chinese food and six want Italian.
“So you order 40 Chinese lunches and 60 Italian lunches. To everyone’s amazement, the vast majority of the people in the room are thrilled with what you ordered…
“Now, what about that ‘plus or minus 10%, 19 times out of 20′ stuff? What does that mean? Let’s go back to the lunchroom. It we succeeded in selecting a fair sample by picking all of the number threes, and didn’t actually select all the vegetarians or low-carb dieters, the rules say we should still expect to be off by a little bit. Not everybody will be satisfied with the lunch order. That’s the margin of error. It might turn out that 10 people in the room were stuck with good they didn’t really want, which would mean our margin of error was 10%. If we picked another random sample, we shouldn’t be surprised if this time five people wanted Chinese food and five wanted Italian. Maybe we should have ordered 50 Chinese lunches and 50 Italian lunches. Great, but even if we had, we shouldn’t have expected fewer people to be upset in the end, based on asking a sample of just 10 people and making a prediction about the rest.
“The really sad news is that even if we followed all of the rules perfectly and did our lunch poll 20 times, one time we would be wrong by even more than 10%. Maybe that was the unlucky time we happened to ask the ten people who were allergic to pasta. We get used to a margin of error in even the best-performing polls, and over time, we learn to predict what that margin probably is.”
The first section of the book – entitled “Where Do You Belong?” – deals with the sensibilities of Canadians by region. As I read my way through it, I might even post an overview of each chapter in this space between now and the election. Torontonians in particular, especially during an election, have a way of thinking of the country as “us versus everyone else” – and if it’s only because we assume that’s the way they think about us, then so be it. Nonetheless, it might be nice to get a better sense of where our fellow Canadians are coming from.