In the past couple of months, twitter has been rocked by quite a few surveys, conducted by random media houses. The most infamous being, the Thomson Reuters one:
India most dangerous nation for women: Poll
There have been many writers commenting on the said survey. I will however stick to the methodology of surveys, and how most media houses are increasingly fudging there and getting away with it. A case in point is again the same survey which had used a sample size of 550 respondents.
The biases associated with sampling for surveys, is chapter 101 for most students of statistics. All we need to do is give a good look at the surveys floating around in Indian media to identify the bias. The three most common biases that most Indian media houses display in their multiple meaningless surveys are:
- Selection bias – This is most commonly associated with a small, not random enough, sample. A good sample, should ideally be representative of the population, which in the case of India, is diverse and resides across the various towns, cities and villages of our countries and not in air-conditioned studios. A survey conducted where the respondents are restricted to the Lutyens bungalow, can hardly be a fair representation of the country.
The Thomson Reuters survey is a prime example of a selection bias. The sample selected was completely unrepresentative of the population as a whole. 550 people was too small a sample size to predict anything about a population as large as a 100 plus crores. Furthermore, the respondents of the survey were drawn from Europe, Africa and America, where the respondents had been fed with media images, stories that confirmed to a particular stereotype. Additionally, there was no attempt to gather responses from Indian villages, cities, towns the actual place where the most affected population resides. - Observer bias – No matter how much the surveyor might try or not, sometimes the questions themselves are framed in such a way that the survey comes out with entirely predictable results. In other words, the questions are worded to reconfirm the biases of the surveyor. Again, another prime example of how questions are framed across media houses in India, specially the left-leaning ones, who pose their questions, solely with an attempt to fox ordinary residents into giving anti-government responses that suit their political agendas. Case in point the widespread surveys conducted in the wake of demonetization, which appeared to reaffirm the deep pain experienced by the left cabal politician, though the average man on the street however had a different experience. (Example: Indians faced ‘serious problems’ due to note ban: Survey)
- Survivorship bias- Where the respondent is ruled out probably because he did not survive, literally or metaphorically. History is rife with such examples where the respondent was excluded because they were not around to give the response. For instance, there are frequent surveys highlighting agrarian distress across countries of the world. Not many of these surveys, would actually have been conducted by getting responses from the farmers who were experiencing distress. In addition, again in the Indian context, there are some surveys that are trimmed post the receipt of response, where only the respondents with the right sort of responses are included in the survey.
Though these are not the only types of survey biases, they are the most prevalent ones in the Indian context. And given that we are now entering the election year, there are going to be more such, high decibel, cacophonous surveys that have been fixed in various ways.
As informed citizens, it would be in our interest to delve deeper into the survey methodology, fine print, before falling victim to it. So, the next time, a media house talks about a survey, let us ask them to divulge details of sample selection and survey methodology.
