Opinion polls and pitfalls

Interview with Venkatesh Athreya, economist.

Venkatesh Athreya.

THE noted economics professor Venkatesh Athreya cautions against the media’s projection of a wave in the ongoing general election. He debunks several myths by exposing methodological flaws in the opinion polls and raises concerns around the transparency of the poll methods and the lack of formulae to derive seat shares from vote shares expressed through voter preferences, which is so essential in multipolar contests.

On whether there is a Modi wave

There is no basis for interpreting the results of various opinion polls to endorse the claim of a Modi wave. There is no great uniformity among these polls, in terms of either sampling design or other aspects of methodology. Nor do all surveys ask the same or even similar questions in a manner that evokes an identical answer across polls. All of them have varying degrees of error in different directions. With all this, it would be difficult to be certain about the presence or absence of any wave. When exactly one can talk about a “wave” is not clear. Moreover, some survey questions ask about party preferences, others about individuals as “prime ministerial candidates”. The choice sets are different for these two kinds of questions.

Perhaps, the strong anti-incumbency sentiment against UPA [United Progressive Alliance] II will generate what seems like a wave in bipolar contest situations. Across India, bipolarity is not a common feature. There are specific regional characteristics and a great deal of diversity. Responses to a question on preferences with regard to prime ministerial candidates could diverge a great deal from preferences across parties in each constituency. Part of the perception of a wave comes from attempts to present the election process as if it was a presidential contest a la the United States. If there are only a handful of persons under serious consideration as prime ministerial candidates, a wave-like appearance is more likely. But none of the polls I have seen so far give any credible argument for a “wave” phenomenon. This has also to do with the weaknesses of these polls in terms of transparency and rigour in sampling design and fieldwork execution.

On whether the media are guilty of manufacturing a Modi wave

With due respect to columnists and anchors, there is no reason to believe that the assumptions they make are grounded in opinion polls, whether done properly or not. What we often get from them are opinions, based on some subjective assessment, perhaps taking off from available opinion polls and other sources of uncertain/unknown reliability. Certainly, none of the opinion polls I have followed in the electronic and print media have had anything to say on non-sampling errors or on the correctness of their formulae for moving from vote shares to seat shares. It may not be quite correct to tar the entire media with one brush and charge them with manufacturing a wave in favour of Modi. That effort has come particularly from the huge advertisement budget of the BJP spent on creating a Modi brand. However, it is true that there have been several over-the-top stories in the media on the so-called Modi wave without any rigorous substance. It may be uncharitable to suggest that some sections of the media that have in the past been largely secular in orientation may be buying insurance, but the fact that such a thought even occurs says something. Finally, the role of corporate interests in manipulating both the conduct of the polls and the communication of their results on media platforms cannot be altogether ignored.

On biases in opinion polls

Given the continued weaknesses in respect of methodological transparency in many of these polls, it is not possible to say categorically that biases are no longer there. But criticisms and public debates over time have made pollsters more sensitive to these issues. It still does not follow that the biases have been eliminated. For instance, you may have a well-designed sample intended to ensure appropriate coverage of all sections, but you may not have much control over the quality of fieldwork, so the required degree of inclusiveness may not be ensured. While sampling design can be improved, it carries with it the implication of a higher cost. Most of these polls are in the nature of hothouse exercises, done to meet near-impossible deadlines. Moreover, the mode of communication/presentation of the results of these surveys, to an audience which is always presumed to be in a hurry so that the focus is on bytes, also has its pitfalls. Overall, one can say that there has been greater sensitivity to these concerns over time.

On regulating the opinion-poll industry

With the increasing reach of the media, especially TV and the Internet, it would be difficult to argue that opinion polls have no effect, one way or the other, on voter choices. But the case for regulation can come from the observed fact that many of these polls lack scientific rigour and are often opaque with regard to methodology. There could be some specific norms evolved with regard to transparency covering all aspects of the exercise, from sample design to fieldwork to analysis to the presentation and communication of results. The relationships, if any, between the agencies commissioning the polls, those carrying them out and the political parties should be made explicit as well.

But a blanket ban may not be warranted although the phenomenon of paid news suggests that the phenomenon of paid polls cannot be assumed away.

Two major concerns

My main problems with polls in India centre around two issues.

One is that they are often non-transparent about the methodology not just of sampling but of the manner in which fieldwork is carried out. I would like to know, for instance, the quality of training that the field personnel received, their educational qualifications and their experience in interacting with people and doing surveys. One would like to know how the questions framed in the survey instrument were translated and posed in the field. One would like to know the deadlines and the time spent in the field per respondent, and so on.

A particularly important instance of non-sampling error occurs in field interview situations where the respondent is not in a position to tell the truth. This can happen in areas where a particular party may be practising strong-arm methods to control the local population’s expression of views or their actions as well. Examples should not be difficult to identify.

The second point is that we have no robust formula for going from voter preferences expressed as vote share to seat shares. In a bipolar context and contest, this would not matter much. But when contests are multipolar, in our first-past-the-post system, vote shares do not easily translate into seat shares. Further, the level of aggregation at which one is making a prediction matters. For instance, if regions within a State differ systematically in terms of the electoral strength of various political forces, an overall prediction for the State can go quite wrong unless the sampling design has taken this regional heterogeneity into account.