Pollsters versus voters

On why psephologists got it all wrong.

THE verdict thrown up by the Indian electorate in the elections to the 14th Lok Sabha is the most dramatic since that of the 1977 parliamentary elections, which threw out the party that declared an internal Emergency and severely abrogated civil and democratic rights. Not surprisingly, the verdict has proved most pre-election opinion polls and the exit polls to have been way off the mark. The key features of the verdict missed by the pollsters can be summarised as follows:

* The National Democratic Alliance (NDA), which was widely forecast to be the alliance that would obtain the highest number of seats among all the formations in the fray, and in any case not less than 230, ended up with just 185 seats.

* The Indian National Congress (INC) and its allies, despite being seen as narrowing the gap in the later stages of the run-up to the elections, was not expected by most polls - opinion polls and exit polls - to reach the 200 mark (except for the NDTV-Indian Express exit poll of May 10, which gave the Congress(I)-led formation between 190 and 205 seats). It ended up with an impressive tally of 217 seats.

The Bharatiya Janata Party (BJP) was widely expected to improve upon its tally of 181 seats in the 13th Lok Sabha. It ended up with 138 seats.

The Congress(I) was not expected to be the single largest party, but it ended up with 145 seats, seven more than the BJP.

Most important of all, and generally least commented upon, the parties not belonging to either the NDA or the Congress(I)-led alliance were expected by most pollsters to get anywhere between 75 and 120 seats, with the mean forecast around 100. They ended up with a remarkable 137 seats.

The enormous margins of error in the various poll forecasts become even more sharply evident if one looks at early forecasts and final outcomes. Thus, the India Today-Aaj Tak opinion poll conducted between January 9 and 17 forecast 105 to 115 seats for the Congress(I) alliance and 330-340 seats for the NDA. The NDA ended up with only slightly more than half of the prediction while the Congress+ had a final tally which was more than twice of what was forecast. Similarly, the Zee News poll, conducted between March 15 and 20, forecast just 75 seats for the parties not belonging to either of the major formations. These parties ended up with nearly double the number of seats forecast. By the time of the second round of opinion polls, held mostly between mid-March and mid-April, the NDA was marked down a little, but was still forecast to get not less than 260 seats. The average forecast was around 275 to 280, a clear, if slender, majority. The Congress(I) alliance, on the other hand, was seen as having improved its position, but was still not seen as likely to go beyond 170.

The first serious "corrections" to the predictions came with the exit polls of April 20 after the first round of polling, but there was no unanimity even in the direction of corrections. The NDTV poll saw no change in the NDA tally accompanied by a slight improvement in the Congress(I) alliance's tally, while the India Today-Aaj Tak poll showed improvement for the NDA and decline for the Congress(I)-led alliance. After the second round of exit polls on April 26, the NDA was marked down by the three major pollsters - NDTV/Indian Express, India Today/ Aaj Tak and Star News - and the Congress(I)-led alliance marked up. Interestingly, this round of corrections was reversed after the exit polls of May 5 in which all pollsters enhanced the likely NDA tally and marked down either the Congress-led alliance (NDTV/Indian Express, Star News) or the rest (India Today/Aaj Tak).

Even after the final round of exit polls held on May 10, none of the pollsters gave the NDA less than 230 at the minimum, while three pollsters - Star News, Zee News and Sahara TV - projected the NDA to get at least 263 seats. The final forecasts for the Congress(I)-led alliance varied from a minimum of 171 seats (Sahara TV) to a maximum of 205 (NDTV/Indian Express). The final seat forecast for the remaining parties varied from a low of 86-98 (Star News) to a high of 100-120 (NDTV/Indian Express). Thus, none of the pre- and post-polls on the Lok Sabha elections got the numbers right, despite several revisions based on information generated by four rounds of exit polls and, earlier, two rounds of opinion polls.

IN a country as vast, heterogeneous and complex as India, with a huge electorate, the conduct of opinion polls designed to elicit voter preferences and to make forecasts of seats for various contending political parties/formations faces many challenges. The sampling design is critical, and has to be sufficiently sophisticated to take into account the significant heterogeneity across regions, social classes, residence (rural/urban), sex and other key variables. The sample size has to be large enough for robust predictions, not only at the national level but also at the State level.

Getting the sampling design right is only the starting point. The quality of the fieldwork process is equally important. The design of the questionnaire, ensuring that there is no ambiguity and that there are cross-checks to assess the consistency of responses, is important. Perhaps even more important is the training given to the field workers who will canvass the questionnaire with the respondents. Large-scale surveys of voter preferences should ideally have adequate numbers of well-trained and highly motivated field workers so that accurate information can be gathered from a large number of respondents in a relatively short period of time. Given the time and budgetary constraints within which most electoral opinion polls work, and the assumed need to meet media deadlines in a highly competitive game, the quality of investigation often suffers, and sizeable non-sampling errors lower the reliability of the data.

Further, even with the best sampling design and quality fieldwork, one can at best arrive at voter preference proportions or percentages. In an electoral system of the kind we have, where the winner takes all, the process of moving from estimates of vote shares to estimates of seat shares is complicated, dependent on a number of factors, such as the degree of polarisation in the contest and the degree of unity among the constituents of each of the coalitions of parties or formations contesting the election and necessarily subject to some subjective judgment.

One can, therefore, understand that opinion polls seeking to make assessments of voter preferences and, based on such assessments, forecasts of seats of various parties or combinations of parties with varying levels of cohesiveness and unity on the ground, are very difficult exercises, with the risk of sizable margins of error. Nevertheless, they are not impossible exercises, and can be made reasonably robust by appropriate research design and good quality fieldwork.

EXIT polls, given the manner in which they have generally been conducted, are much more prone to error, and should be considered less credible in the Indian context. This is because rarely are efforts made to obtain a truly random sample, in the scientific sense of giving every unit in a population an equal chance of being selected and not in the "popular" sense of accosting arbitrarily some voters as they come out of polling booths. Interviewing voters as they come out of polling booths also makes for a far-from-ideal situation, where the respondent may not give a genuine response for a variety of reasons. Over the years, pollsters in India - the serious ones at any rate - have been fine-tuning and improving their methodology, trying to improve the quality of the field work, and have had a modicum of success when it comes to predictions. They have also, on several occasions, got it all wrong. Most pollsters do not provide a clear and transparent statement of the methodology followed in detail and the strengths and weaknesses of the data gathering process in the field.

Often, the process and method by which vote shares are converted into seat shares is also not made clear. These issues need to be addressed seriously if electoral opinion polls are to acquire any kind of credibility. This is important, since even opinion polls conducted with sound methodology can go wrong for more than one reason, but a transparent statement of the methodology pursued will give the public a clear idea of the reliability of such polls in the absence of unforeseen external shocks (such as the bomb blast in Coimbatore in 1998, which made an otherwise methodologically sound forecast appear way off the mark in relation to the final outcome), and will not destroy the credibility of a potentially meaningful exercise guided by sound, scientific, statistical methods.

One final comment is in order. In the case of many of the opinion polls in the run-up to the elections, media perceptions seem to have influenced interpretation of the data. The hype over the assumed strength of the NDA combination and its allegedly creditable record in terms of stability, growth and development, pushed by the NDA's "India Shining" campaign, seems to have clouded professional judgment when it came to reading the data carefully for predictive purposes. Nowhere is this more evident than in the consistent underestimation by all pollsters at all stages of the number of seats likely to be bagged by the parties not belonging to the NDA or to the Congress-led alliance. This has been the case with opinion polls pertaining to previous general elections as well. The underestimation of the Samajwadi Party in Uttar Pradesh, and of the Left parties generally, has been a recurrent feature in successive elections.