Forecasts and outcomes

Electoral Outcomes and Poll Surveys: Assembly Polls 2001.

THE results of the Assembly elections have brought to public attention the question of the validity and credibility of poll forecasting efforts in India. Four pre-poll surveys received coverage in the English language print media during the run-up to the elections. Subsequently, programmes relating to an exit poll conducted for two television channels and post-poll comments on the pre-poll and exit poll efforts continued sporadically. It would be interesting to take stock of the performance of the pollsters, and to try and draw some lessons from them for improving the reliability and credibility of poll forecasting in the Indian context.

The four pre-poll surveys were those conducted by India Today-ORG- MARG, The Week-Sofres-Mode, Outlook-CMS, and The Times of India-DRS. The exit poll was done by DRS for two television channels. The pre-poll surveys used different methodologies and sample designs, and were conducted at different points in time. Their predictions and those of the exit poll were different from one another. And the actual electoral outcomes differed in significant respects from both the pre-poll surveys and the exit poll.

The results of the Assembly elections are shown along with the predictions of the pre-poll surveys and the exit poll in Tables 1 and 2. Table 1 compares the shares of vote obtained by the major political formations in the fray in each of the States with those predicted by pollsters. Table 2 provides the corresponding comparison for seats obtained. Table 3 summarises the available information on methodological and other features of the pre-poll and exit poll surveys.

How have the pollsters performed?

* All of them have got Kerala's electoral outcome broadly right, though all except the India Today-ORG-MARG survey have underestimated the UDF vote-share by a margin of 1-2 percentage points, and without exception all have underestimated the LDF vote-share by margins varying from 5.7 percentage points to 1.6% points.On seat projections, both Timespoll and the exit poll - both done by DRS - were almost bang on target, while the others erred on the side of caution!

* Practically all the pollsters concurred, and as it turned out, correctly, that the AIADMK front would get a higher vote-share, but except for The Week-Sofres-Mode poll, all the others grossly underestimated the vote-share differential between the two fronts in Tamil Nadu. The Week poll overestimated the difference slightly. On the matter of seats, both The Week and the Outlook poll were correct in predicting a decisive majority for the AIADMK front, with the former overestimating and the latter underestimating the number of seats for the front.

* The two pollsters who made predictions on the Assam verdict - Outlook (pre-poll) and DRS (exit) - got the direction of vote-share differential between the Congress(I) and the AGP-BJP front right, although both overestimated the Congress(I) vote-share by a sizable margin. Their seat predictions have been quite close to the target for the AGP-BJP combine, and on opposite sides of the actual outcome in the case of the Congress(I).

In the case of West Bengal, all the pollsters were way off the mark, in terms of both vote-share and seats for the Left Front and the Trinamul-Congress alliance. They all underestimated the Left Front vote-share by margins varying from 4.4 per cent to 5.9 per cent. Symmetrically, they overestimated the Trinamul-Congress alliance share by around 3.4 per cent to 7.3 per cent. The Week, which got Kerala and Tamil Nadu reasonably right, was the worst performer in this regard, but the DRS polls - both pre-poll and exit - were not far behind! While the Outlook poll hedged its bets on the seats forecast, the India Today poll at least got the general direction right when it came to the seats forecast, though it was still wide off the mark.

What is one to make of this rather mixed bag of performances by the pollsters? To be fair to the pollsters, one can straightway admit the enormous practical problems that confronts anyone seeking to make poll predictions in the Indian context. First of all, there is the sheer size of the electorate. In the Assembly elections to four States and the Union Territory of Pondicherry, the total size of the electorate was nearly 13.3 crores. There were over 824 constituencies and 1,57,837 polling stations. Some 9.17 crore people, accounting for 70.5 per cent of the total electorate, cast their votes.

The States involved were spread over a wide territory. The electorate was heterogeneous in terms of language, culture, rural or urban residence, levels of formal education and many other variables of relevance.

All these features do not, of course, make forecasting necessarily less reliable. But they do imply high costs and time constraints in getting the poll done, and the data needed to be be processed quickly to meet print or electronic media deadlines. Invariably, corners are cut, and rigour in terms of sampling design and an adequate sample size often suffers. Besides, even the best sampling design is by itself inadequate to ensure reliable predictions. There are non-sampling errors such as inadequate quality of actual data collection in the field. Voter intentions may change between the time the survey was done and the actual date of polling, especially if some significant events that are germane to the voting decision occur in between.

Then there is the very difficult and complicated problem of converting vote-shares into seats in an electoral system where the first-past-the-post is the sole winner. Moving from vote-share predictions to seat forecasts is especially complicated when there are more than two serious contenders or contending political formations, as is often the case. Again, when there are two nearly equally strong contending formations, even a small change in vote-shares can imply dramatic changes in the number of seats won. Kerala is an example where even a small swing can make a considerable difference, and a third force deciding to transfer its votes to one of the two main formations can also affect the results decisively.

The sharp decline in the number of seats for the LDF has come about in good part because of transfer of votes from the BJP to the UDF, together with a small decline in the LDF share of votes compared to the previous elections. In a way the pollsters got Kerala right despite going wrong about the LDF's vote-share, because of the grater degree of bipolarity whereby ground lost by the BJP was ground gained by the UDF.

Having said this, it must also be noted that the pollsters have not always been adequately transparent in spelling out their sample design and in explaining the details of their methods and processes of data collection. While some of them mention that their sample is a stratified random sample, others simply state that care was taken to see that all sections of the electorate were adequately represented in their sample, leaving us to wonder whether the sample was purposive rather than random - in which case arriving at population estimates of voter preferences from sample data becomes an unreliable and arbitrary exercise. Even with a stratified random sample, one needs to know the exact sampling design to be able to cross-check estimates arrived at for the population as a whole.

Finally, the formula for conversion from vote-share to seats invariably varies from pollster to pollster, and involves some degree of informed judgment. Besides the lack of transparency on the part of many pollsters, there is the question of implicit biases. It is noteworthy, for instance, that in the case of both Kerala and West Bengal the Left's share of vote has been significantly underestimated. One must not of course rush to conclusions of any particular bias, but there is nevertheless something to ponder over here. Is it possible that since most of these polls are commissioned by the media, their own biases/perceptions cloud their judgments?

While no definitive answers are immediately possible, it must nonetheless be strongly demanded that pollsters in India must be far more transparent and forthcoming with the methodology as well as the subjective judgments that enter into their formulae for conversion of vote-shares into seats and their sample design as well as processes of data collection. This alone will ensure that their predictions will become credible eventually.

Venkatesh Athreya is Professor and Head of the Department of Economics, Bharathidasan University, Tiruchirappalli.