Data of questionable merit

Questions about the methodolgy adopted cast a shadow over the latest estimates of poverty by the National Sample Survey Organisation.

POVERTY, or rather the official estimates of it, is back in the news. The latest estimation of the number of poor people in India has triggered a heated debate over the methodology adopted by the National Sample Survey Organisation (NSSO), which is in ch arge of the exercise. The debate is not confined to academia or to esoteric issues relating to the methodologies adopted. The data, and the controversy surrounding them, have been inextricably tied to the fundamental question of what impact the wide-rang ing economic reforms introduced in 1991 have had on poverty. After all, a decline in the proportion of the poor between 1993-94 and 2000, from about one-third to one-fourth, is not an empirical result that can be ignored by those wishing to link poverty to the nature of economic reforms.

Partial results of the latest NSSO survey, conducted between July 1999 and June 2000, which have been highlighted in the media, indicate that poverty levels in India have declined dramatically since 1993-94, the year in which the previous full-fledged su rvey was conducted. The partial results, from the first two sub-rounds done between July and December 1999, indicate that the number of people below the poverty line in rural areas has declined from 37.3 per cent to 27.6 per cent between 1993-94 and 1999 -2000; in urban areas the decline has been from 32.4 per cent to 25.2 per cent.

Not surprisingly, the issue of statistical integrity has assumed a strong political overtone because opposing viewpoints tie their interpretations of the estimates to their respective positions on the economic reforms. Those on the side of the establishm ent argue that a sharp reduction in the number of the poor is a vindication of the theory that market-led growth has improved the living conditions of the poor. However, critics of the reforms view the latest estimates as "statistical jugglery", in line with the trend of official agencies publishing data that suit the establishment rather than the objective needs of society. They have alleged serious flaws in the methodology adopted for the latest survey; they say that the NSSO has made changes in the m ethodology without any consideration to methods of statistical enquiry.(The NSSO, the premier agency, also conducts national-level surveys on employment, consumption, nutrition and other issues.)

Experts from the NSSO and the Planning Commission have expressed scepticism about the data. At the Economic Editors Conference in early October, K.C. Pant, Deputy Chairman of the Planning Commission, announced that a committee of experts would be constit uted to compare the latest survey estimates with those of the 1993-94 survey. However, on October 30, Arun Shourie, Minister of State for Statistics, Programme Implementation and Disinvestment, ruled out a review, claiming that "no cogent reasons had bee n advanced which should lead the government to doubt the comparability of the data."

Officially a poor person is defined as one who is unable to meet a minimum food intake of 2,100 calories in urban areas and 2,400 calories in rural areas. Since the actual calorific intake of people is almost impossible to estimate in an across-the-count ry survey, consumption expenditures serve as an estimate of poverty levels. A person whose spending on food is below the level required to purchase the minimum calorific intake is considered to have fallen below the poverty line. The sample used in the s urvey is then "blown up" to the State and national levels to estimate the number of people who fail to meet this most basic standard, people who are in abject poverty.

Although the NSSO had conducted smaller surveys based on "thin samples", where about 20,000 respondents were quizzed, the results of the latest survey were eagerly awaited because the 1999-2000 survey has been the only one since 1993-94 to use a "large s ample". The latest exercise involved canvassing questionnaires from about 1.2 lakh respondents across the country.

Until the 50th round of the National Sample Survey in 1993-94 the NSSO had been using a method by which respondents were asked to provide details about their food consumption in the previous 30 days. In the 50th round, the concept of a seven-day recall w as introduced, because of the suspicion that the longer time scale caused respondents to miss from memory certain items that he/she had actually consumed. In the 1993-94, the nationwide sample was divided into two separate samples, one using the seven-da y recall and the other the 30-day recall.

Indeed, the shorter recall period did reveal higher consumption expenditure levels and, consequently, lower poverty levels. In 1993-94, poverty levels based on the seven-day recall were about 16-17 per cent lower than those based on the 30-day recall. Pr avin Visaria, Chairman of the governing council of the NSSO, pointed out that between 1994 and 1998 poverty levels in India, based on the seven-day recall method, were lower by about 40-50 per cent than estimates using the 30-day recall.

However, what has raised the hackles of critics is the fact that the same set of respondents were quizzed about their consumption expenditures using the two methods.

Experts in statistics have argued that this kind of a survey would seriously "contaminate" the data because respondents would tend intuitively to multiply the seven-day recall by four when responding to the questions based on the 30-day recall. Field sta ff would also tend to commit the same error when compiling the results. While admitting that the "concurrent collection of data" using the two methods was "unusual and a last-minute compromise", Visaria has claimed that the investigators were instructed to collect the two sets of data "independently". He has also asserted that "these instructions are unlikely to have been overlooked".

However, doubts about the integrity of the data have been raised because for close to two months from the beginning of the survey, until August 1999, the data obtained by using the seven-day recall were collected before the same respondent was approached again with the 30-day recall method. Critics have argued that this sequencing would contaminate the results even more because the four-fold multiplication would come even more easily to the respondents. Visaria has said that instructions were issued in mid-August to reverse the sequencing by posing the 30-day recall first.

An economist well-versed with the methods of the NSSO told Frontline that respondents were likely to suffer "fatigue when approached twice with the same set of questions". This, he said, was particularly so in the case of consumption expenditure s urveys, which were long and detailed. Although the seven-day recall appears to have the advantage of imposing a lesser strain on the memory of the respondent, others argue that a longer recall also offers certain advantages, particularly because of the n ature of poverty. For instance, a shorter recall may be unable to capture the variations in the day to day consumption expenditures. This may be particularly crucial in the case of casual workers or people on the margins of society. The consumption of th e poor varies not only across seasons but also across days and weeks. On the use of the shorter recall, an economist sarcastically commented that a short recall of just a day may yield "excellent data", but this would simply miss out the variations that were an integral part of the daily existence of the poor. In any case, nothing prevents the NSSO from publishing two sets of results, based on two different methodologies. After all, NSSO's employment estimates use different yardsticks.

Visaria has justified the use of the seven-day recall by pointing out that the 30-day recall is not used anywhere in the world. Asked why the NSSO changed the methodology, Visaria said that there was a "viewpoint" that advocated this change, and that the NSSO just wanted to "try this method out". He admitted that two separate samples for the seven-day and 30-day recall "are probably better". He also said that the NSSO was willing review the changes after all the results are compiled by December.

Despite advocating caution in the use of the partial results for definitive conclusions, Visaria has invited criticism by demonstrating in a newspaper analysis that the difference in poverty levels using the two methods has fallen sharply as a result of the changes introduced in the latest survey. However, critics have been quick to point out that the difference between the two estimates could have collapsed simply because the 30-day recall is "contaminated". They allege that estimates using the 30-day recall have lost their independent standing because they are tied to the seven-day estimates because of the "multiplication" factor. Indeed, they point to the estimates based on the seven-day recall method to prove their point that poverty levels in Indi a have actually worsened (see tables). Going by Visaria's seven-day recall figures the percentage of rural poor increased from 19.1 per cent in 1995-96 to 24.8 per cent in July-December 1999 and urban poverty levels increased from 15.2 per cent to 23.4 p er cent. However, because the 1993-94 survey, the last one with a "large sample", did not use a seven-day recall, the latest estimates cannot be compared to the previous estimates.

S.L. Shetty, director of The Economic and Political Weekly's Research Foundation, told Frontline that "there is no statistical or scientific basis for the shift". He also argued that poverty levels were generally on a secular trend. "Povert y levels," he said, "do not change so drastically in such a short period."

Shetty outlined his "broad hypothesis" that poverty ratios have worsened, pointing to a number of economic developments to "corroborate" his position. Since the reforms, per capita consumption of foodgrains has fallen and so has employment levels; these would imply a contraction in incomes and consumption.

Moreover, the curtailment of social expenditures, another facet of the reform process, Shetty says, meant a diversion of incomes by the poor to the costlier private health services, away from the public facilities. This, he argues, would have been the ca se particularly with the marginally poor households. Moreover, agriculture, by all accounts, has stagnated in the last few years, thereby impacting on employment, incomes and consumption of the poor. Shetty says that the higher growth in the services sec tor would not have benefited the poor because there "is less percolation of incomes" in this sector. "Radical changes in methodology are not acceptable in poverty statistics," he said.

A study conducted in 1999 by S.P. Gupta, Member, Planning Commission, revealed that although national income, as measured by the gross domestic product (GDP), increased at an annual rate of 6.9 per cent between 1993-94 and 1999-2000, poverty levels actua lly increased during this period, from 35.07 per cent to 37.23 per cent and employment grew by 0.6 per cent per annum. Gupta's study also indicated that poverty levels were inextricably linked to employment levels, suggesting that stagnation in employmen t was the prime cause of the high levels of poverty in the 1990s.

Academics have protested that the NSSO has made a "fundamental shift" in methodology without getting the opinion of experts. Shetty said that the NSSO ought to have consulted the "academic community at large before implementing such a major shift in meth odology."

The opportunity to answer authoritatively the fundamental question about reforms and its impact on poverty appears to have been lost because of the controversial nature of the survey. In effect, it appears that the shift in methodology will not allow for comparison with the 1993-94 data. Asked if this did not pose a problem, Visaria turned philosophical: "Comparability is after all not the only thing in life."