I read this article in the FORUM, March-April 2007 issue, written by Celeste Ann Castillo Llaneta — a faculty of UP Diliman. And i think many people should know about this stuff, unfortunately FORUM is only published within the university, so I posted the article here for other people to get enlightened of the things they should know behind economic surveys.

The Game of Numbers: Reading Economic Surveys

“Hunger remains at a record-high of 19%,” announced a March 19 press release from the polling agency Social Weather Station [SWS]. The number has remained more or less unchanged since November 2006 survey. Another SWS press release, dated March 22, showed that 53% of Filipino families think of themselves poor, roughly the same number arrived at by similar surveys dome in February and November.
The government was quick to react to the figures, with Press Secretary Ignacio Bunye issuing a statement saying that official data on poverty incidence shows improvement with each passing year. “For instance, “the statement notes, “the average self-rated poverty during different administrations is as follows: Marcos 64.5; Aquino 63.4; Ramos 62.2; Estrada 59.56 and; Arroyo 56.32.”
In other news, the Bangko Sentral ng Pilipinas recently launched the first Nationwide Consumer Expectations Survey for the first quarter of 2007. The survey showed that the overall consumer confidence index for the whole country was at -33.3 percent, with consumer sentiment expected to improve in the next quarter and to turn positive within the next year. This, Secretary Bunye finds encouraging. “The Filipino people have more than enough reason to be optimistic as the economy continues to ride on the crest of stability and resiliency, which not even poisoned politics can shake.”

Confusing and at times seemingly contradictory though they may be, it is through these figures, numbers and statistical data that people may be able to view even a part of the huge, complex picture that is the economic state of our country. A survey, simply put, is  a way to gather information. But surveys have also become a means through which people can express their views in a way that can have impact on policy-making. In an online essay titled “What SWS Stands for: Democratic Discourse” dated January 19, 2006, Dr. Mahar Mangahas, president of SWS, writes; “Our emphasis on surveys…is premised on our faith that Filipinos answer survey questions sincerely and that their answers deserve to be publicized and brought to bear on important public issues of our times.”
Fixing the Standards
Generally speaking, two kinds of curveys are used to asses the economy. The first is the one favored by the government and is usually conducted by development agencies such as the National Statistics Office [NSO]. This includes the Census, the Philippine Labor Force Survey, the Customer Satisfaction Survey, etc. According to Dr. Emmanuel de Dios, professor and chair of UP School of Economics, the government surveys are based on a statutorily or externally defined poverty line. “You have fixed set of goods and you want to find out whether people can or cannot affprd it. That’s the way government defines poverty.”
For instance, the Family Income/Expenditures Survey [FIES] conducted every three years by the National Statistical Coordination Board uses a predetermined subsistent bundle, a metaphorical basket containing a certain amount of goods, both food and non-food expenditures, and proceeds to find out how many people can afford this basket on their income. The government sets a certain threshold based on physiological requirements as well as certain cultural factors; people who fall below this threshold are considered poor.
“In principle, what should happen is that this basket should be unchanging,” De Dios says. “Even as income rises and fall and people move above and below the threshold, the basket of goods remains fixed. Based on that, the government is saying that the percentage of people who fall below that threshold has been declining. Very slowly, but declining nonetheless.”
Poverty surveys done by international agencies are even simpler. They ask how many people in a particular country have incomes below $2 or even $1 a day, and they make comparisons acroos countries. But basing poverty or the quality of life on a fixed standard tends to invite more questions, according to De Dios. “Why $1 or $2? What do people think about the standard of living that $2 can buy? Maybe for some people, $2 is still too little. If you have an income of $3 a day, do you consider yourself wealthy?
The Personal Point of View
The main weakness of a fixed-standard survey is that it fails to accurately reflect the ever-changing values of the population. This is what  the second type of survey based on the perceptions of people — for which agencies such as SWS, Pulse Asia, Ibon Foundation and Vox Populi are known — is particularly good in capturing. “Agencies such as the SWS and Pulse Asia argue that, ultimately, what’s important is what people think,” De Dios points out. “This lies behind political decisions after all.”
Subjective surveys have an additional edge: they are much easier to conduct, not to mention much less expensive. The FIES, for instance, can only be conducted every three years because of the massive amount of work and money it requires. In fact, in July last year, the NSO announced the cancellation of the 2006 Cencus due to — of course — lack of funds. Subjective surveys, on the other hand don’t need a large sample, since 1200 respondents would suffice to gain a generally accurate idea of the perceptions of a population of 87 million. How that happens is part of the magic statistics. “Statisticians will always tell you that as long as you mix the soup well, you only need a teaspoonful to taste what it is,” says De Dios. “As long as you sample properly, even 1200 will be enough.”
Hitting a Moving Target
Of course, subjective surveys also have their disadvantages. Some people dislike them because, De Dios says, it is like trying to hit a moving target. Subjective surveys are based on people’s perceptions and perceptions are relative. A country where people’s aspirations are much higher might actually be better off compared to Bangladesh or some countries in Africa, but those people might still consider themselves poor.
Another weakness of the subjective survey, says De Dios, is that perceptions can be influenced not just by current conditions, but by future as well. For instance, right after EDSA revolution in 1986 fewer people said they were poor, perhaps because of high optimism about the future. “The subjective survey is a very complicated thing. It reflects the future, the mood, the relative standards of people, the inequality in society. But it is for that reason, the survey agencies would say, that subjective surveys are good. Because they include everything and people will ultimately make decisions based on everything.”
To Err is Human
How accurately can these surveys reflect the greater picture, given that most times they all seem to be reporting different figures? Prof. Solita Colla Monsod, also UP School of Economics faculty member, says, “First of all, you have to make sure that the survey agency is using the correct statistical methodology. Secondly, you must understand that the survey is a snapshot of a point in time. If there is a difference in the time frame between one survey and another, then it is not correct to  think that they should have the same picture. Conditions are constantly evolving. If they are using the correct method and are talking about the same point in time, there should be no difference in the results whatsoever.”
She stresses that if one survey reports a 52% poverty level and another survey shows 55%, then there really is no significant difference between them, thanks to the margin of error of plus or minus 3%. “In other words, ” Monsod says, “it is the interpretation of the survey, not the data, that is faulty.”
Margin of error, or the lack of awareness of it, is what is proving to be a major stumbling block in the interpretation of survey results. The margin of error refers to the amount of random sampling error in a survey. Simply put, if the same survey is conducted a hundred times, in the majority of those times the results would fall within a certain number of points — three in this case — above or below the percentage. The margin of error increases as the sample size decreases; the larger the sample, the less the chances of error. “It’s part of the statistical theory that when the sampling was done, you can make these mistakes simply because of the way you fixed the respondents that you were sampling,” De Dios explains.

Any reputable survey agency, Monsod says, will make known the margin of error in a survey. The problem is, the media often don’t know what to do with it, or even what the margin of error means, Newspapers tend to publish press releses from the survey agencies without even taking a look at the data to find out if the analysis in the press releases is really up to scratch.
Monsod cites two hunger incidence surveys conducted by the SWS, one done for the third quarter of 2006 with the headline “Hunger hits record 16.9% of families again,” and the other done for the fourth quarter of last year, this time headlined “Hunger at new record-high of 19% of families.”
“I actually called Mahar’s [Mangahas’s] attention to this,” she relates. “I said, ‘You should not say that [hunger has hit a new record high] because there is that margin of error, and within that margin of error you cannot say anything.’ The increase from 16% to 19% hunger incidence does not mean that there is a significant difference because the margin of error is plus or minus 3%. Or rather, the differnce between them is not significantly different.”
More often than not, the margin of error and its implications are lost in the media’s haste to publish the latest results — the more dramatic the results, the better. “What the results mean is that probably nothing has happened, so there is no reason to panic,” says De Dios. “But the media can’t report somethings that is probably nothing. ‘Hunger reaches its peak’ is much more interesting than ‘Hunger reaches a statistically insignificant level.'”
“The media should be more responsible, ” Monsod says. “They should not just take the press releases on faith. There is no substitute for doing your homework.”
Through the Cracks

In the end, even the most fool-proof survey, whether using the subjective or fixed-standard approach, will not be able to capture the many changing complexities of human life. Moreover, no single survey will be enough to portray the state of the economy. In trying to find out how poor our country is, for instance,  a labor force survey might be conducted in order to figure out the unemployment rate, but Monsod points out that that sort of thinking is already based on a misconception. “The poor cannot afford to be unemployed. It’s a mistake to think that employment in this country automatically means salaried employment. The other kind of unemployment — having no job — is not an indicator of poverty.”
The same goes with access to electricity and water. “Access has increased but that does not mean that it is sifficient and that the distribution is equitable,” says Monsod. Another possible knothole is the fact that most surveys, both subjective and fixed-standard, are based on households and this presents a risk of under-representation of a sector. De Dios relates, for example, how hard it is for surveyors to penetrate the exclusive subdivisions of the rich, which results in Classes A,B and C being typically lumped together. Even the poor cannot all be represented since countless numbers live, not in houses, but in cardboard shanties beneath flyovers. And there are the Muslin Communities in Manila, where surveyors sometimes fear to tread. These are only few of the complexities that don’t quite find their way to the charts and tables.
Economic surveys are like pieces of a gigantic puzzle that have to be taken together and seen as a whole in order to produce a complete picture of the state of the economy of the country
So which is the better type of survey? De Dios answer is: “I say we need both.”