Report: How media tell survey results without proper detail

Submitted on December 18, 2014 as a report for a journalism course at HKU.

1. Introduction

In this report, the writer shall examine and evaluate on various examples of statistical reports by different media in real life.

 

2.1 Example of insufficient survey details presented in media I (survey & sampling)

  • Russian President Vladimir Putin selected as “Man of the Year”

In this section, the writer shall compare three reports run by RT News, Business Insider, and Mashable on Vladimir Putin selected as “Man of the Year” by the Russian public. Then, the writer shall examine the original polling results by Russia’s Public Opinion Foundation (FOM), and compare the original survey results and media reports.

 

2.1.1 How Russian state media reported the story

RT News – Russian public names Putin “Man of the Year” – fresh poll

RT news survey

 

Comments:

First of all, the title of this news report is “Russian public names Putin ‘Man of the Year’”. However, when we look at the first paragraph of the news report, we can see that Putin is given such an honour by the Russian public when pollsters asked them to choose from serving Russian politicians. Hence, this survey is actually conducted in the domain of Russian politicians. This fact is not reflected in the title. Thus, if a reader just skims through the title of the news report, he/she will tend to think that Putin was selected as the “Man of the Year” by the Russian public out of all Russians (or even out of everyone on Earth). Hence, it can be seen that the title is misleading as it tries to give an impression to readers that this survey is of much greater representation.

 

It can be noted from this news report that it failed to provide quite a lot of information regarding the survey. We are able to understand the following information solely by looking at this news report:

·       68% of Russians considered Putin worthy of the title of “Man of the Year”.

·       It is conducted by an organization called “Public Opinion Foundation (FOM)”

·       Pollsters asked the respondents to choose from among serving Russian politicians.

·       The runner-up was Vladimir Zhirinovsky, who got 4% of the votes.

 

However, a lot of details went missing. For example: –

·       What is the number of Russian actually surveyed?

·       Why was this research created? Is it a yearly survey?

·       When was this survey conducted?

·       How was this survey conducted (over the phone or street interviews)?

·       What is the margin of error and confidence level of this poll?

·       What was the exact wording of the survey? Are respondents given a list of candidates to choose from? If so, why are those candidates on the ballot?

·       What is the Public Opinion Foundation (FOM)? Where does this ‘independent’ organization receive money from?

 

In general, the reporter just jumped to the conclusion to say that Putin was selected as the “Man of the Year”, without giving the readers details about the raw data to let them judge the validity by themselves.

Let us now compare how other non-Russian media report this story.

 

2.1.2 Business Insider – Putin named Russia’s “Man of the Year” for the 15th Time in a Row

All information regarding the poll from this news story are included below:

Vladimir Putin has been named Russia’s “Man Of The Year” for the 15th time in a row, Interfax news agency reports.

The Russian president won by a landslide, claiming 68% of votes. The runner-up got only 4% of votes. The poll was conducted by the Public Opinion Foundation and included 1,500 respondents in 43 regions of Russia, according to Interfax.

The propaganda channel Russia Today said of the poll: “The public affirmation about Vladimir Putin’s major role in the life of the country looks even more decisive considering researchers within the same poll asked who among scientists and artists was worthy of the mantle. Some 75 percent of Russians said they had no answer to this question.”

Putin has won the “Man Of The Year” title every year since he rose to power in 1999, when he was appointed prime minister.

 

2.1.3 Mashable – No surprise here: Putin named Russia’s ‘Man of the Year’ for 15th year

All information regarding the poll from this news story are included below:

In a not-so-shocking turn of events, Russian President Vladimir Putin has been named “Man of the Year,” holding on tightly to his 15-year winning streak in a recent poll conducted by Russia’s Public Opinion Foundation.

According to Russian media outlet Interfax, the poll was conducted on Dec. 7. Some 1,500 residents of 43 regions of Russia were asked which Russian politican or public figure they would call man of the year.

Putin was at the front of the pack with 68% of the vote, more than doubling his popularity since the 2013 poll when he only secured 32% of the vote. His closest runner-up in the 2014 poll was the “hard to say, no answer” category, which pulled in a hefty 17% of the public vote.

The closest actual human runner-up was Vladimir Zhirinovsky, a Russian politician and fierce nationalist who pulled in a whopping 4% of the vote.

Other categories were less definitive. The poll asked respondents to pick a sportsman and an artist or culture icon who they felt was “Man of the Year” material. but 64% said they “couldn’t say” for the sportsman category; 75% of those who answered picked no one for the “cultural icon or artist” category.

 

Comments:

We can see easily that both reports run by Business Insider and Mashable provided much more detail than the RT News report.

Both reports cited reference to Russian media outlet Interfax. Besides the very brief information given by RT News, both reports also included the following information:

·       Vladimir Putin has won this title for the 15th consecutive year.

·       There were 1,500 respondents from 43 regions of Russia

·       Mashable: the poll was conducted on December 7th.

However, the title of both reports still failed to address the point that this poll was conducted under different domains, with Putin winning in the domain of Russian politicians. Moreover, both reports failed to give information about how these respondents are selected, what was the margin of error, the exact wording of the question etc. To understand this, we have to take a deeper look into the original report of the survey.

 

2.1.4 Actual survey result (retrieved from FOM website)

Note: the original website is in Russian. The following translation is by the limited Russian ability of the writer with assistance from Google Translate service.

Title: Man of the Year 2014 in culture, politics, sports

Related survey question: Whom among Russian politicians and public figures would you refer as man of the year?

(Polling options: open question)

Data source: “FOMnibus” – a survey of Russian citizens 18 years or older. 7 December 2014. 43 regions of the Russian Federation, 100 settlements, 1,500 respondents. Home interview. Stat error not exceeds 3.6%.

 

Comments:

From the original survey report, we are able to note that the poll was conducted in the form of an open question. Also, we can see that there are 1500 respondents who are Russian citizens of 18 years or older, from 43 regions of the Russian Federation. It was conducted in the form of a “home interview”, and was conducted on 7th December 2014, with statistical error not exceeding 3.6%.

It is noted that even from the original report, the surveyors did not provide certain crucial items. For example, the confidence level was not provided (as to how to calculate the statistical error of <= 3.6%). The method of selection of the samples was also missing. This piece of information is of huge significance as we have to avoid selection bias, and to ensure that all people in the population have an equal chance of being selected.

It can be noted that all 3 media (whether Russian or non-Russian) were unable to provide all information given in the survey report accurately. This limits the ability for the readers to determine the validity of the survey results themselves. To make matters worse, Mashable was the only media that included the link to the actual survey report by FOM. Readers of the other 2 news stories were “forced to” believe in the statistics, as they were not given further information to judge the validity by themselves.

 

If I were the reporter, I would report the news story in this way:

Russian President Vladimir Putin held a 15-year winning streak as the “Man of the Year” among Russian politicians, a recent poll by Russia’s independent Public Opinion Foundation (FOM) found.

The poll, which surveyed 1500 Russian citizens aged 18 or older in 43 regions of the Russian Federation, was conducted in the form of an open question and asked respondents to name one individual among all Russian politicians and public figures as the “Man of the Year”.

Vladimir Putin had an easy victory with 68% of the votes, according to FOM. The runner-up, Vladimir Zhirinovsky, founder and leader of the Liberal Democratic Party of Russia, received only 4% of the votes. Russian Foreign Minister Sergey Lavrov was the second runner-up with only 3% of the votes. 17% of the respondents found this question “hard to say”.

According to FOM, the survey was conducted on 7th December 2014 in the form of “home interview”. It also asked respondents to name “Man of the Year” from both the field of “science, culture and art” as well as “athletes”. Unlike President Putin, the winners of both categories did not have a significant margin of victory.

Putin has won the “Man of the Year” title every year since he became a major figure in Russian politics in 1999, when he was appointed as prime minister.

Russia’s Public Opinion Foundation (FOM) was originally established as a division of the Russian Public Opinion Research Center, a state-owned government institution under Russia’s Ministry of Labour and Social Affairs. It departed the center to become an “independent research agency” in 1992.

The statistical error of this survey does not exceed 3.6%, FOM said.

 

2.2 Example of insufficient survey details presented in media II (survey & sampling)

Polls relating to public opinion on the Hong Kong Police Force

After examining media coverage on foreign survey results, the writer shall now take a look at the media reports of poll ratings in Hong Kong. In this section, the writer shall compare different media reports on the release of popularity of Hong Kong disciplinary forces by the Public Opinion Program of the University of Hong Kong (HKUPOP). In particular, the writer shall look especially at the figure on the police force. Related news reports from the South China Morning Post and Apple Daily are also examined.

 

2.2.1 Original survey result – retrieved from HKUPOP

Note: In order to avoid confusion, the writer shall only quote the parts related to the rating of the Hong Kong Police Force.

 

Question: Please rate on a scale of 0-100 your satisfaction with the Hong Kong Police Force as a disciplinary force. 0 stands for very dissatisfied, 100 stands for very satisfied, 50 stands for half-half. How would you rate it?

Result:

Date of survey: 25-28/11/2014
Total Sample: 1012
Satisfaction rating: 61.0
Standard error: 1.2
Number of raters: 534

 

Question: Are you satisfied with the performance of the Hong Kong Police Force?

Result:

Date of survey: 25-28/11/2014
Total Sample: 1012
Very positive: 19.5%
Quite positive: 36.6%
Half-half: 15.2%
Quite negative: 14.7%
Very negative: 12.3%
Don’t know/Hard to say: 1.6%
Net value: 29.1%

 

2.2.2 Press release by HKUPOP (excerpt):

POP conducted a double stage survey on people’s satisfaction with the top 5 disciplinary forces in late November 2014 by means of random telephone surveys conducted by real interviewers. The survey shows that compared to five months ago, the popularity ratings and net satisfaction rates of most disciplinary forces have increased except for Hong Kong Police Force … that of Hong Kong Police Force registers positive 29 percentage points, which is record low since July 1997. The popularity drop of the Police is obviously due to the recent Occupy Movement, which has caught the Police in between different political forces. To overcome this problem, the Police will have to strengthen its professionalism in executing its duties, and also its affection and care for the society. It should not lean towards any political force, nor resort to improper means, just let political problems be resolved in political ways.

The maximum sampling error of all percentage figures is +/-4 percentage points, while that of rating figures is below +/-2.8 marks at 95% confidence level, and the sampling error of net values need another calculation. The response rate of the satisfaction survey is 67%.

 

Comments:

There are two types of data published by HKUPOP relating to public opinion on Hong Kong’s disciplinary forces. The first set of data is the popularity ratings, where respondents are asked to rate on the scale of 0-100. The second set of data concerns the net satisfaction rates, and is calculated by calculating the percentage of respondents expressing a positive view minus those expressing a negative view [i.e. (Very positive + quite positive) – (quite negative + very negative)].

HKUPOP has incorporated various elements regarding the survey into the report, including the entire questionnaire (and hence the exact question), the number of samples, confidence level, margin of error, survey period, method of selecting samples etc. The writer reckons that the survey report can basically address the following questions: –

 

·       Who created the statistics?

HKUPOP. In the “About POP” section, it can be noted that the program is now under the Faculty of Social Sciences in the University of Hong Kong. Hence, it is a publicly-funded academic institution that created the statistics.

·       Why were these statistics created?

In the “Background” section of the press release, it is mentioned that POP began regular surveys on people’s satisfaction with the performance of the Hong Kong Police Force and PLA Hong Kong Garrison. Hence, this survey result is part of an ongoing study.

·       How were these statistics created?

The methods of collecting data (telephone interview), definition of key terms (net satisfaction), and the sample (Hong Kong citizens aged 18 or above) are clearly mentioned in either the press release and/or the full questionnaire of the survey. Hence, the writer reckons that the survey method and the results are transparent and clear.

 

However, the writer expresses doubt when the researcher mentioned the red words in the press release above. The researcher claimed that the drop in popularity of the police is related to the recent Occupy Central movement, and that it suggested the police force to “not lean towards any political force” and “not resort to improper means” when dealing with Occupy Central. The writer reckons that such claims cannot be supported by what is obtained through the survey.

In fact, there is absolutely no mentioning of “Occupy Central” in the entire questionnaire. Respondents are not asked to express their views towards the movement, or comment on police’s handling on the issue. They are not asked follow-up questions on the reasons behind their ratings. Thus, there is no proof to confirm that the drop in rating of the Hong Kong Police Force must be related to their handling of the Occupy Central movement.

Also, the claim that police “not leaning towards any political force” and “not resorting to improper means” can overcome the problem of ratings drop is nothing more than assertion. Since respondents are not asked questions like “What do you think the police force can do to enhance its public image”, there is no way to understand the core problem related to the ratings drop. Such interpretation only reflects what the researcher believes, and cannot be supported statistically. Hence, the writer believes that such wordings should not be included into the press release if the researcher wishes to make the survey more scientific.

In the following part, the writer shall evaluate on different media reports of the story.

 

2.2.3 Media reports on the survey results

SCMP – Hong Kong police now less popular than China’s PLA, after Occupy clashes

The police are the least popular among the main disciplined services in the city, and rank lower than the People’s Liberation Army in terms of public satisfaction, according to a poll conducted by the University of Hong Kong.

The poll results, released yesterday, came amid criticism of officers using excessive force against Occupy protesters, but the force maintains this has had no impact on recruitment.

But the findings of yesterday’s poll seemed to suggest the reputation of the force had been tarnished because of its recent operations against protesters. The poll, conducted late last month, asked 545 people to rate their satisfaction with the major disciplined forces in Hong Kong, as well as the PLA Hong Kong garrison.

On a scale of zero to 100, the Fire Services Department came first with a score of 80.2. Second and third respectively, were the Immigration Department (73.3) and the Customs and Excise Department (73). The police came fifth with 61, behind the Civil Aid Service, which scored 71.6.

It was the lowest score for the police in the past two years. In the last poll in June, its score was 62.3. This itself was down on the figure of 63.7 last December. Meanwhile, the PLA scored 63.1 in the latest poll, up from 62.5 in June and 61.4 a year ago.

Dr Robert Chung Ting-yiu, director of the public opinion programme at the University of Hong Kong, said: “The popularity drop of the police is obviously due to the Occupy movement, which has seen the police caught between different political forces.

“To overcome this problem, the force will have to strengthen its professionalism in executing its duties, and also its affection and care for society. It should not lean towards any political force, nor resort to improper means.”

 

Apple Daily – 警隊民望創回歸後新低 (Popularity of Police Force drops to new low since handover)

All information relating to the police force rating are translated below: –

·       HKUPOP announces polling results of popularity of disciplinary forces

·       Hong Kong Police Force is ranked 5th, with a rating of only 61 marks, and net satisfaction of 29%

·       Dr. Chung Ting-yiu said police force will have to strengthen its professionalism in executing its duties, and also its affection and care for society. It should not lean towards any political force, nor resort to improper means, let political problems be solved politically. This can help them overcome the difficulties.”

·       Sources showed that net satisfaction of police force was 74.8% in July 1997, even at 80.5% in June 1007, but is only at 29% in November this year

·       HKUPOP interviewed more than 1000 people at the end of November, with a response rate of about 67%. The rating is ranged from 0 to 100, with 0 representing very dissatisfied, 100 representing very satisfied and 50 representing half-half.

·       Hong Kong Police Force was ranked the fifth, with a rating of 61 and satisfaction rate of only 29%, which is the lowest since the handover in July 1997.

 

Comments:

The writer reckons that both media failed to provide sufficient information on the HKUPOP survey for the readers to judge the reliability themselves.

On the first note, the SCMP only quoted the rating of the Hong Kong Police Force, and did not report on the net satisfaction rate as surveyed by HKUPOP. It mentioned that 545 people gave the rating, and the poll was conducted in late November. It also reported that the rating is the lowest since handover.

For Apple Daily, they mentioned both the rating and net satisfaction of the police force, and quoted that HKUPOP interviewed more than 1000 people at the end of November. They also provided the response rate and the rating nature.

However, both media organizations failed to further enquire the doubtful statement by HKUPOP in the press release. Both media organizations simply quoted what was said in the statement, but failed to follow-up on why HKUPOP said this statement. As noted in the previous section, such interpretation cannot be supported by what was found in the survey. Hence, the writer strongly believes media organizations should take the lead to question the validity of the statement made by HKUPOP.

Also, both failed to provide the margin of error and the sampling methods in their reports. There were even seemingly conflicting information presented in both reports. Apple Daily said HKUPOP interviewed more than 1000 people, while SCMP noted that 545 people were asked to give the rating. In fact, for the ratings part, there were only 545 “number of raters”. Hence, Apple Daily gave readers an impression that all 1000 gave a rating, which was statistically inaccurate.

In addition, even though Apple Daily quoted the net satisfaction rate of 29%, it failed to provide the definition of “net satisfaction rate” to its readers. Without such definition, the public can easily be misled into believing that only 29% of the entire Hong Kong population expressed support to the Hong Kong Police Force, which would be absolutely incorrect. Hence, the writer reckoned that they should have included the definition for readers to judge the number themselves.

Furthermore, both media failed to provide the link to readers to interpret the numbers in the original report themselves (especially in the online version of the news report). The writer reckons that it is absolutely necessary for readers to judge the validity of the survey beyond the report provided by the news agency.

 

To sum up, the writer would have the following suggestions to improve the news report:

·       Include the definition of net satisfaction rating if it is to quote the number

·       Ask for further comments from HKUPOP regarding the views expressed in the press release (And include “they said they would not comment further” if they chose not to respond. The writer would also note that this piece of information could not be supported from what was found in the survey.)

·       Include sampling method (telephone interview)

·       Include margin of error and confidence level (if space allows)

·       Mention clearly that only 545 respondents gave the rating, despite there were 1012 samples

·       Include link to the original survey report at the end of the news article

 

2.3 Example of causation claims (correlation and causation)

In this short section, the writer shall explore unsupported causation claims in the media. The example used is the claim that people who have frequent sex will make more money. A story from US Web Blog Gawker is illustrated below.

 

2.3.1 Gawker – More Buck For Your Bang: People Who Have More Sex Make The Most Money

A German study has found that people who have the most sex also make the most money, further depressing the world’s low-paid plebes who don’t get any.

Scientists in the adonis-laden European country found that people who have sex more than four times a week receive a 3.2 percent higher paycheck than those who have sex only once a week. God forbid you don’t have sex at all.

The study, done by the Institute for the Study of Labor in Germany (which usually issues reports finding only the most unimpeachable efficiency), also found that people who have more sex are happier and healthier and probably sexier.

“Wages are higher for those with extraversion and openness traits who are sexually active,” the study found after surveying 7,500 German households.

Essentially, wages are higher for those who are down to get down. A lot. And in weird ways. Germany.

 

2.3.2 Original research report – The Effect of Sexual Activity of Wages

Main points from the research report are summarized below:

  • The data were gathered from January 2008 through December 2008 in the Greek Behavioural Study (GBS). It consisted of random telephone-based surveys to approximately 7,500 households. Respondents in the GBS were asked how many times they engaged in sexual activity (SA)
  • Respondents were asked to choose among seven options: no sex (code 0); sex once or twice a year (code 1); sex once a month (code 2); sex two to three times a month (code 3); sex weekly (code 4); sex two to three times a week (code 5); and sex more than four times a week (code 6)
  • In this study, we hypothesized that because the medical and psychological literature suggest that sexual activity is associated with good health, endurance, mental well-being, mental capacities and dietary habits, it could be perceived as a health indicator, which might influence returns to labour market activity.
  • The rationale was that the economics literature suggests that physical and mental health, as well as personality characteristics, is important factors that affect wages.
  • Maslow’s Need Hierarchy Theory claims that the happier and more fulfilled individuals are in their lives, the more productive and successful they will be in their work, translating to higher wages. In the absence of these elements, many people become susceptible to loneliness, social anxiety, and depression that affect their working life.
  • This comparison suggests that the 2008 Greek Behavioural Study is, to a large extent, representative of individuals in Greece.

 

Comments:

First of all, the writer would like to state the main conditions for causality. In order to establish causal relationships between 2 variables, it is important to prove that there is a strong and consistent correlation between the two, explain the plausibility, and also eliminate all other possible causes that might explain the change in both variables. It should be noted that mere correlation does not necessarily imply causation.

It can be noted that the report in this web blog has largely neglected the explanation of these causation conditions when reporting on the causal relationship. It has largely jumped to conclusions and even has wrong factual information.

For instance, the report claimed that the study was established after surveying 7,500 German households. However, in the report, it can be seen that the German study actually employed the survey results obtained from the 2008 Greek Behavioral Study on individuals in Greece. Hence, the person who wrote the article has clearly not read through the report properly.

The report did not address on some of the key points mentioned in the study, such as the use of medical, psychological and economic literature in explaining the plausibility of why people with more sex may earn more, or the Maslow’s Hierarchy of Needs model. It only reported the catchy headline of “people who have the most sex make the most money,” which basically failed to give room for readers to judge the validity themselves.

The writer believes the study has tried to provide evidence and explained on the plausibility of how increased sexual activity lead to better health and enhance one’s salary. It has quoted medical, psychological and economic literature in establishing this proof.

However, the writer believes that it still cannot establish the causal relationship that increased sex lead to better wages. That is because the study report still cannot eliminate the possibility of a third variable that can explain the increase in sex and better wages. For example, a person with greater energy or with lower sleeping needs may be able to have more sex and earn more at the same time. Since the study failed to directly eliminate these possibilities, a causal claim cannot be established.

 

3. Conclusion

Statistics is a common tool for us to understand more about our society. However, one should understand that statistics could be misleading, and might distort public opinion on various social issues. Through the examination and revision of all 3 cases, it is not hard to see the inadequacy of reporting statistics relating to surveys and studies by some media. The writer believes that we have to develop the skills to critically evaluate numbers. When we interpret statistics, we should ask ourselves: who created the statistics? Why were the statistics created? How were these statistics created? This can help us differentiate valid and invalid statistics, and make ourselves stay smart when evaluating and interpreting numbers in different surveys and studies.

—–

Submitted for HKU course: Statistics for Journalists (JMSC1003).

Copyright © 2015 Eric Cheung. All rights reserved.
Please note that copying without proper acknowledgement (plagiarism) is a very serious offence in the academic world.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s