Problems with Online Panel Surveys Confronted by Research Associations

Problems with Online Panel Surveys Confronted by Research Associations

 

Atlanta, GA  April 2, 2015 – The Council of American Survey Research Organizations (CASRO) and The Global Research Business Network (GRBN), in conjunction with the research firm Esomar, have recently issued guidelines to help researchers and marketers to overcome some of the vulnerabilities of online panel research and to enhance credibility for such research.

In the announcement of the new guidelines, the organizations said the following:

“Most online samples are derived from panels of people who have agreed to answer questionnaires online, usually in exchange for an incentive. Market, opinion and social researchers use the results to provide a fast reflection of consumer views and opinions.

However, the rapid growth of online research has been accompanied by concerns about the integrity of the resulting research data. These are due to the number of professional research participants who enter multiple surveys to secure incentives or money, inattentive or untruthful respondents, unrepresentative target groups, and the potential for duplicated respondents as research providers broaden their sources to expand sample sizes.

The proliferation of mobile devices is also posing challenges as well as opportunities for online researchers, most notably the potential to create bias. Between 20 and 30 percent of research respondents now respond using a smart phone or mobile device – they are typically younger, male, and more ethnically diverse than in conventional sampling. The limitations of mobile devices can also skew completion rates. The guidelines underline the need for transparency when reporting data in this area.  

Another area of concern relates to how online sampling has evolved over time, from online panels to routers to exchanges. Newer techniques such as ‘river sampling’ – recruiting samples from real time adverts or offers – can make it harder to validate respondent identities and their relevance to the target population.

In addition, the trend towards recruiting respondents from schemes such as ‘frequent flyer’ or particular websites opens up the potential for duplicate participants. Blending together samples from several panel sources to create greater balance can also lead to individuals answering surveys more than once.”

Finn Raben, Director General of ESOMAR, said: “The rapid growth in online research presents real challenges for brands and marketers to assess the quality of their samples. This Guideline is designed to help data users to understand the issues affecting online sample quality and so they can make judgments with full confidence. It underlines the need for a transparent sampling process, knowledge of how the sample was selected, and specifically to ensure that the same individual only answers a particular survey once.”

The American Affluence Research Center, which does not use online panel research and focuses exclusively on mail surveys, has often cautioned marketers about the accuracy and reliability of research among online panels and whether panel members are truly representative of the affluent and luxury consumers.

Ron Kurtz, President of American Affluence Research Center, has recommended that marketers consider three reports that demonstrate the problems associated with online panels: Sample Quality of Online Panels: Putting Lipstick on the Piggy Bank,  Dirty Little Secrets of Online Panels, and  More Dirty Little Secrets of Online Panel Research. The latter two were the subject of an article that appeared in the American Marketing Association’s Greenbook Market Research Newsletter.

###

About CASRO and GRBN:

GRBN connects 38 research associations and over 3500 research businesses on five continents. More than US$24.9 billion (€18.8 billion) in annual research revenues (turnover) are represented by these businesses.

CASRO is the national association established in 1975 to represent the U.S. research industry and those organizations engaged in the conduct, support, or education of market, opinion, and social research, often described as data analytics, consumer insights, or business intelligence. CASRO’s mission is to provide the environment and leadership that will advance the integrity, quality, and best interests of research businesses, as well as the U.S. and global research industry.

CASRO’s 325+ member companies represent nearly $8 billion in global annual revenue–about 85% of the US research industry and 30% of the global research industry. All CASRO members must adhere to the CASRO Code of Standards and Ethics, the enforceable standard for research businesses for more than 30 years.

 

Contact:

Ron Kurtz
(770) 740-2200
rk@affluenceresearch.org

 

About Us: Established in 2001 and with an exclusive focus on the affluent market, The American Affluence Research Center conducts the original and only continuous twice-yearly tracking studies of the mood and future spending plans of the wealthiest 10% of U.S. households based on net worth. AARC has become a recognized authority and a credible source of reliable insight and marketing information about the values, lifestyles, attitudes, and purchasing behavior of America’s most affluent consumers.