With the emergence and general acceptance of online samples for market, opinion, and social research comes a responsibility to measure and ensure the quality of research results using such samples. A number of concerns about sample quality have been raised across the industry. They include concerns about:
- Professional research participants who through various means try to maximize their survey opportunities
- Inattentive if not deliberately untruthful research participants
- The potential for duplicate research participants as online sample providers broaden their sources in search of more diverse samples and low incidence populations
- Representativeness, that is, the degree to which research results accurately reflect the target population, however defined.
This Guideline responds to those concerns by providing guidance on the operational requirements for the provision of online samples for market, opinion, and social research. It sets out methods to be used by online sample providers, buyers, and end clients to ensure that a sample meets widely accepted quality criteria. It is recommended reading for all stakeholders in the research process, from survey designers to data users.
It is meant to apply to all types of online samples, including those recruited from panels, from social media, and by web intercept methods. It is not meant to cover client-supplied samples such as lists of customers. Nor is it meant to cover samples recruited for qualitative research such as focus groups or one-on-one interviews since direct interaction between the researcher and research participant provides opportunities for quality assurance that is generally lacking in self-administered quantitative research. Nonetheless, researchers may find some of its suggested practices useful when working with those sample sources.
This Guideline is not intended to substitute for a thorough reading and understanding of the ICC/ESOMAR International Code on Market and Social Research, which has been adopted by over 60 local associations worldwide, or the individual codes of the 38 associations that comprise the GRBN. Rather, it is intended to be an interpretation of the foundational principles of those codes in the context of online research. It also draws on a number of sources for its basic principles including ISO 20252 – Market, opinion, and social research and ISO 26362 – Access panels in market, opinion, and social research. It recommends full transparency with clients and encourages the use of a common set of terms and definitions. It offers a set of suggested practices for online sample quality, although it is not meant as a substitute for either ISO 20252 or ISO 26362, both of which require an external audit of a sample provider to ensure compliance. Many buyers of online samples may wish to limit their sample purchases to ISO-certified providers because of the added assurance that an external audit provides.
This Guideline also is not intended to be inclusive of all factors that might impact online research quality. Research on research in this area continues to evolve and there are many unsettled issues. However, there is a limited set of issues that have come to the fore despite some on-going controversy about their impact on data quality. They are:
- Research participant validation
- Survey fraud prevention
- Survey engagement
- Category and other types of exclusions
- Sampling (including sample selection, sample blending, weighting, survey routers, profiling and screening).
Throughout this document the word “must” is used to identify mandatory requirements. We use the word “must” when describing a principle or practice that researchers are obliged to follow in order to comply with existing codes and standards such as those cited above. The word “should” is used when describing implementation and its usage is meant to recognize that researchers may choose to implement a principle or practice in different ways depending on the design of their research.
Read the full guideline: ESOMAR GRBN ONLINE SAMPLE QUALITY GUIDELINE