ESOMAR’s 37 questions to help buyers of online samples

Novus is a member ESOMAR, the global membership association for market research and insights.

ESOMAR has developed 37 questions to support buyers of online research, with the aim of facilitating the evaluation of suppliers’ methodologies, quality, and transparency. These 37 questions replace ESOMAR’s previous version of 28 questions.

Below are Novus’ answers to these questions:

Company Profile

1. What experience does your company have in providing online samples for market research? How long have you been providing this service? Do you also provide similar services for other uses such as direct marketing?

Novus was established in 2006 and is recognized as Sweden’s leading research company, operating as a part of Gallup Nordic and Gallup International. We specialize in providing analyses related to branding, business and operational development, societal issues, and public opinion. Our goal is to deliver insights into what the target audience thinks, the reasons behind their opinions, and actionable recommendations for our clients.

From the outset, we have utilized the Gallup method to build and maintain our proprietary web panel, “Novus Sverigepanel” (Novus Sweden Panel), which is based on unbound random population samples. Our team actively manages processes, sample handling, panel maintenance, and addresses the so-called “panel effect.”

Currently, Novus Sverigepanel comprises approximately 60,000 randomly recruited panel members from across Sweden, all of whom, in accordance with the guidelines set by the industry organization Esomar, have participated in at least one survey in the past year. Our surveys can be conducted swiftly, cost-effectively, and with a high level of quality, ensuring that the results are nationally representative and reliable. Given that the panel is a dynamic resource, we continuously recruit new members to maintain its freshness.

Surveys conducted through Novus Sverigepanel are the most frequently cited web panel surveys in Swedish media. It is important to note that Novus does not offer similar services for any purposes other than research; specifically, we do not provide services for direct marketing.

 

2. Do you have staff with responsibility for developing and monitoring the performance of the sampling algorithms and related automated functions who also have knowledge and experience in this area? What sort of training in sampling techniques do you provide to your frontline staff

Novus exclusively employs statistical methods and does not use any algorithms or associated automated functions for sample selection. Instead, Novus stratifies its samples based on variables relevant to each project.

 

3. What other services do you offer? Do you cover sample-only, or do you offer a broad range of data collection and analysis services?

Novus does not provide sample-only services, as we prioritize maintaining full control over the quality of data collection across all surveys. We utilize a range of data collection methods, including email, telephone, SMS, and traditional mail. Novus delivers processed data in the form of reports that include comprehensive analysis and insights.

 

Sample sources and Recruitment

4. Using the broad classifications, from what sources of online sample do you derive participants?

Novus utilizes the Novus Sweden Panel for conducting surveys within Sweden. For surveys in smaller geographical areas, we implement the Novus SMS selection method. When conducting surveys outside of Sweden, we use Gallup Nordic or partner with Gallup International or other affiliated organizations.

 

5. Which of these sources are proprietary or exclusive and what is the percent share of each in the total sample provided to a buyer? (Assume proprietary to mean that the sample provider owns the asset. Assume exclusive to mean that the sample provider has an exclusive agreement to manage/provide access to sample originally collected by another entity.)

The Novus Sweden Panel is exclusively utilized by Novus, meaning that external parties cannot purchase sample-only access.

 

6. What recruitment channels are you using for each of the sources you have described? Is the recruitment process ‘open to all’ or by invitation only? Are you using probabilistic methods? Are you using affiliate networks and referral programs and in what proportions? How does your use of these channels vary by geography?

Novus does not recruit from multiple sources; instead, we conduct recruitment regularly through independent random population sampling. Panel members are recruited using a range of relevant methods, including telephone interviews, SMS invitations, and postal mailings.

 

7. What form of validation do you use in recruitment to ensure that participants are real, unique, and are who they say they are? Describe this both in terms of the practical steps you take within your own organization and the technologies you are using.

Novus utilizes an independent random recruitment method among the Swedish population, which prohibits self-recruitment. This approach ensures that all recruitment initiatives originate from Novus, and individuals cannot sign up or “apply” to join the online panel. We also verify that recruits are not already members of the panel and that their background information aligns with the data available in our recruitment sample.

The rationale behind not permitting self-recruitment is to maintain full control over the selection process, thereby preventing the overrepresentation of professional opinion leaders, extremists, or others who may seek to impose their views. Furthermore, a randomly recruited population sample is essential for accurately reporting statistical margins of error.

 

8. What brand (domain) and/or app are you using with proprietary sources? Summarise, by source, the proportion of sample accessing surveys by mobile app, email or other specified means.

The Novus Sweden Panel features a dedicated website that provides information regarding the panel’s purpose and how Novus manages personal data in accordance with GDPR regulations. Members can also update their email addresses and passwords, as well as request rewards. Invitations to participate in surveys are sent to Novus panel members via email.

 

9. Which model(s) do you offer to deliver sample? Managed service, self-serve, or API integration?

Novus does not offer sample selection services exclusively (Not applicable).

 

10. If offering intercepts, or providing access to more than one source, what level of transparency do you offer over the composition of your sample (sample sources, sample providers included in the blend). Do you let buyers control which sources of sample to include in their projects, and if so how? Do you have any integration mechanisms with third-party sources offered?

This is not a service offered by Novus (Not applicable).

 

11. Of the sample sources you have available, how would you describe the suitability of each for different research applications? For example, is there sample suitable for product testing or other recruit/recall situations where the buyer may need to go back again to the same sample? Is the sample suitable for shorter or longer questionnaires? For mobile-only or desktoponly questionnaires? Is it suitable to recruit for communities? For online focus groups?

Novus exclusively offers the Novus Sweden Panel for all types of panel surveys, including both quantitative and qualitative research. Our panel surveys can be conducted across various devices. While Novus is equipped to carry out both short and long surveys, we are mindful of limiting the number of lengthy surveys sent to our panel members.

 

Sampling and Project Management

12. Briefly describe your overall process from invitation to survey completion. What steps do you take to achieve a sample that “looks like” the target population? What demographic quota controls, if any, do you recommend?

Novus employs an independent random recruitment method among the Swedish population, meaning that self-recruitment is not utilized for the Novus Sweden Panel. Prior to each survey, a stratified sample is drawn from the web panel based on known variables relevant to the target group. During the data processing stage, results are weighted according to established variables such as gender, age, and region.

In all sampling surveys, results should be regarded as estimates. The reliability of the results is influenced by the formulation of questions and the characteristics of the sample. Consequently, Novus places significant importance on these activities. Overall, findings from the Novus Sweden Panel yield nationally representative results.

 

13. What profiling information do you hold on at least 80% of your panel members plus any intercepts known to you through prior contact? How does this differ by the sources you offer? How often is each of those data points updated? Can you supply these data points as appends to the data set? Do you collect this profiling information directly or is it supplied by a third party?

Novus exclusively utilizes Novus Sverigepanel for our research needs. We update the panel’s background data at least once a year, with the flexibility to incorporate new data as needed. Approximately 90-95% of panel members update their information annually. All available background data can be provided to clients upon request and included in proposals as necessary.

 

14. What information do you need about a project in order to provide an estimate of feasibility? What, if anything, do you do to give upper or lower boundaries around these estimates?

The information we require includes the purpose, the survey topic, the number of interviews, the number of questions/interview duration, and the estimated penetration rate. In instances where penetration data is not available from official statistics or within Novus’s structural assets, we can efficiently generate an estimate by surveying a subset of Novus Sweden Panel, thereby providing an approximation of the penetration rate.

 

15. What do you do if the project proves impossible for you to complete in field? Do you inform the sample buyer as to who you would use to complete the project? In such circumstances, how do you maintain and certify third party sources/sub-contractors?

Novus maintains strict control over our data collection capabilities. In the unlikely event that Novus Sverigepanel is unable to deliver the promised number of interviews, we will first supplement our efforts using our data collection method, Novus SMS. Ultimately, we will consult with the client to determine whether additional interviews are necessary or if the number of completed interviews is sufficient to meet the survey’s objectives. If a subcontractor is required, we will inform the client of the advantages and disadvantages of the chosen method, whether it involves telephone interviews or web interviews.

 

16. Do you employ a survey router or any yield management techniques? If yes, please describe how you go about allocating participants to surveys. How are potential participants asked to participate in a study? Please specify how this is done for each of the sources you offer.

This is not relevant and not applicable to Novus.

 

17. Do you set limits on the amount of time a participant can be in the router before they qualify for a survey?

This is not relevant and not applicable to Novus.

 

18. What information about a project is given to potential participants before they choose whether to take the survey or not? How does this differ by the sources you offer?

The invitation includes information regarding the estimated time required to complete the survey, any potential rewards, and the response deadline. In exceptional cases, such as research projects, the invitation may also provide details about the subject matter of the survey.

 

19. Do you allow participants to choose a survey from a selection of available surveys? If so, what are they told about each survey that helps them to make that choice?

Novus invites participants to our surveys exclusively through random selection. As a result, it is not possible to choose from multiple surveys to participate in; however, responding to the surveys to which one is invited is always voluntary.

 

20. What ability do you have to increase (or decrease) incentives being offered to potential participants (or sub-groups of participants) during the course of a survey? If so, can this be flagged at the participant level in the dataset?

Should it be necessary to adjust the reward during an ongoing project, this can be accommodated, and we can accurately identify which participants have received specific rewards.

 

21. Do you measure participant satisfaction at the individual project level? If so, can you provide normative data for similar projects (by length, by type, by subject, by target group)?

We regularly assess respondent satisfaction by asking participants to evaluate various aspects of the survey, including content, question design, survey length, the attention given to questions and answers, and any additional comments they may have. The collected feedback is classified according to target audience, survey length, and type of survey.

 

22. Do you provide a debrief report about a project after it has completed? If yes, can you provide an example?

We consistently provide information regarding the target audience, the number of interviews conducted, and the field period. Upon request, we can also include additional details such as participant frequency, the percentage of respondents screened out, the percentage who do not complete the survey, and/or dropout rates by target group.

 

Data quality and Validation

23. How often can the same individual participate in a survey? How does this vary across your sample sources? What is the mean and maximum amount of time a person may have already been taking surveys before they entered this survey? How do you manage this?

On average, Novus panel members participate in over 10 surveys annually. After completing a survey, members are placed in a quarantine period, the duration of which depends on the type of survey and whether the member may have acquired information that could introduce bias in future surveys. The length of the quarantine period is therefore determined on a case-by-case basis.

 

24. What data do you maintain on individual participants such as recent participation history, date(s) of entry, source/channel, etc? Are you able to supply buyers with a project analysis of such individual level data? Are you able to append such data points to your participant records?

We maintain a record of when panel members joined, along with detailed information about all the surveys they have been invited to, completed, were screened out of, or did not finish, including the dates for each of these events.

 

25. Please describe your procedures for confirmation of participant identity at the project level. Please describe these procedures as they are implemented at the point of entry to a survey or router.

Novus recruits panel members using unbound random population sampling, ensuring reliability when inviting participants to surveys. We utilize unique survey links and, when necessary, include verification questions such as gender, age, and postal code to confirm participant information.

 

26. How do you manage source consistency and blend at the project level? With regard to trackers, how do you ensure that the nature and composition of sample sources remain the same over time? Do you have reports on blends and sources that can be provided to buyers? Can source be appended to the participant data records?

Novus exclusively uses Novus Sverigepanel for this purpose, ensuring that it is not an issue.

 

27. Please describe your participant/member quality tracking, along with any health metrics you maintain on members/participants, and how those metrics are used to invite, track, quarantine, and block people from entering the platform, router, or a survey. What processes do you have in place to compare profiled and known data to in-survey responses?

Novus exclusively utilizes Novus Sverigepanel for this purpose, making it a non-issue.

 

28. For work where you program, host, and deliver the survey data, what processes do you have in place to reduce or eliminate undesired in-survey behaviours, such as (a) random responding, (b) Illogical or inconsistent responding, (c) overuse of item nonresponse (e.g., “Don’t Know”) (d) inaccurate or inconsistent responding, € incomplete responding, or (f) too rapid survey completion?

Novus conducts thorough checks on response times, the proportion of “don’t know” answers, and performs logical and plausibility assessments of incoming responses. However, these considerations are not problematic when working with randomly recruited panels based on population sampling.

 

Policies And Compliance

29. Please provide the link to your participant privacy notice (sometimes referred to as a privacy policy) as well as a summary of the key concepts it addresses. (Note: If your company uses different privacy notices for different products or services, please provide an example relevant to the products or services covered in your response to this question).

This information is available to panel members on the panel site at https://panel.novus.se/.

 

30. How do you comply with key data protection laws and regulations that apply in the various jurisdictions in which you operate? How do you address requirements regarding consent or other legal bases for the processing personal data? How do you address requirements for data breach response, cross-border transfer, and data retention? Have you appointed a data protection officer?

Novus has a designated Data Protection Officer and complies with GDPR legislation.

 

31. How can participants provide, manage and revise consent for the processing of their personal data? What support channels do you provide for participants? In your response, please address the sample sources you wholly own, as well as those owned by other parties to whom you provide access.

Novus offers dedicated panel support to members of Novus Sverigepanel, primarily through email, with the option for phone support also available.

 

32. How do you track and comply with other applicable laws and regulations, such as those that might impact the incentives paid to participants?

Novus complies with Swedish legislation.

 

33. What is your approach to collecting and processing the personal data of children and young people? Do you adhere to standards and guidelines provided by ESOMAR or GRBN member associations? How do you comply with applicable data protection laws and regulations?

Novus adheres to the guidelines set forth by Esomar. We do not retain any personal data from individuals under the age of 16.

 

34. Do you implement “data protection by design” (sometimes referred to as “privacy by design”) in your systems and processes? If so, please describe how.

We limit access to data to a few in our organization that need it. We delete personal data that is not needed.

Novus utilizes pseudonymization and encryption to safeguard data security. Personal data and response data are stored in separate databases to ensure a robust and secure data management system.

 

35. What are the key elements of your information security compliance program? Please specify the framework(s) or auditing procedure(s) you comply with or certify to. Does your program include an asset-based risk assessment and internal audit process?

Novus has a business continuity plan and a group with defined responsibilities for risk assessments of all assets including internal audit and training for everyone.

Novus has its own developed quality system.

 

36. Do you certify to or comply with a quality framework such as ISO 20252?

Novus is not ISO certified but has its own equivalent quality management system.

 

Metrics

37. Which of the following are you able to provide to buyers, in aggregate and by country and source?

Novus has the ability to report the points below that are relevant and applicable to the Novus Sweden Panel if the client requests this.

1. Average qualifying or completion rate, trended by month
2. Percent of paid completes rejected per month/project, trended by month
3. Percent of members/accounts removed/quarantined, trended by month
4. Percent of paid completes from 0-3 months tenure, trended by month
5. Percent of paid completes from smartphones, trended by month
6. Percent of paid completes from owned/branded member relationships versus intercept participants, trended by month
7. Average number of dispositions (survey attempts, screenouts, and completes) per member, trended by month (potentially by cohort)
8. Average number of paid completes per member, trended by month (potentially by cohort)
9. Active unique participants in the last 30 days
10. Active unique 18-24 male participants in the last 30 days
11. Maximum feasibility in a specific country with nat rep quotas, seven days in field, 100% incidence, 10-minute interview
12. Percent of quotas that reached full quota at time of delivery, trended by month