A common misconception is that web panels need to be representative based on the demographic distribution of the population. However, it is the survey results that must be representative.

What produces representative and thus reliable results is:

• Recruitment to the panel via probability sampling.
• Randomly stratified samples in each panel survey.
• Weighting on relevant variables.

This if we disregard other possible sources of error.

Randomly recruited panels VS. self-recruited panels

Setting up a web panel that allows valid statistical inferences about the public requires recruitment via probability sampling. This, in turn, requires a reliable sampling frame (the population register). Probability sampling means that everyone in the sampling frame has a known probability of being asked. Since it can only be done via phone, SMS, or mail, which entails costs, many panel managers choose to use self-recruitment instead.

If the panel recruitment is not random, an opinion bias will be built into the panel, this bias is dependent on which recruitment sources were used. Many self-recruited panels in CINT have an over-representation of, for example, parents of children who are members of various associations or newspaper readers from various newspapers. For example, Demoskop recruits at Aftonbladet under the heading “Do you want to be involved and influence politics”. In self-recruitment, each person has an unknown probability of being asked, making it impossible to construct reliable estimates.

Self-recruited panels often work with quota selection. However, using quota sampling to collect a perfect distribution of the number of interviews in different demographic groups does not compensate for the opinion bias found in a self-recruited panel. In self-recruited panels, statistically significant differences cannot be calculated partly because of the recruitment method, partly because quota sampling is used. In mathematics, minus times minus equals plus with this cannot be applied in this case.

In the Novus panel, we work with random samples where the data is then weighted to the right proportions, which allows us to use inference theory and make statements about statistically significant differences.

In a population with people aged 18 – 79, approx. 20% are aged 18 – 29. As you understood from above, it is e.g. invariably prefer a randomly recruited panel survey with 10% of respondents aged 18 – 29 compared to a self-recruited panel where 20% of respondents are aged 18 – 29.

Novus’ randomly recruited Sweden panel is therefore also used for data collection in many large research projects where reliability is important, including in research projects for Stanford.

Mikael Ohlsson,

Research manager at Novus