The importance of having control over the recruitment of panel members

By allowing self-recruitment to your panel, you open yourself up to problems. Many of the problems that exist in data acquisition in surveys are rooted in the fact that you do not have control over your recruitment. The only way to really control who or what comes in is to use a random public selection where you yourself control who is invited to be on the panel.

Our advisor, Peter Rosin, who has extensive experience in the industry, writes about the quality problems he has seen within self-recruited panels.

In recent years, it has become increasingly difficult to recruit panelists. To meet this, the panel suppliers take new measures to create reach in their panels, they have opened up to self-recruitment and partnerships that do not always have the same quality and safety mindset. This has created new opportunities for fraudulent behavior.

Larger organized operations have become more common and the methods increasingly sophisticated. Globally, this is a much bigger problem than it is in the Nordics. The main reason is that we in the Nordics are small markets where a project rarely has more than 1000 respondents. Fraudulent answers risk standing out more and there is no big money to be made from organized fraud in the same way. In addition, we do not pay out cash as compensation for answering surveys in Sweden, which is common in most markets globally. The biggest threat to the industry in Sweden is probably influence operations, where you combine “fake news” that you try to plant in traditional media and large campaigns in social media, combined with fraudulent answers to surveys with the aim of trying to influence public opinion, politicians and the public.

The reason why respondents answer a survey in a fraudulent way varies widely, but they can be divided into 2 categories:

• Cheaters who do it for their own gain
• Fraudsters who do it on a large scale with the aim of influencing, harming or making money

As in all contexts where there is money to be made or the possibility of influence, there is a battle between “The good and the bad guys”. Modern technology & AI has brought new sophisticated opportunities to cheat. Fictitious respondents, bots are created using AI and answer in a way that mimics what a real person might have answered. One deceives the panel companies’ security system by manipulating both the IP address, location and response pattern.

The major global panel companies are doing what they can to meet the threats. More and more sophisticated security systems are developed that measure everything from response time, speedsters, straight-liners, patterns, etc. The new systems can measure the time it takes to read a question on the mobile until you click on the answer. For open-ended responses, intervals are measured in keystrokes – some letter combinations take a little longer than others and these intervals can be measured to determine if it is really a human hand typing. You can measure logic errors in real time and the IP address can be checked using the geolocation of your mobile phone. Added to them are solutions such as digital identification to be able to get your compensation, as well as checks against official registers to determine if the person really exists, etc.

With more and more security systems and solutions in place, bad guy solutions are also becoming more sophisticated, not only technologically but also practically. In recent years, several large so-called “Click-farms” exposed. Several of these have been located in Myanmar, where the IT industry invested in large call centers in the late 90s and early 00s. These quickly turned out not to deliver, but the premises and infrastructure have been given a different function, i.a. like click farms. A Click-farm is a room in which they sit 100s, sometimes 1000s of people who sit for little compensation and e.g. answer surveys according to a script they have been given. A similar method to Russia’s so-called troll armies use. When it is people who answer the surveys, you get past many of the touch-points of modern security systems that determine whether it is cheating or not. Via rigged IP addresses, it looks as if the people are really sitting in the country/place that they are supposed to do according to the survey they answer.

All these solutions and systems to prevent cheating are needed for one primary reason. You do not have control over your recruitment of panelists. A major advantage of a recruitment based on a random public sample is that fraudulent individuals, bots and others with fraudulent motives do not enter the panel. Of course, individual respondents to individual surveys may answer in a non-serious manner out of sheer nonchalance, laziness or lack of interest. But we find these easily with the checks we do today.

Novus panel is recruited based on a random sample of the public, it is your best guarantee that no fraudulent or inaccurate people will be able to answer your survey and that the data the results are based on is valid and holds a high

Peter Rosin

Senior Client Advisor at Novus