Update on conducting market research during the COVID-19 pandemic

We knew that in the first couple of weeks after the COVID-19 travel ban, survey response rates were on the rise — or at least flat — across most industries. This was not surprising, since iPhone screen time reports skyrocketed for many users, causing us to also question how much screen time is too much.

Now, in Week 6 of the quarantine, we need to know if people have continued responding to survey invitations at the same rate and if we have to consider a "coronavirus bias" when evaluating data quality or interpreting respondents' ratings.

Response rates are still up over pre-quarantine rates

We examined data across two of our longstanding trackers to find that response rates are still up over pre-quarantine rates.

We conduct multiple financial services tracking studies across various product lines. The number of completes across the studies has increased by about 10-20 percentage points and hit levels we have not experienced in the past two years.

We first reported 2 percentage point gains in our utility tracker. Since then, rates have plateaued, but remain 2 points higher than pre-quarantine. While people have expressed difficulties making other "at-home" activities part of their routine, they are continuing to respond to survey invitations.

This improvement in response rates enables us to conduct the research faster and more efficiently than we have in the past. Surveys that previously took two weeks to field are now being completed 10-20% faster, which means more data in a shorter time.

Smartphone usage is also up and, in some of our studies, mobile completes have increased by as many as 13 points. Increased mobile response taps into only- and mostly-mobile groups like never before. This extra downtime in the Age of Quarantining and social distancing suggests an unexpected outcome of this pandemic: Now may be a better time than ever to conduct research, especially in hard-to-reach populations generally thought not to have time to answer survey questions.

Quality or quantity: Why choose when you can have both?

Still, none of this matters if there is a negative impact on overall quality of the data. An ostensible "coronavirus bias" could artificially change response behaviors simply because we are experiencing something wildly atypical. While we can obtain more responses, they might be impacted by our unique situation and not grounded in normalized truth.

One way to examine this is to compare scores based on opinion (e.g., satisfaction, likelihood to recommend a product, etc.) before and after the quarantine. While easier said than done, with all things held equal, there should be no impact on satisfaction. Indeed, we see no change to these types of scores within our financial services tracker.

A different picture emerges for utilities. In some cases, such as transactional contact studies, satisfaction scores are higher, possibly because utility companies are not disconnecting services due to unpaid bills. In other cases, satisfaction is lower, perhaps because people working from home are more dependent on uninterrupted service from 9-5. In other words, pre-pandemic, a daytime service interruption was less impactful for residential customers.

We do not feel the data from the utility tracker is artificially biased. These changes are to be expected during a time when we are living differently than we have before. It doesn't mean that people are unnaturally rating their satisfaction higher or lower, but that changes to their behaviors are resulting in shifting opinions.

Based on our research, you can have both improved response rates and responses that will hold up once things have returned to some form of normal. However, we will advise caution with our clients and work with them to understand any potential changes to the data based on the impact of COVID-19.

It's not all about the Benjamins

Another consideration regarding the increased response rate is the incentive paid for participating. Response rates could be up simply because many have been sidelined from their jobs and are looking for a few extra bucks. While we think incentives could increase response rates even higher, the trackers we use to report this data do not offer a monetary incentive. Thus, the 2- to 3-point response increases we see are due to other factors, not the need for some quick cash.

I initially expected to create this post alerting readers about conducting research during this time. Instead, we have been able to complete research faster and just as effectively as we did before the nationwide lockdown. Surprisingly, more time on our hands and more time at home has led to an increased response to most of our research. It also does not appear to be temporally biased by COVID-19 impacts. 

Of course, research during this time should still be sensitive and mindful of the current situation, but now is a good time to consider getting feedback from customers that could inform how to better move forward on the other side of this pandemic. If you have questions about conducting research during the COVID-19 pandemic, we can help.