Research-on-Research

This page contains all of the RoR (Research-on-Research) included in the book and additional material. Additional in the sense of more material for the quoted studies, and more studies.

Title Summary
The Threat of Satisficing in Surveys: The Shortcuts Respondents Take in Answering Questions
Krosnick, J
2000
Tags: Krosnick, Satisficing
Krosnick outlines the phenomenon and problems associated with satisficing, respondents doing enough to complete the survey, for example by engaging in short-cuts such as straight lining.
Link
Does Adding One More Question Impact Survey Completion Rate?
Chudoba, B
2010
Tags: Completion Rate, Survey Length
Meta-analysis of 100,000 online surveys conducted on the SurveyMonkey platform 2009 and 2010, looking at 2000 single question surveys, 2000 two question surveys, 2000 three question surveys, and so on up to 2000 50 question surveys.
This study confirmed that longer surveys lead to more dropouts. The paper highlights that from 1 to 15 questions the growth in dropouts is relatively steep. From 16 to 35 questions the increase in the dropout rate slows, and above 30 questions there is relatively little additional increase in dropouts beyond 35 questions.
Link
How Much Time are Respondents Willing to Spend on Your Survey?
Chudoba
2011
Tags: Time
A meta-analysis of 100,000 online surveys conducted on the SurveyMonkey platform in 2011. The key finding was that respondents spend longer per question when answering shorter surveys, and less time per question with longer surveys. This finding supports the suggestion that research participants can sometimes engage in satisficing, i.e. just doing enough to be able to complete the survey (Krosnick 2000).
Link
Differences between mobile and PC responses
Vision Critical
2014
Tags: PC Mobile Comparison, Time
This study was conducted in 2011 with a sample of 2025 participants, spread across the USA, UK, and Canada. The three key findings were that most survey question types give similar answers on PCs and smartphones, that multi-select grids tend to give different answers on PC and smartphone, and that smartphone versions of PC surveys can take 50% longer.
Link
Impact of survey length on responses
Vision Critical
2014
Tags: PC Mobile Comparison, Time

This study was conducted in 2011 with a sample of 2000 participants spread across the USA, UK, and Canada. Participants were allocated to a PC or a smartphone cell and three survey lengths were tested. The longest survey length tested had a mean completion time of 9.5 minutes.

The two key findings were: (a) forcing participants to respond via mobile reduced the response rate, and (b) the completion rate for a 9.5 minute survey was the same as for shorter mobile surveys.

Link

Australian Mobile Test 2012
Ipsos ASI
2014
Tags: PC Mobile Comparison, Ad testing
The study was conducted in 2012 to evaluate mobile ad testing in conjunction with MobileMeasure, using three TV ads with a sample size of approximately 120 participants seeing each ad. The test used a subset of the normal Ipsos ASI test. For those items included in the mobile test the results were comparable.
Harnessing Mobile Technology to Draw Insights from Health Care Professionals
Kantar Health
2014
Tags: PC Mobile Comparison, Heatlhcare
(p>The study looked at using a mobile survey with 600 US doctors, comparing the results with a reference study conducted as a conventional online study. The survey was re-designed for mobile and reduced in length from 45 minutes to 15 minutes.

The three key findings included: (a) the results from the mobile study were comparable with the online study, (b) some doctors expressed reluctance at being allocated to the mobile cell, and (c) the surveys on smartphones took longer than the same surveys on iPads.

I would … but the data would be different
Courtright, M
2013
Tags: PC Mobile

The presentation was based on 30,000 interviews conducted by Research Now in 2012. The analysis covered multiple countries and compared results from participants using PC and mobile. The study showed very comparable responses in mean responses in attitudes towards sports and in the proportions of participants choosing to follow the Olympics via different media.

Research Goes Mobile: Findings from Initial Smartphone Application Research
Dubreuil and Joubert
2012
Tags: PC Mobile Comparison, Ad Testing

This paper reported two separate advertising comparison tests. The first was in 2011 and used a sample of 400 people to evaluate a print ad, the second test was in 2012 and evaluated two TV ads. Both waves of testing were based on Luma’s ad+impact test.

The two key findings were: (a) the results for the TV ad were comparable between PC and mobile, and (b) the print ad results were generally comparable, but the ‘stand out’ score for the print ad was lower for the mobile cell.

The who, when, where, and how of Smartphone research
Fine and Menictas
2012
Tags: PC Mobile Comparison, Response rates, Survey length, Time

The study comprised 1514 participants in 2011 and a further 640 in 2012, in Australia. The three key findings were: (a) the mobile responses had a different demographic profile, (b) after controlling for demographic differences the survey responses were similar, and (c) the response rates and results for 5, 10, and 15 minute long surveys were similar.

Touchscreen devices versus non-touchscreen devices
Vision Critical
2014
Tags: Touchscreens, Response rates, Time
The study investigated 902 participants with touchscreen phones and 1115 non-touchscreen (BlackBerry) users. The three key findings were: (a) the response rate amongst non-touchscreen users was lower, (b) the non-touchscreen participants were less happy with the process, and (c) the non-touchscreen users took about 40% longer than the touchscreen users.
Using Dual-Frame Telephone Surveys To Include The ‘Mobile Phone-Only’ Population
Pennay at al
2013
Tags: CATI, Dual Frame

This study was conducted as a CATI (telephone) study in Australia in 2012, with a sample of 2000 people. The key finding, for mobile market research, was that the profile and responses of people with both a mobile phone and landline differed depending on whether the interviews were completed on a landline or a mobile phone.

Can Mobile Web Surveys Be Taken on Computers? A Discussion on a Multi-Device Survey Design
de Bruijne and Wijnant
2013
Tags: PC Mobile Comparison, Time

This study was a two cell study with a sample size of 2722 and was conducted in The Netherlands. Both cells answered via mix of devices (about 15% using a mobile device). One cell saw a PC style survey, the other saw the survey configured for mobile devices.

The key findings were that the results were similar, but the mobile version took slightly longer and was slightly less popular.

GRIT Consumer Participation in Research Report
GreenBook
2013
Tags: Global Stats

This study reported the findings from using the RIWI method globally in 2013, utilising data from about 1.6 million devices. About 19% of the survey invitations were served to mobile devices, with iOS as the most commonly used operating system.

Understanding Respondent Scale Usage Across Border and Devices
Jue and Luck
2014
Tags: Global Stats, Completion rates

This report was based on the surveys delivered by the Decipher platform in 2013, with the numbers of survey starts varying from 1.5 to 5 million per month. The key findings included:

  1. The number of people using mobile devices to take online surveys has grown from less than 10% in Q1 2012 to about 20% in Q4 2013.
  2. Panel members are less likely to be using mobile devices (7%) and participants from client lists are more likely to be using mobile devices (27%).
  3. Completion rates are lower for mobile devices (59%) than for PCs (76%), but Decipher report that well-designed mobile studies do better than this.
  4. 60% of mobile devices used in Q4 2013 were iOS, 38% Android, with all other operating systems coming to just 2%.