In these days of web-based survey tools, you’d think the survey process would be simple and foolproof. And free, too, don’t forget about free. It all happens through the magic of the Internet, after all. Is it really that easy? Not so fast. After you field your survey, you might see the responses barely trickling in. Receiving an inadequate number of responses is by far the worst thing that can go wrong with your dissertation survey.
Oh, no! People aren’t responding to my survey!
Here’s what happens:
- You draft a bunch of survey questions (or adapt an existing set of questions).
- You get IRB approval to recruit a sample of participants to take your survey.
- You prepare your survey (most likely a web-based survey, but you might use old-fashioned paper and pencil).
- You get a list of email addresses for potential respondents (or contact a gatekeeper who has access to a list of postal addresses).
- You send out your survey link (or mail out your paper surveys).
- You sit back and wait for responses to come flooding in.
You wait. And wait. And wait.
You might think this can’t possibly happen to you, but sadly, it could. One of my dissertator clients needed at least 150 respondents to be able to perform her planned data analyses. She received fewer than 30 responses. After multiple attempts to broaden her sampling frame, she eventually ran up against the end of her Ph.D. program timeline; she had no choice but to settle for a second Master’s degree. All that time, all that money, all those hopes… gone because she couldn’t motivate enough people to respond to her survey.
In case you are wondering, her sampling frame consisted of critical care nurses. She knew quite a few nurses personally. She counted on respondents to forward the survey link to colleagues (a “snowball” recruitment strategy). It didn’t work. She offered a gift-card incentive: That didn’t work either. She expanded her sampling frame to include retired nurses. Still no success. She was out of time. If she had prepared a contingency plan earlier in her dissertation process, she might have been able to pivot and recruit enough respondents before her program ended.
What happens if no one responds?
If you get no responses to your survey, you won’t have any data to analyze, and that means your study is dead. No data means no study. It’s unlikely you will get zero responses. But it is quite possible—likely, even—that you will receive fewer responses than you need to do the data analyses you planned. From my experience as a dissertation editor, low survey response rates are quite common. Dissertators somehow assume that everyone who sees a link to their surveys will eagerly click to comply, as if they have nothing better to do with their time.
Why are people unwilling to respond?
I have learned to assume nobody cares about helping us with our surveys. To save yourself some heartache, I encourage you to resign yourself to this sad fact. Unless they have a specific bone to pick about the topic, or they know you and take pity on you, or they just love the research process, respondents will not be beating down your door to take your survey. People are busy. They care more about their own problems than they do about helping you achieve your dream of earning a Ph.D. I know, hard to believe, but it’s true.
Think about it. Do you remember the last time a researcher called you on the phone? Did you drop everything and say, “Yes, I’d be happy to help you with your research! Ask me anything!” Right. Maybe I’m the only one who does that. I’m a research junkie, I confess. I bet you get survey invitations in your email inbox from time to time. I’m sure you see the occasional popup pestering you to take a survey. How often do you take time to offer your opinions?
Sometimes people don’t respond for other reasons. They might not understand the survey questions and quit the survey in embarrassment or frustration. They might not have been screened properly at the beginning of the survey, and realized part way through they didn’t actually qualify for the study. The survey questions might be too personal or cause discomfort, leading to partially completed surveys and missing data. Boring questions, poorly worded questions, misspelled words… respondents are quick to exit for any reason, no matter how small.
10 things that can go wrong with your dissertation survey
What happens if you don’t get enough responses?
Receiving too little data compromises your data analysis plans. You can’t robustly correlate between groups, for example, if you only have a few observations in each group. Statistical analysis can be a powerful tool, but it depends completely on the quantity—and the quality—of your data. Collecting too few data points means your nifty statistical tools won’t work reliably. You’ll have to report in your dissertation why you conducted an ANOVA analysis with only 13 observations. It happens—ugh, so embarrassing. How could you trust any conclusions that emerged from such a paltry sample? You might as well just make it all up. Wait, no, I did not just recommend you cheat. See my rant on cheating.
The main problem with low response rates is that the people who are willing to fill out surveys are often very different from those who are unwilling. The differences between the two groups may include differences in demographic characteristics, as well as personality, attitude, motivations, and preferences. If you base your conclusions on the responses of those who were willing to fill out your survey, and don’t somehow account for the differences compared to those who were unwilling, then your conclusions may be totally off target. This is because your tiny (willing) sample was not representative of the larger (mostly unwilling) population from which it was drawn.
Simply put, bogus data lead to weak analyses, which lead to invalid conclusions.
What can you do to get people to respond?
If you are working on your proposal, it’s really good you are reading this now, because you can plan for the worst. Forewarned is forearmed, as they say. Take these steps to mitigate problems before they happen.
If you are fielding your survey and you find your survey is not generating the response rates you need, you may need to take some steps to get more data. You may need to do one or all of the following:
- Make sure you have time. It takes time to field a survey and analyze data. If you are running out of time in your program, pare down your study to the essentials. Use an existing (validated) question set, choose a simple statistical analysis technique that doesn’t require a gigantic sample, and make sure you have a sampling frame that is (a) accessible and (b) big enough for your analysis technique.
- Prepare a contingency plan. You may think you have enough time, but something is likely to go wrong, because that is typical when we survey humans. Have a backup plan. What will you do if you can’t generate enough responses? Talk to your Chair and Committee members about actions you can take if recruiting or data collection go gunnysack.
- Be ready to submit a new IRB application. The IRB application grants you approval to survey human subjects. If you need IRB approval to field your survey, keep in mind that major changes to your recruiting plan may require a second IRB application.
- Revise your survey questions. Keep questions simple, one idea per question. Test the questions in a pilot study (or use expert reviewers) and revise the questions that people don’t understand. Use the least number of questions possible and put the demographic questions at the end of the survey. Don’t ask for private information (income, race/ethnicity, etc.) if you don’t need the data.
- Broaden your sampling frame. You might prefer to talk to critical care nurses who worked in a hospital ER within the past year, but you might generate more responses if you opened up your sampling frame to critical care nurses who have ever worked in a hospital ER. Be clear about how many potential respondents exist in your target population and then realistically predict what percentage you can conceivably access. From that sampling frame, select as broad a sample as possible. Some populations are easier to reach than other populations. If you are using statistical methods that require a probability sample, you may need a rather large sampling frame so you can choose every nth member until you reach your desired sample size.
- Reach out to gatekeepers with access to populations that qualify for your study. Recruiting a robust sample is often the most difficult part of the entire dissertation study. Don’t rely on your respondents to forward the survey link to their colleagues (the snowball method). Instead, contact someone who has access to a large list of email addresses, who can forward your survey link to members. You could consider renting a mailing list from an organization or association. Wherever your sample target population congregates, that is where your survey link needs to be visible. Be ready to keep your survey open for at least a month. Publicize it multiple times. Be a pest. Your Ph.D. depends on it. Within the bounds of ethics, be relentless.
- Offer an incentive to participate. Motivating people to participate in your study may turn you into the worst type of academic—the “used-car” scholar: Please, please, please, take my survey, you can win a free iPad! Incentives can sometimes motivate potential respondents to click, but think it through carefully before you promise something to everyone who participates, especially if you promised them you wouldn’t collect any personal information and then close with, “To enter the iPad drawing, enter your name and email address here.”
There are few things more discouraging for a dissertator than waiting day after day for the survey data to trickle in. Receiving no responses from your survey is by far the worst thing that can go wrong with your dissertation survey. Now you know a few approaches to help mitigate the problem if this happens to you. In a future post, I’ll discuss what else can go wrong with a dissertation survey.
If you want more tips and suggestions, take a look at my book 28 ½ Reasons Why You Can’t Get Your Dissertation Proposal Approved.
Print book
kindle version
I’ve written the book I wished I had had when I was sweating over my proposal, watching the clock ticking and thinking, hmmmm, failure is apparently an option after all. I wanted simple, practical advice from someone who had been in my shoes, someone who didn’t throw more academic jargon at me, but shared real stories to give me insight into the obstacles that were holding me back. I wrote the book I didn’t have when I could have really used the help.
