10 things that can go wrong with your dissertation survey

I edit dissertations and proposals, which means I’ve seen many questionnaires and surveys, both proposed and completed. I’m guessing most dissertators think their survey questions are the greatest thing since Survey Monkey. However, sometimes things go wrong, sometimes seriously wrong. Here are ten potential problems you might encounter with your dissertation survey.

By the way, a questionnaire is a set of questions you will ask during your survey. In other words, the questionnaire is the instrument, and survey is your method. You can ask questions in other ways. Right now I’m just talking about questions you might ask using an online survey method.

1. Your survey questions don’t align with your research questions

Before you ask anyone anything, make sure your survey questions align with your research questions, so the data you collect will actually be relevant. There’s no point asking people for data you can’t use. The entire project will be a colossally embarrassing waste of time. You’ll spend months desperately trying to contort your data to address the purpose of your study. At that point, I can understand the temptation to change your research questions to align with your data! (Please, don’t do that. )

TIP: One fail-proof approach is to design your research questions to reflect your theoretical framework, and then build your survey questions to echo your research questions.

2. You can’t find qualified respondents to fill out your questionnaire

Finding qualified people can make you feel like poor old Diogenes lugging around a lantern looking for an honest man. Once you find them, motivating them to participate in your study may turn you into the worst type of academic—the used-car scholar: Please, please, please, be in my study, you can win a free iPad!—and once you sign them up, getting people to answer thoughtfully, thoroughly, and honestly without shaming them is a real challenge, although to be honest, at that point, you won’t care about respect for your subjects. Belmont Schmelmont!*

Assume nobody cares about your study. Unless they have a specific bone to pick about the topic, or they know you and take pity on you, or they just love the research process, participants will not be beating down your door to fill out your online questionnaire. People are busy. They care more about their own problems than they do about helping you achieve your dream of earning a Ph.D. I know, hard to believe, but it’s true.

The main problem with low response rates is that the people who are willing to fill out sur­veys are often very different from those who are unwilling. The differences between the two groups may include differences in demographic characteristics, as well as personality, attitude, motivations, and preferences. If you base your conclusions on the responses of those who were willing to fill out your survey, and don’t somehow account for the differences compared to those who were un­willing, then your conclusions may be totally off target. This is because your tiny (willing) sample was not representative of the larger (mostly unwilling) population from which it was drawn.

TIP: Design your study with the audience in mind. Before you submit your proposal to your reviewers, do some homework to determine how hard it will be to recruit the sample you need. Make friends with gatekeepers at the institutions where your audience congregates. Locate mailing lists early. Find out what you need to do to get access to the people who need to take your survey. You may need to spend some money.

The worst thing that can go wrong with your dissertation survey: People don’t respond

3. You use online panels whose members are not randomly selected

People sometimes use the word random to indicate something odd happened, like “Wow, I had a totally random morning.” For purposes of devising our sampling plan, though, random has a specific meaning. When we talk about choosing a random sample we mean a probability sample. A random sample means that every person in your population of interest has a known and equal chance of being selected to participate in your study.

All you quantitative dissertators should pay attention to the concepts of probability and randomization. Harnessing these concepts makes you a powerful wizard—in my view, anyway. You may ask, why do we care so much about probability and randomization? Well, if we use some kind of randomization technique to select participants from a group of potential participants (the sampling frame), then we can use probability theory to infer that our sample will most likely represent the whole population from which the sample was drawn. Yowza! Now that’s power!

Unfortunately, we can’t harness the amazing power of probability if we don’t collect data from a randomly selected sample. Research vendors (e.g., Survey Monkey) who manage online panels may be able to recruit a sample from their respondents specifically for your project. The problem is, only people who like to fill out surveys (and win prizes) join these online panels. That means the sample is likely to be heavily biased in favor of a certain type of respondent.

In addition, although the managers of these online panels go to great lengths to validate the quality of their sample, nobody can prove that all those people who claim they are 18 to 24 really are. They might actually be a bunch of bored retirees filling out online surveys for Cheetos coupons. You might whine and say, but don’t I just need a large sample to compensate for problems? My answer is, not if the target audience you need to reach is missing from the sample in the first place.

4. You fail to properly qualify respondents and thus collect unusable data

Some respondents may not be in your sampling frame, but may somehow get a link to your survey. If you don’t build in screening questions up front, you won’t be able to screen out the respondents who don’t qualify to take your survey. Let’s say you want to collect data from females aged 25 to 64 to ask them about their experiences with online dating services.

A couple screening questions will take care of most of the respondents who don’t belong in your sample. Question 1: “Please indicate your gender.” Anyone who clicks “Male” (or “Decline to answer”) will be terminated from the survey. Question 2: “Please indicate which age bracket describes you.” Anyone who clicks “Less than age 25” or “Over age 64” (or “Decline to answer”) will be terminated from the survey. If all goes according to plan, you will be left with a sample of 25- to 64-year-old females.

Of course, without some corroborating information, you have no way of knowing who is telling the truth and who might be lying. The odds are very low that a respondent will lie for your dissertation survey, but a lot depends on your topic. A juicy or controversial topic may attract more respondents who are willing to say they meet your qualifications just to take your survey. You might say wow, cool! Sorry. If you suspect your data are bad, you won’t be able to trust your conclusions. It’s a dismal dissertator who must report that her data were compromised and therefore her entire study is invalid.

5. Your survey questions are vague, misleading, or misspelled

Validate your survey questions before you go live. That means test them on real people, get feedback from them after they answer the questions, and collect and analyze the responses to make sure you get the data you seek. You may need to apply to your IRB to do a pilot study before you get to the main event. I know, it’s a hassle, but the alternative is you spend weeks or months fielding a survey that yields unusable data.

TIP: Ask your screening questions up front to screen out unqualified participants. Ask general questions first, then move to specific questions. Place intrusive (demographic) questions at the end.

TIP: Avoid asking two questions in one. For example, “Please indicate your satisfaction with the course and the teacher.”

TIP: Avoid leading questions. For example, “Given the sad state of the current healthcare system, what actions would you recommend policymakers take in the upcoming year?”

TIP: Use plain language. Give simple instructions. Make sure everything is spelled correctly. I know you want to be an academic scholar, but for your survey questions, write like a normal person—write the way normal people talk. If your respondents are confused about what you mean, they will make up stuff, and you will end up with bad or missing data. This is why we test our questionnaires on real people first.

6. A respondent could click the SUBMIT button more than once

Depending on the software you use to present your online survey, your respondents may not see a “redirect” option that takes them to another URL when the survey is submitted. Other than the message you provide or the generic “Your response has been recorded” message, your respondent won’t see much of anything change. She may not be certain the survey was actually submitted, so she might feel motivated to click SUBMIT again, just in case, because she’s expecting to see something dramatic, like a link to claim her incentive. Survey Monkey sends respondents to a generic Survey Monkey page. As of this writing, Google Form does not.

If respondents click SUBMIT more than once, this error will show up as duplicate rows in your spreadsheet. If you have any open-ended questions, you will quickly realize that the records are duplicates: The odds of two people typing “I had a baloni and cale sanwich for lunch today” are relatively slim.

To fix this error, do not delete the row in the web-based spreadsheet. Actually, you won’t be able to delete a row, but you could delete the data in the row. It’s best not to edit the data in the actual web-based spreadsheet itself. This is your pristine data source, and you don’t want to mess it up. Download the spreadsheet to Excel, create a mirror copy of the data (see details in REASON 17 in my book), and then clean the data by deleting the data only (leave the row intact—don’t delete the row! It’s like retiring a football jersey number—it still exists, but you won’t use it, and nobody else can use it either).

TIP: Make sure your downloaded spreadsheets match row for row, column for column to the web-based spreadsheet. This is important. Rows are records. Each record number is a respondent. In the web-based survey spreadsheet, you can’t delete a row, but in Excel you can. Don’t do it. Make sure the record (row) numbers match the web-based spreadsheet.

7. A respondent could take the survey more than once

Last I heard, Survey Monkey Pro has the capability to track the IP addresses of computers, but that doesn’t mean a respondent can’t use multiple computers or other devices. I don’t think this is common (most people avoid surveys, they don’t go out of their way to take one twice); however, I suppose it could happen. Odds are, though, it won’t happen to you. I think Google Form can now track email and IP addresses, too.

You may spot something fishy in some responses that may lead you to think someone took the survey more than once. Again, responses to open-ended questions may be a tip-off. The responses may not match exactly, but spelling or wording might be the same (“cale sanwich” would be a clue). However, unless there is some monetary reward for taking the survey, this problem should not occur. It’s more likely you’ll have the problem of motivating people to fill out your survey just once, let alone twice.

8. A respondent could enter data incorrectly

For example, someone might type their age as 544 when they meant to type 54. (Darn those smartphones!) For each variable, you’ll be checking for responses that fall outside the expected range. However, if someone typed 54 when they meant to type 45, unless you have another question to corroborate their age, you’ll never catch this error.

Sometimes respondents enter data incorrectly (from their point of view) because the question is confusing or you haven’t offered them a range of choices that includes their preferred response. Most survey questions will offer an “Other” option so people can add their thoughts in a small text box. You might prefer to force them to answer one of the responses you provide, because including responses outside your range introduces all kinds of data analysis and reporting challenges for you. However, if you don’t give people choices, they either won’t answer the question, or they will click randomly, which defeats the purpose of collecting data from them in the first place.

TIP: Again, make sure your questions are clear and that you test them on real people first before you go live with your survey.

9. A respondent could answer the wrong questions

On a paper and pencil survey, this problem is common. Respondents don’t tend to read instructions. They may not see a direction that says, “skip Question 4 if you aren’t attending graduate school online.” They will flail ahead and answer the questions intended for online learners, even if they attend their program entirely onsite. On a web-based survey, using skip logic, you can allow respondents to bypass questions they aren’t qualified to answer.

Mistakes can happen, though. These data entry errors are easy to fix. On your spreadsheet, delete the data entered by all the people who shouldn’t have answered the question. This process is called cleaning the data.

10. A respondent could fail to answer one or more questions

This is by far the most common problem in a survey. Tell me you haven’t skipped over some questions on surveys before! Especially open-ended questions that require us to think, or type, or both. Ugh. Any question that is hard to read, understand, or answer is more likely to be purposely skipped by respondents. Unfortunately, we can’t force respondents to answer our questions (see REASON 21, the chapter on ethical issues, in my book). Respondents sometimes bail before they get to the end, and there’s not a darn thing we can do about it.

If you are missing too much data, your analysis could be seriously jeopardized. You need to have a plan for how you will handle missing data. First, though, write a good set of survey questions! Duh.

What kind of data are missing and what effect could this have on your analysis? If you have a large enough sample, leaving out the missing cases altogether in the analysis of that question might work. But you run the risk that the people who didn’t answer the question are in some important way different from the people who did. Then you’ve got a biased analysis. Bummer.

SPSS lets you evaluate the patterns in missing data to help you figure out if the missing data patterns are random. If they are, you can exclude the cases from the analysis. If it looks like the pattern of missing data is not random, then you have to decide how you will handle the problem. Some researchers have recommended plugging in the mean, although that can lead to biased outcomes (Trochim & Donnelly, 2008). SPSS can generate the missing data for you using several methods (IBM, n.d.).

Final thoughts

Recruiting qualified respondents is hard, and they rarely cooperate, darn it. But few things are more fun for a researcher than asking people what they think about some obscure topic most people could not care less about. It’s not unlike the story of Sisyphus, rolling a boulder up a hill.

At this point, you are most likely just struggling to get your proposal approved so you can move on to data collection. Whether you are doing quantitative or qualitative research, if you need some help, check out my book.

*The Belmont Report summarizes ethical principles and guidelines for research involving human subjects.

Sources

National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. Belmont report. Retrieved from https://en.wikipedia.org/wiki/Belmont_Report

IBM. (n.d.). Impute missing data values (multiple imputation). Retrieved from https://www.ibm.com/support/knowledgecenter/en/SSLVMB_24.0.0/spss/mva/idh_idd_mi_variables.html

Trochim, W. M. K., & Donnelly, J. P. (2008). Research methods knowledge base [3rd ed.]. Mason, OH: Cengage Learning.

 

Print version
kindle version 

The worst thing that can go wrong with your dissertation survey: People don’t respond

In these days of web-based survey tools, you’d think the survey process would be simple and foolproof. And free, too, don’t forget about free. It all happens through the magic of the Internet, after all. Is it really that easy? Not so fast. After you field your survey, you might see the responses barely trickling in. Receiving an inadequate number of responses is by far the worst thing that can go wrong with your dissertation survey.

Oh, no! People aren’t responding to my survey!

Here’s what happens:

  1. You draft a bunch of survey questions (or adapt an existing set of questions).
  2. You get IRB approval to recruit a sample of participants to take your survey.
  3. You prepare your survey (most likely a web-based survey, but you might use old-fashioned paper and pencil).
  4. You get a list of email addresses for potential respondents (or contact a gatekeeper who has access to a list of postal addresses).
  5. You send out your survey link (or mail out your paper surveys).
  6. You sit back and wait for responses to come flooding in.

You wait. And wait. And wait.

You might think this can’t possibly happen to you, but sadly, it could. One of my dissertator clients needed at least 150 respondents to be able to perform her planned data analyses. She received fewer than 30 responses. After multiple attempts to broaden her sampling frame, she eventually ran up against the end of her Ph.D. program timeline; she had no choice but to settle for a second Master’s degree. All that time, all that money, all those hopes… gone because she couldn’t motivate enough people to respond to her survey.

In case you are wondering, her sampling frame consisted of critical care nurses. She knew quite a few nurses personally. She counted on respondents to forward the survey link to colleagues (a “snowball” recruitment strategy). It didn’t work. She offered a gift-card incentive: That didn’t work either. She expanded her sampling frame to include retired nurses. Still no success. She was out of time. If she had prepared a contingency plan earlier in her dissertation process, she might have been able to pivot and recruit enough respondents before her program ended.

What happens if no one responds?

If you get no responses to your survey, you won’t have any data to analyze, and that means your study is dead. No data means no study. It’s unlikely you will get zero responses. But it is quite possible—likely, even—that you will receive fewer responses than you need to do the data analyses you planned. From my experience as a dissertation editor, low survey response rates are quite common. Dissertators somehow assume that everyone who sees a link to their surveys will eagerly click to comply, as if they have nothing better to do with their time.

Why are people unwilling to respond?

I have learned to assume nobody cares about helping us with our surveys. To save yourself some heartache, I encourage you to resign yourself to this sad fact. Unless they have a specific bone to pick about the topic, or they know you and take pity on you, or they just love the research process, respondents will not be beating down your door to take your survey. People are busy. They care more about their own problems than they do about helping you achieve your dream of earning a Ph.D. I know, hard to believe, but it’s true.

Think about it. Do you remember the last time a researcher called you on the phone? Did you drop everything and say, “Yes, I’d be happy to help you with your research! Ask me anything!” Right. Maybe I’m the only one who does that. I’m a research junkie, I confess. I bet you get survey invitations in your email inbox from time to time. I’m sure you see the occasional popup pestering you to take a survey. How often do you take time to offer your opinions?

Sometimes people don’t respond for other reasons. They might not understand the survey questions and quit the survey in embarrassment or frustration. They might not have been screened properly at the beginning of the survey, and realized part way through they didn’t actually qualify for the study. The survey questions might be too personal or cause discomfort, leading to partially completed surveys and missing data. Boring questions, poorly worded questions, misspelled words… respondents are quick to exit for any reason, no matter how small.

10 things that can go wrong with your dissertation survey

What happens if you don’t get enough responses?

Receiving too little data compromises your data analysis plans. You can’t robustly correlate between groups, for example, if you only have a few observations in each group. Statistical analysis can be a powerful tool, but it depends completely on the quantity—and the quality—of your data. Collecting too few data points means your nifty statistical tools won’t work reliably. You’ll have to report in your dissertation why you conducted an ANOVA analysis with only 13 observations. It happens—ugh, so embarrassing. How could you trust any conclusions that emerged from such a paltry sample? You might as well just make it all up. Wait, no, I did not just recommend you cheat. See my rant on cheating.

The main problem with low response rates is that the people who are willing to fill out surveys are often very different from those who are unwilling. The differences between the two groups may include differences in demographic characteristics, as well as personality, attitude, motivations, and preferences. If you base your conclusions on the responses of those who were willing to fill out your survey, and don’t somehow account for the differences compared to those who were unwilling, then your conclusions may be totally off target. This is because your tiny (willing) sample was not representative of the larger (mostly unwilling) population from which it was drawn.

Simply put, bogus data lead to weak analyses, which lead to invalid conclusions.

What can you do to get people to respond?

If you are working on your proposal, it’s really good you are reading this now, because you can plan for the worst. Forewarned is forearmed, as they say. Take these steps to mitigate problems before they happen.

If you are fielding your survey and you find your survey is not generating the response rates you need, you may need to take some steps to get more data. You may need to do one or all of the following:

  • Make sure you have time. It takes time to field a survey and analyze data. If you are running out of time in your program, pare down your study to the essentials. Use an existing (validated) question set, choose a simple statistical analysis technique that doesn’t require a gigantic sample, and make sure you have a sampling frame that is (a) accessible and (b) big enough for your analysis technique.
  • Prepare a contingency plan. You may think you have enough time, but something is likely to go wrong, because that is typical when we survey humans. Have a backup plan. What will you do if you can’t generate enough responses? Talk to your Chair and Committee members about actions you can take if recruiting or data collection go gunnysack.
  • Be ready to submit a new IRB application. The IRB application grants you approval to survey human subjects. If you need IRB approval to field your survey, keep in mind that major changes to your recruiting plan may require a second IRB application.
  • Revise your survey questions. Keep questions simple, one idea per question. Test the questions in a pilot study (or use expert reviewers) and revise the questions that people don’t understand. Use the least number of questions possible and put the demographic questions at the end of the survey. Don’t ask for private information (income, race/ethnicity, etc.) if you don’t need the data.
  • Broaden your sampling frame. You might prefer to talk to critical care nurses who worked in a hospital ER within the past year, but you might generate more responses if you opened up your sampling frame to critical care nurses who have ever worked in a hospital ER. Be clear about how many potential respondents exist in your target population and then realistically predict what percentage you can conceivably access. From that sampling frame, select as broad a sample as possible. Some populations are easier to reach than other populations. If you are using statistical methods that require a probability sample, you may need a rather large sampling frame so you can choose every nth member until you reach your desired sample size.
  • Reach out to gatekeepers with access to populations that qualify for your study. Recruiting a robust sample is often the most difficult part of the entire dissertation study. Don’t rely on your respondents to forward the survey link to their colleagues (the snowball method). Instead, contact someone who has access to a large list of email addresses, who can forward your survey link to members. You could consider renting a mailing list from an organization or association. Wherever your sample target population congregates, that is where your survey link needs to be visible. Be ready to keep your survey open for at least a month. Publicize it multiple times. Be a pest. Your Ph.D. depends on it. Within the bounds of ethics, be relentless.
  • Offer an incentive to participate. Motivating people to participate in your study may turn you into the worst type of academic—the “used-car” scholar: Please, please, please, take my survey, you can win a free iPad! Incentives can sometimes motivate potential respondents to click, but think it through carefully before you promise something to everyone who participates, especially if you promised them you wouldn’t collect any personal information and then close with, “To enter the iPad drawing, enter your name and email address here.”

There are few things more discouraging for a dissertator than waiting day after day for the survey data to trickle in. Receiving no responses from your survey is by far the worst thing that can go wrong with your dissertation survey. Now you know a few approaches to help mitigate the problem if this happens to you. In a future post, I’ll discuss what else can go wrong with a dissertation survey.

If you want more tips and suggestions, take a look at my book 28 ½ Reasons Why You Can’t Get Your Dissertation Proposal Approved.

Print book 

kindle version

Verified by MonsterInsights