methodology – Page 2 – Love Your Dissertation

February 7, 2017January 27, 2018

The worst thing that can go wrong with your dissertation survey: People don’t respond

In these days of web-based survey tools, you’d think the survey process would be simple and foolproof. And free, too, don’t forget about free. It all happens through the magic of the Internet, after all. Is it really that easy? Not so fast. After you field your survey, you might see the responses barely trickling in. Receiving an inadequate number of responses is by far the worst thing that can go wrong with your dissertation survey.

Oh, no! People aren’t responding to my survey!

Here’s what happens:

You draft a bunch of survey questions (or adapt an existing set of questions).
You get IRB approval to recruit a sample of participants to take your survey.
You prepare your survey (most likely a web-based survey, but you might use old-fashioned paper and pencil).
You get a list of email addresses for potential respondents (or contact a gatekeeper who has access to a list of postal addresses).
You send out your survey link (or mail out your paper surveys).
You sit back and wait for responses to come flooding in.

You wait. And wait. And wait.

You might think this can’t possibly happen to you, but sadly, it could. One of my dissertator clients needed at least 150 respondents to be able to perform her planned data analyses. She received fewer than 30 responses. After multiple attempts to broaden her sampling frame, she eventually ran up against the end of her Ph.D. program timeline; she had no choice but to settle for a second Master’s degree. All that time, all that money, all those hopes… gone because she couldn’t motivate enough people to respond to her survey.

In case you are wondering, her sampling frame consisted of critical care nurses. She knew quite a few nurses personally. She counted on respondents to forward the survey link to colleagues (a “snowball” recruitment strategy). It didn’t work. She offered a gift-card incentive: That didn’t work either. She expanded her sampling frame to include retired nurses. Still no success. She was out of time. If she had prepared a contingency plan earlier in her dissertation process, she might have been able to pivot and recruit enough respondents before her program ended.

What happens if no one responds?

If you get no responses to your survey, you won’t have any data to analyze, and that means your study is dead. No data means no study. It’s unlikely you will get zero responses. But it is quite possible—likely, even—that you will receive fewer responses than you need to do the data analyses you planned. From my experience as a dissertation editor, low survey response rates are quite common. Dissertators somehow assume that everyone who sees a link to their surveys will eagerly click to comply, as if they have nothing better to do with their time.

Why are people unwilling to respond?

I have learned to assume nobody cares about helping us with our surveys. To save yourself some heartache, I encourage you to resign yourself to this sad fact. Unless they have a specific bone to pick about the topic, or they know you and take pity on you, or they just love the research process, respondents will not be beating down your door to take your survey. People are busy. They care more about their own problems than they do about helping you achieve your dream of earning a Ph.D. I know, hard to believe, but it’s true.

Think about it. Do you remember the last time a researcher called you on the phone? Did you drop everything and say, “Yes, I’d be happy to help you with your research! Ask me anything!” Right. Maybe I’m the only one who does that. I’m a research junkie, I confess. I bet you get survey invitations in your email inbox from time to time. I’m sure you see the occasional popup pestering you to take a survey. How often do you take time to offer your opinions?

Sometimes people don’t respond for other reasons. They might not understand the survey questions and quit the survey in embarrassment or frustration. They might not have been screened properly at the beginning of the survey, and realized part way through they didn’t actually qualify for the study. The survey questions might be too personal or cause discomfort, leading to partially completed surveys and missing data. Boring questions, poorly worded questions, misspelled words… respondents are quick to exit for any reason, no matter how small.

10 things that can go wrong with your dissertation survey

What happens if you don’t get enough responses?

Receiving too little data compromises your data analysis plans. You can’t robustly correlate between groups, for example, if you only have a few observations in each group. Statistical analysis can be a powerful tool, but it depends completely on the quantity—and the quality—of your data. Collecting too few data points means your nifty statistical tools won’t work reliably. You’ll have to report in your dissertation why you conducted an ANOVA analysis with only 13 observations. It happens—ugh, so embarrassing. How could you trust any conclusions that emerged from such a paltry sample? You might as well just make it all up. Wait, no, I did not just recommend you cheat. See my rant on cheating.

The main problem with low response rates is that the people who are willing to fill out surveys are often very different from those who are unwilling. The differences between the two groups may include differences in demographic characteristics, as well as personality, attitude, motivations, and preferences. If you base your conclusions on the responses of those who were willing to fill out your survey, and don’t somehow account for the differences compared to those who were unwilling, then your conclusions may be totally off target. This is because your tiny (willing) sample was not representative of the larger (mostly unwilling) population from which it was drawn.

Simply put, bogus data lead to weak analyses, which lead to invalid conclusions.

What can you do to get people to respond?

If you are working on your proposal, it’s really good you are reading this now, because you can plan for the worst. Forewarned is forearmed, as they say. Take these steps to mitigate problems before they happen.

If you are fielding your survey and you find your survey is not generating the response rates you need, you may need to take some steps to get more data. You may need to do one or all of the following:

Make sure you have time. It takes time to field a survey and analyze data. If you are running out of time in your program, pare down your study to the essentials. Use an existing (validated) question set, choose a simple statistical analysis technique that doesn’t require a gigantic sample, and make sure you have a sampling frame that is (a) accessible and (b) big enough for your analysis technique.
Prepare a contingency plan. You may think you have enough time, but something is likely to go wrong, because that is typical when we survey humans. Have a backup plan. What will you do if you can’t generate enough responses? Talk to your Chair and Committee members about actions you can take if recruiting or data collection go gunnysack.
Be ready to submit a new IRB application. The IRB application grants you approval to survey human subjects. If you need IRB approval to field your survey, keep in mind that major changes to your recruiting plan may require a second IRB application.
Revise your survey questions. Keep questions simple, one idea per question. Test the questions in a pilot study (or use expert reviewers) and revise the questions that people don’t understand. Use the least number of questions possible and put the demographic questions at the end of the survey. Don’t ask for private information (income, race/ethnicity, etc.) if you don’t need the data.
Broaden your sampling frame. You might prefer to talk to critical care nurses who worked in a hospital ER within the past year, but you might generate more responses if you opened up your sampling frame to critical care nurses who have ever worked in a hospital ER. Be clear about how many potential respondents exist in your target population and then realistically predict what percentage you can conceivably access. From that sampling frame, select as broad a sample as possible. Some populations are easier to reach than other populations. If you are using statistical methods that require a probability sample, you may need a rather large sampling frame so you can choose every nth member until you reach your desired sample size.
Reach out to gatekeepers with access to populations that qualify for your study. Recruiting a robust sample is often the most difficult part of the entire dissertation study. Don’t rely on your respondents to forward the survey link to their colleagues (the snowball method). Instead, contact someone who has access to a large list of email addresses, who can forward your survey link to members. You could consider renting a mailing list from an organization or association. Wherever your sample target population congregates, that is where your survey link needs to be visible. Be ready to keep your survey open for at least a month. Publicize it multiple times. Be a pest. Your Ph.D. depends on it. Within the bounds of ethics, be relentless.
Offer an incentive to participate. Motivating people to participate in your study may turn you into the worst type of academic—the “used-car” scholar: Please, please, please, take my survey, you can win a free iPad! Incentives can sometimes motivate potential respondents to click, but think it through carefully before you promise something to everyone who participates, especially if you promised them you wouldn’t collect any personal information and then close with, “To enter the iPad drawing, enter your name and email address here.”

There are few things more discouraging for a dissertator than waiting day after day for the survey data to trickle in. Receiving no responses from your survey is by far the worst thing that can go wrong with your dissertation survey. Now you know a few approaches to help mitigate the problem if this happens to you. In a future post, I’ll discuss what else can go wrong with a dissertation survey.

If you want more tips and suggestions, take a look at my book 28 ½ Reasons Why You Can’t Get Your Dissertation Proposal Approved.

Print book

kindle version

December 5, 2016December 5, 2016

How to choose your dissertation methodology and method

When I was a doctoral candidate struggling to get my dissertation proposal approved, I was confused about what exactly I should propose as my research plan. I knew what my topic was, but I needed to be a lot more specific, and I also needed to justify every item on my plan. Where to start?

I read several of Creswell’s research design books and got even more confused. So many choices! So many terms that seem to mean the same thing but apparently don’t … words like approach, worldview, paradigm, theory, method, design, strategy, technique, tactic … it’s enough to make a poor dissertator insane.

The chart shows four quadrants with a buffet of choices. Quadrant 1 shows some “worldviews.” Quadrant 2 shows some methodologies (research designs). Quadrant 3 shows some research strategies (approaches), and Quadrant 4 shows a list of methods (tactics). For best results, choose ONE element from each quadrant.

When I was a dissertator, I assumed I was supposed to work through these elements in sequential order; I saw the word worldview and immediately froze in terror. It took me a long time to get past that first quadrant.

In the following paragraphs, I offer a slightly different approach, based on my experience and the experiences of other dissertators whose papers I have edited. Maybe this simplified approach to choosing your research methodology and method will help you quickly get your proposal approved so you can start the fun part of your dissertation journey—collecting your data!

Quadrant 1. First, according to Creswell (2009), we have to identify our overall approach—our worldview. Are you a postpositivist researcher? Are you a constructionist? Are you all about participation and advocacy? Or are you a down-to-earth pragmatist? What does all that even mean?

Quadrant 2. Next, we are required to choose a methodology. Methodology is the overall research design. You have three methodological options: quantitative, qualitative, or mixed methods (which is a combination of qualitative and quantitative). You might be perplexed: Should you do a quantitative study (focused on numbers), a qualitative study (focused on words or images), or should you do both (mixed methods)? You have to choose one. How should you make that important decision?

Quadrant 3. Within each methodology, we are also expected to choose some sort of approach or strategy. For example, if you chose a quantitative methodology, you have to decide between experimental, quasi-experimental, or non-experimental approaches. Which one would be best for your study? If you chose a qualitative methodology, you could select from five classic strategies: phenomenological, ethnography, narrative, case study, or grounded theory. (I know—I was like, what? What bizarre buffet did I just accidentally get invited to? I don’t speak this language!)

Quadrant 4. Finally, we must choose our research method—the actual tactic we will employ to collect data. Should you collect primary data or use secondary data? If you collect our own data, should you survey people? Would you interview people? Would you observe people? Some combination of these tactics? So many choices! Should you just close your eyes and throw a dart? Should you consult a psychic or an astrologer? Are the planets properly aligned? Maybe the Magic 8 Ball has the answer.

Nope, apparently not. What is a frustrated, confused dissertator to do?

Start with Quadrant 4

Instead of taking the quadrants in order one at a time, from Quadrant 1 through Quadrant 4, I suggest you consider starting with the methods (tactics) in Quadrant 4. This strategy worked well for me. I had no clue what my philosophical research worldview should be, but I knew that I needed to talk to people about my research topic. That meant conducting some interviews.

Method is the way we conduct our research. Method encompasses the what, who, where, and when of the study. The tactics are the blueprint for your study. Now, practically speaking, how are you going to get ahold of some data? You have essentially three choices. You can survey people, you can talk to people, or you can observe people, or any combination thereof. For example, you might survey a group of people before and after an event. Or you could interview people about their perceptions of the event. Or you could observe people’s behaviors during the event. Or you could do all of the above.

Now consider Quadrant 2

Once you choose your tactical-level method, it easy to determine which overall research methodology encompasses your method. If you are talking to people, that will likely generate text data—in other words, words—and that is by definition a classic qualitative methodology. In contrast, if you are sending out a survey that requires respondents to click numbers to indicate their level of agreement with some statements, that method will generate numerical data, which by definition is a classic quantitative study. If you have a combination of both words and numbers, then you have chosen a mixed-methods methodology.

Which one should you use, qualitative, quantitative, or mixed method? The correct answer is, whichever one answers your research question most effectively. Are you really asking, which one is easier? That depends. Are you a number person or a word person? Do you want to challenge yourself, or do you just need to get this thing done? Are you wondering which methodology is faster? Quantitative, usually.

Now you are ready for Quadrant 3

Now that you know your research methodology, you can determine which subcategory from Quadrant 3 is most relevant for your study. Quadrant 3 is a refinement of your methodology choice from Quadrant 2. Your choice of methodology is important, because it leads logically to your method—and vice versa. They need to align logically. You can’t proclaim your intention to use a survey to collect numerical data and then call that a qualitative methodology. Likewise, you can’t say you are going to conduct interviews and call that quantitative. According to Creswell (2009), you have five traditional options for qualitative methodology and three main options for quantitative. Remember, the best choice is the one that best answers your research questions.

Back to Quadrant 1

Finally, we come back to Quadrant 1. What is all this stuff about worldviews or paradigms? What is a philosophical worldview? That question is not too hard to answer: A worldview is a mindset, a basic set of beliefs or assumptions about how things are. A way of thinking about things. Like Republican versus Democrat, sort of, but less fraught.

The worldview we choose provides the philosophical foundation for the strategy and methods we will use for our research. Using Creswell (2009) as our guide, we have four choices when it comes to worldviews or paradigms: Post-positivism, constructivism, advocacy/participatory, and pragmatist. It should (eventually) become apparent to you that the first term in Quadrant 1, postpositivism, usually refers to quantitative ways of finding out stuff, and constructivism usually refers to qualitative ways of finding out stuff. That is a simplification, but for our purpose, it works.

Odds are your project will be one of these two worldviews. However, it’s best to choose your worldview based on the research problem you have identified. So, for example, if you plan on getting down and dirty with your data, like going undercover into A.A. meetings to find out how the members manage to “govern” their organization with no bosses, or interviewing LGBTQ teens with a goal of helping schools build inclusive communities, then the advocacy/participatory paradigm is the worldview for you.

The fourth worldview, pragmatist, is a smorgasbord hodgepodge of whatever you want it to be. I recommend you steer clear of this worldview—it’s difficult to pin down, because it can encompass just about any approach you can cook up, and reviewers won’t understand it. You’ll waste a lot of time defending your choice.

Dissertators sometimes want to implement the most complex study they can, as if that will prove something. You don’t have to prove anything. I encourage you to keep it simple from the beginning. If you want to get your proposal approved in the least amount of time, go with what works: either postpositivism (quantitative) or constructivism (qualitative), depending on your research problem and your propensity toward numbers versus words.

Now you’ve covered all four quadrants, from general to specific. Make the easier choices first—tactics and methodology, and then work your way to the approach and worldview, using peer-reviewed guidance gleaned from the literature in your field. With these research elements in place, you’ll soon get your dissertation proposal approved and be on your way to collecting data.

Reference

Creswell, J. W. (2009). Research design. Thousand Oaks, CA: SAGE Publications, Inc.