In addition to administering our own surveys, the Office of Institutional Research, Analysis, and Planning supports the Bates community in its efforts to engage in survey research. This support includes consultation about survey design, distribution, and analysis of results. Additionally, the office provides guidance about how to use Qualtrics, the web survey tool that Bates has licensed and made available to all faculty, staff, and students. For instructions about setting up a Qualtrics account, please visit this page.
Support for your survey is not guaranteed and is dependent upon the capacity of the office. In order to make sure that you get the support that you need, it is strongly recommended that you give yourself sufficient time to plan and to consult the guidelines on this site.
A General Guide to Administering Surveys
1. Formulate research questions
Formulating your research questions is a critical step in this process as these questions will guide your research from how you design your study to how you analyze and report your results. Your research questions should be clear and purposeful. With clear research questions, it should be obvious what your research is trying to accomplish. Additionally, the rationale for your study should go beyond “I’m interested to know;” there should be practical or theoretical implications for your study. In order to develop strong research questions, it is important to review the literature relevant to your topic. It is likely that researchers have already addressed they topic you wish to study and their work can inform the theoretical framing and research design that you use. For those with little experience conducting a literature review, consider consulting the Research Services staff at Ladd Library.
2. Consider whether you really need to conduct a survey
Having formulated your research questions, you may find that administering a survey is not appropriate for your study. For example, depending on your research questions, it may make more sense to use constructivist approaches like interviews or focus groups.
If you determine that survey data are the appropriate data source for your study, it is possible that the data have already been collected. For decades, Bates College has engaged in survey research with students, alumni, faculty, and staff. The Survey page on the IRAP website provides information about the surveys that we regularly administer. For more information, please contact Tom McGuinness (email@example.com).
3. Establish an appropriate timeline
While setting up a web survey and distributing it via email can be a quick process, the preparations one needs to do in advance of survey administration can be fairly time consuming. This process can take several months and should involve careful consideration of your target distribution date and the amount of time it will take to accomplish all that needs to be done prior to this date. It is important to give yourself ample time to plan, develop, and test your survey.
Avoiding conflicts with other survey activity
When choosing a target distribution date, it is also important to consider the other things that are may be happening at the time. For example, May is a busy time due to the administration of the Senior Survey and Your First College Year Survey, while November and March are particularly busy for student-administered surveys. Other considerations are times that are difficult to attract responses, such as vacations and midterm and final exam periods. Please consult the Bates Survey Calendar to determine whether there are any potential conflicts. If you would like to add your survey to the Bates Survey Calendar, please contact Tom McGuinness (firstname.lastname@example.org).
4. Identify your target population
Your choice of the target population of your survey will largely be determined by your research question. For example, if your research question addresses the experience of first-year students, the target population would probably be limited to first-year students (or possibly faculty and staff who work with first-year students). Once you have identified the appropriate target population, another thing to consider is whether it makes more sense to survey the entire population (a census) or survey a sample of your target population. Determining an appropriate sample for your survey is not a service that IRAP provides, though the office can direct you to books and online resources about sampling.
Important note: The best way to reach your population is frequently through the moderated listservs on the Bates campus. For either a census or a sample, you may want the contact information of your target population. This information will only be provided by IRAP or the Registrar’s Office in special circumstances and this determination will be made on a case by case basis. If you do need the contact information of your target population, it is recommended that you contact Tom McGuinness (email@example.com) as early as you can in your survey development process.
5. Determine the mode of survey
There are advantages to administering a survey online. Web surveys are popular because it is easy to develop them, reach your target population, and download and analyze your data. Other options include paper surveys, which yield high response rates when you have a captive audience, or interviews, which can generate a deeper, richer examination of the subject matter.
6. Design the survey instrument
There is voluminous literature on the art and science of designing a survey, so summarizing this body of work in this space would not do it justice. The following are tips and resources you may want to consider using as you design your survey.
Here are a few general survey design tips:
- Keep in mind that your survey items should align with your research questions.
- Always number your items and keep them in a bold font.
- Avoid leading questions. Keep your questions neutral and make sure they don’t lead respondents to a particular answer.
- Avoid double‐barreled questions. Example: “How satisfied were you with the sound and visuals of the presentation?” It is unclear whether you want them to evaluate the sound or the visuals. Split it into two questions.
- Be mindful of the length of your survey. Shorter surveys tend to generate stronger response rates.
- Limit the number of open-ended questions and place them at the end of the survey. Open-ended questions can be disruptive to the flow of your survey.
- Have someone review your survey and read through the questions. What makes sense to you may not make sense to the audience.
- Always test your survey. Fill it out yourself and look at the results to make sure that it is error‐free.
- An excellent place to start is the Qualtrics website. Qualtrics is not only a highly intuitive web survey tool, it also provides excellent support. This link is a useful guide for both those who are new to web surveys and those who are more experienced.
7. IMPORTANT: Include informed consent and evaluate whether you need IRB review
Some surveys will need to be reviewed by the Institutional Review Board (IRB). The IRB, which oversees research that uses human participants as subjects, aims to protect research participants by ensuring that they are treated safely and respectfully. Federal law requires that all federally funded research involving human participants receives IRB approval. For more information about IRB review and whether it applies to your survey, please visit the IRB website.
Whether your survey requires IRB review or not, ALL surveys at Bates are expected to include informed consent language. Prior to the start of your survey (e.g., in a signed consent form, in the body of an email invitation to participate, at the start of the survey, all of the above), you will need to inform your potential participants about the purpose of the survey and how their data will be used. Examples of the information you provide include how the data will be stored, who will have access, whether you can provide anonymity or confidentiality, and how and for what purpose will the data be reported. If you have any questions about whether you are providing an acceptable level of informed consent, please contact Tom McGuinness (firstname.lastname@example.org) in the Office of Institutional Research, Analysis, and Planning, Kathy Low (email@example.com) in the Dean of Faculty’s office, or the current IRB representative.
8. Consider offering incentives to respondents
If you are concerned about your response rate, you may want to consider offering incentives to your respondents. A common approach is to raffle off gift cards, with the eligible population being those who complete the survey. However, do not expect that offering incentives will guarantee a high response rate for your survey. Research has shown that while incentives have a positive effect on response rates, this effect is quite small (Porter & Whitcomb, 2003).
Keep in mind that compensating individuals for participating in your study, through either payment or a raffle, may have tax consequences. For additional information, consult IRS Publication 15-B on de-minimis (minimal) payments or email Shirley Govindasamy (firstname.lastname@example.org), Director of Payroll.
Something to keep in mind if you are considering offering incentives is that you will need to gather the contact information of survey respondents. If you intend to make your survey anonymous, it is recommended that you collect the contact information in a separate survey. Qualtrics has a straightforward way of accomplishing this.
- Set up a separate survey in Qualtrics to collect the contact information of those who wish to be entered into your raffle. A title and a text entry form with name and email address is probably all that you need.
- Select “Launch Survey” and then “Activate your survey to collect responses” and copy the anonymous link.
- Go to the edit screen of your original survey and select “Survey Options.” In the “Survey Termination” section of the “Survey Options” menu, there is an option to “Redirect to a URL.” When you select this option, paste in the anonymous URL from the new survey you have set up.
- Now, once your respondents complete and submit the survey, they will be automatically directed to the form to collect their personal information for the raffle.
9. Administer survey and collect data
Crafting an email inviting your target population to respond
When distributing your survey via email, consider how the recipient will react to receiving it. In order to encourage the recipient to open the email and click on the link to respond to your survey, it is important to employ a subject and email language that explains the value of responding to the survey and minimizes the burden of the respondent. Pieces of information that demonstrate the value of responding include the importance of the survey, how the data will be used, and incentives the respondent might receive as part of his or her participation. Minimizing the burden to the respondent includes providing an accurate estimate of the time required to complete the survey, clearly explaining the purpose of the survey, and using discretion when considering approaching your target population again with a reminder email.
Distributing your invitation email
In Qualtrics, there are two approaches you can use to administer your survey and collect data. The first approach is to use an anonymous link and distribute the survey via email to your population. The second is to distribute the survey through the Qualtrics system by setting up a Contact List. Each of these approaches has its own advantages and disadvantages.
The advantage to the anonymous link approach is that you can guarantee anonymity to your survey population. It is additionally a very straightforward and easy approach. All you have to do is activate the survey, copy the anonymous link, paste it into an email, and send the email. The potential drawback to this approach is that it is impossible to identify your respondents and, when sending reminders, you will not be able to remove any of the individuals in your population who have already responded to the survey.
Using Qualtrics to distribute your survey is a bit more complicated, but there are some real advantages to using the Qualtrics distribution system. Using this system, you pre-load in the information about your population into a Contact List in Qualtrics. (Qualtrics previously referred to this a “panel.”) The system can distribute a custom email to each person in your population. For example, by including a name and major in your contact list, you can address each individual by name and include his or her major in the text of the email. These customized emails provide a personal touch that may encourage the recipient to respond to the survey. Another advantage to this approach is that it might allow you to shorten the length of your survey. If you have relevant background information about your population (e.g., gender, race/ethnicity, class year, field of study) you can link the survey response to these data rather than including questions about these characteristics on the survey. Finally, when sending reminders from the Qualtrics system, it automatically removes individuals who have already responded from the reminder distribution, so you do not have to contact them unnecessarily. You can also take advantage of this feature and provide anonymity to your respondents by following the directions at the following link: Qualtrics University
The potential disadvantage of this approach is that it requires you to have, at a minimum, the email addresses for all of the individuals in your population. If you do not have this information, your best option is likely to be distributing an anonymous link via the Bates listservs or Bates Today.
Monitoring your survey responses
Once your survey is distributed, make sure to monitor the responses as they come in. The main Qualtrics screen (“Projects”) shows the number of respondents who have started your survey. The “Data & Analysis” tab in Qualtrics provides detail about both the individual responses and the “Reports” tab provides information about aggregate data. You will want to look at the more detailed view in order to determine the number of people who have completed your survey. Most responses come within the first 48 hours of distribution. After your initial distribution, you may want to send a reminder email to your population. Once your responses stop coming in, you will want to close your survey. This can be accomplished by selecting “Close” in your survey menu on the right side of the screen when you are on the “Projects” screen.
What is a good response rate?
Many people wonder what constitutes a good response rate. Unfortunately, there is not a simple answer. The extent to which a response rate is good is dependent upon a variety of factors, including the size of the population and whether your respondents are representative of the population you have surveyed. For more complex surveys and those based on samples, there are statistical procedures and response rate calculators that you can use to make judgments about the number of responses you may need. For simpler surveys, the key question you will want to ask is whether the results are believable and generalizable to your target population, based on the number of responses you have.
10. Analyze and report results
By the time you distribute your survey, you should have a plan for how you will analyze and report the results of your survey. You will want to have a good sense of your intended audience, the data you would like to share with this audience, and the format in which you will disseminate the results. These decisions may be a factor in whether you need IRB review. For more information about this, please consult the IRB website.
For many users, the reports that Qualtrics generates may be adequate for descriptive analyses. Reports can be viewed by selecting the View Reports button on the Projects screen or, when you are working on your project, by selecting Reports at the top of the screen. Each survey has a report, called Default Report, that displays the results of the survey in real time. It is easy to manipulate these results and export the report in a variety of file formats (i.e., Word, Excel, PowerPoint, PDF). For more information about the Qualtrics reporting feature, visit this link.
If you need a more robust analysis, it is easy to export your raw data to Excel or SPSS by selecting the Data & Analysis tab and clicking on the Export & Import button.