Survey methodology is a vast and complex discipline, one that often requires years of research practice and advanced academic credentials (don’t worry, you don’t need a PhD to understand this primer!). Surveys are created and deployed by researchers, market research teams, government entities, and organizations to understand the behaviors, opinions, and attitudes within and across different populations. Surveys are also a popular method; a recent Google search for “creating a survey” yielded more than one billion results.

This primer offers advice on survey best practices, with a specific focus on the beginnings of the survey creation process—from choosing a survey tool and writing survey questions, to articulating the purpose of a survey and accessibility considerations. These best practices will provide a foundation to help you collect reliable, quality data to understand your target audiences more deeply. Please note: This primer does not claim to be an all-encompassing source of survey knowledge; it is an introduction to 6 considerations when creating surveys.

1. Think Like a Respondent

One way to approach creating a survey requires thinking like a survey respondent. Dillman et al.¹ describe this approach as the respondent state of mind, which is meant to push the survey creator to think through the objectives, questions, language, response options, and other survey aspects from the perspective of a respondent. The authors provide several questions to consider when doing this, such as, “What will the respondent here or see first?”, “Will [they] be able to understand the questions?”, and “How will [they] know where to start and what the navigational path is through the questionnaire?”.^1p94 Other questions to consider include: “Will the respondent understand the survey instructions and what is expected of them?”, “What information will the respondent feel comfortable sharing?”, and “Are the response choices clear, and do they provide respondents with satisfactory options to answer the questions?”.

Thinking like a respondent also involves defining a clear set of objectives—or goals—for your survey. Why are you creating the survey, and what is its purpose? What do you want to know, and who do you want to hear from? What are the topics you will investigate? For instance, are you interested in assessing the reach of a journal’s readership, the effectiveness of editorial policies, or measuring authors’ and reviewers’ satisfaction levels with the manuscript submission process to improve the publication experience? You will position yourself for success when the goals of your survey align with the questions you intend to ask.

2. Write Good Survey Questions 

Writing quality survey questions is arguably the most important part of the design process. It is also difficult to do. However, the questions you want answered may have already been tested and vetted. You can search for these questions using the Pew Research Center,² Gallup,³ and other publicly available market research surveys. But if you are authoring the survey questions, it is important to first define concepts. Concepts are ideas that drive the research.⁴ They help connect and anchor what you aim to investigate.

If we consider the research question, “How do editorial policies impact authors’ publication experiences?”, our main concepts are editorial policies and authors’ publication experiences. Main concepts can be further defined into subconcepts, which will help focus what you will test and measure.¹ For example, editorial policies can be broken into subconcepts, such as the number of policies a journal has, correction or retraction rates, and overall author satisfaction with editorial policies. Authors’ publication experiences can be segmented into number of publications, challenges encountered during the publication process, and average time to publication. Remember: Clearly defined survey objectives will help you articulate your concepts. Once your concepts are aligned with the survey’s objectives, it is time to consider the different question types to measure the concepts. Examples of 5 common question types are presented in the Table.

Table. Common survey question types with examples.

Crafting survey questions can be challenging when testing multiple concepts. To create an effective, nonbiased survey experience for your respondents, the following should be avoided when authoring survey questions:

• Leading, biased questions.   Avoid questions that prompt—or lead—respondents to select a particular answer. Consider this example: “Given the delays authors face during the publication process, do you agree that journal editors often lack efficiency in their handling of submissions?” This question assumes that authors encounter delays in the publication process, and editors are inefficient when handling manuscript submissions.
• Double-barreled questions.   Double-barreled questions ask about 2 topics simultaneously, which can cause confusion and prevent an accurate analysis of the respondents’ answers.⁵ Take the question: “How familiar are you with the publication process and editorial policies?” This question asks about familiarity with the publication process and editorial policies. Any answer provided by respondents will not accurately capture their familiarity with either the publication process or editorial policies.⁴
• Omitting “don’t know”, “prefer not to answer”, and “none of the above” response options.   Allow respondents to opt-out of questions they might be uncertain about or do not wish to answer.
• Inconsistent scales.   Maintain a consistent presentation of scales in your survey responses, regardless of whether they are displayed in a positive (strongly agree 🡪 strongly disagree) or negative (strongly disagree 🡪 strongly agree) direction. Dillman and colleagues¹ caution that a sudden change in scale direction often goes unnoticed by respondents, not because “they are being lazy respondents”, but because they do not anticipate these changes in the survey flow.^1p154 As such, respondents will select the wrong responses.
• Misleading language.   “Did you find the publication process easy?” primes respondents to believe that the publication process was easy. Instead, use language that is simple, concise, and prevents confusion and bias in your questions and responses: “How satisfied are you with the publication process?” (response options: strongly dissatisfied 🡪 strongly satisfied).
• Overburdening respondents.   Limit the respondents’ response burden by ensuring question response options are not confusing, the survey is manageable to complete, and questions are organized by importance and/or relevance.
• Lengthy surveys.   Optimize respondents’ survey experiences by facilitating short survey completion times to get the best data possible. This avoids survey “satisficing”—when respondents speed through answers—survey abandonment, and respondent fatigue.6 While there is not a prescribed completion time for online surveys, previous research suggests aiming for 10–15 minutes.⁷ 

To become better acquainted with survey best practices, different survey types/formats, and to participate in surveys firsthand, consider exploring well-known survey distribution platforms, such as Amazon Mechanical Turk,⁸ Branded Surveys,⁹ and SurveyJunkie.¹⁰ These platforms compensate respondents to complete professionally developed market research surveys.

3. Choose a Survey Tool

The landscape of online survey tools is constantly evolving. Where you choose to host your survey will ultimately inform the type of survey you are conducting. The choice of tool also depends on how robust your survey needs are. Examples of popular survey software on the market today include:

• SurveyMonkey,¹¹ a well-known, easy-to-use platform with an available free option
• Qualtrics,¹² a platform known for market research and complex survey design capabilities and features, like analytics and dashboards and automation
• Google Forms,¹³ a free tool available in the Google Workspace that has several accessible premade survey templates
• Microsoft Forms,¹⁴ a survey tool from the Microsoft 365 suite that contains pre-built templates, similar to Google Forms

Some software options listed above require subscriptions to access additional features. Check with your institutions and/or organizations to see if you have access. There are also several other survey software and platform options available and searchable via Google.

4. Articulate the Survey’s Purpose

Before distributing your survey, it is important to ensure that the content of your initial reach out—whether through email, a marketing campaign, or the survey software you choose—and survey landing page—the entry point of your survey—are accessible and easy to understand. To increase respondent engagement in your survey, Stantcheva¹⁵ recommends several best practices:

• Indicate an estimated timeframe for how long the survey will take to complete. You want to ensure that your survey can be completed in a reasonable amount of time and does not require an excessive commitment from your respondents.
• Simple language and a user-friendly survey design will help to ensure your survey is accessible to all respondents.
• Reveal just enough information about the survey’s sponsor—the individual or organization responsible for funding and/or backing the survey—to establish credibility, gain your respondents’ trust, and avoid bias wherever possible. For example, Stantcheva¹⁵ asks us to think about the difference between including, “We are a group of nonpartisan academic researchers” and “We are a group of faculty members from the Economics Department at Harvard and Princeton” on a survey’s landing page.^15p212 What is gained and/or lost by ex/including a department, institution, and/or organization name?
• Share the benefits of the research with the respondents. How will the research be used? What might the respondents learn if they participate in your survey?

5. Create an Accessible Survey Experience

There are many resources available to societies, publishers, and publishing professionals to facilitate accessible survey experiences, such as the Pew Research Center,¹⁶ Qualtrics Support,¹⁷ and Gartner¹⁸ The Web Content Accessibility (WCAG) guidance¹⁹ offers helpful resources and standards to make online content accessible. SurveyMonkey²⁰ also provides design and content guidance on creating accessibly compliant surveys, offering advice on survey themes, colors, images, icons, and formatting. Some survey platforms, such as Qualtrics,²¹ even contain built-in features that review your survey for accessibility compliance. 

6. Incentivize Survey Participation

If your society or organization has the resources to incentivize respondents for completing surveys, it is worth considering to potentially increase survey engagement. Respondents should be informed about incentives prior to completing a survey, and incentives should ideally support multiple currencies (if engaging a global respondent base). 

Closing Thoughts

This primer presents 1 perspective on 6 best practices and principles in survey design. Survey design is complex and can be a daunting task. Be empowered to learn more about the different aspects of surveys and how they can help you unlock your understanding of your target audiences’ attitudes, behaviors, and perceptions for better experiences.

References and Links

1. Dillman DA, Smyth JD, Christian LM. Internet, phone, mail, and mixed-mode surveys: the tailored design method. 4th ed. Wiley; 2014.
2. https://www.pewresearch.org/
3. https://www.gallup.com/home.aspx
4. Bryman A. Social research methods. 4th ed. Oxford University Press; 2012.
5. Pew Research Center. Writing survey questions. [accessed February 29, 2024].  https://www.pewresearch.org/our-methods/u-s-surveys/writing-survey-questions/.
6. Chudoba B. Survey completion times. Survey Monkey. [accessed February 29, 2024]. https://www.surveymonkey.com/curiosity/survey_completion_times/.
7. Revilla M, Höhne JK. How long do respondents think online surveys should be? New evidence from two online panels in Germany. Int J Market Res. 2020;62:538–545. https://doi.org/10.1177/1470785320943049.
8. https://www.mturk.com/
9. https://surveys.gobranded.com/
10. https://www.surveyjunkie.com/
11. https://www.surveymonkey.com/welcome/sem/?program=7013A000000mweBQAQ&utm_bu=CR&utm_campaign=71700000059189106&utm_adgroup=58700005410205670&utm_content=43700049188971266&utm_medium=cpc&utm_source=adwords&utm_term=p49188971266&utm_kxconfid=s4bvpi0ju&language=&test=&gad_source=1&gclid=Cj0KCQiAw6yuBhDrARIsACf94RXETdKV2dnewRgDPUBLRcWcrU2QwY_iudBNukzO2nEFaEtKRd9z53caAgSxEALw_wcB&gclsrc=aw.ds
12.  https://www.qualtrics.com/
13.  https://www.google.com/forms/about/
14. https://www.microsoft.com/en-us/microsoft-365/online-surveys-polls-quizzes
15. Stantcheva S. How to run surveys: a guide to creating your own identifying variation and revealing the invisible. Ann Rev Econ.2023;15:205–234. https://doi.org/10.1146/annurev-economics-091622-010157.
16. https://www.pewresearch.org/
17. https://www.qualtrics.com/support/
18. https://www.gartner.com/en
19. https://www.w3.org/WAI/standards-guidelines/wcag/
20. https://www.surveymonkey.com/mp/508-website-accessibility/?program=7013A000000mweBQAQ&utm_bu=CR&utm_campaign=71700000096739641&utm_adgroup=&utm_content=&utm_medium=cpc&utm_source=adwords&utm_term=&utm_kxconfid=s4bvpi0ju&language=&test=&gad_source=1&gclid=Cj0KCQiA5rGuBhCnARIsAN11vgSbZ7dwarlY7y7k480nud-qKYXjLYmMQS8L8_qrtC9ITji5FIyqwJMaAvdfEALw_wcB&gclsrc=aw.ds
21. https://www.qualtrics.com/support/survey-platform/survey-module/survey-tools/check-survey-accessibility/

Rebecca A Costantini, PhD (https://orcid.org/0000-0001-6951-047X; rcostantin9@gmail.com) is a UX researcher and an adjunct in the Departments of Communication, William Paterson University of New Jersey and Villanova University.

Opinions expressed are those of the authors and do not necessarily reflect th e opinions or policies of their employers, the Council of Science Editors, or the Editorial Board of Science Editor.