Measurement in Survey Design: It’s Not for Novices!

Measurement in Survey Design: It’s Not for Novices!

0

Measurement in Survey Design: It’s Not for Novices!

By Dr. Simon Crawford-Welch, Founder, The Critical Thought Lab

Follow me at www.linkedin.com/in/simoncrawfordwelch

Creating a great survey is both an art and a science. At the heart of it lies measurement – the “behind-the-scenes” hero that makes sure the data we gather is accurate, reliable, and truly reflects what we’re trying to understand. Measurement is so crucial in survey design and we have to get it right – here’s how we do it.

Why Measurement Matters

Measurement in surveys is all about turning abstract concepts into numbers or labels that we can analyze. Whether we’re looking at customer satisfaction, employee engagement, or public opinion, good measurement practices ensure our data is meaningful and actionable.

Key Components of Good Measurement

  • Validity
  • Construct Validity: Are we measuring what we think we’re measuring?
  • Content Validity: Does the survey cover all relevant aspects of the concept?
  • Criterion Validity: Do our survey results align with other established measures?
  • Reliability
  • Internal Consistency: Do the questions consistently measure the same thing?
  • Test-Retest Reliability: Are the results stable over time?
  • Inter-Rater Reliability: Do different people interpret the survey consistently?
  • Precision
  • Scale Sensitivity: Can the survey detect subtle differences or changes?
  • Question Clarity: Are the questions clear and easy to understand?
  • Standardization
  • Uniform Administration: Is the survey administered the same way each time?
  • Response Options: Are the response choices standardized to make comparison easy?

Best Practices for Accurate Measurement

  • Define Clear Objectives. Start with clear goals. Know what you want to measure and why. This clarity will guide the entire survey design process.
  • Pilot Testing. Test your survey on a small group first. This helps catch any confusing questions or other issues before you roll it out broadly.
  • Use Established Scales. When possible, use measurement scales that have been tested and validated. This adds credibility and reliability to your survey.
  • Train Survey Administrators. Ensure everyone involved in administering the survey knows how to do it consistently and correctly. This reduces errors and bias.

 

  • Continuous Improvement. Regularly review and update your survey based on feedback and changing needs. This keeps it relevant and accurate.

Tackling Measurement Challenges

  • Response Bias. Be aware that people might give answers they think are socially acceptable or might have tendencies like always agreeing or choosing extreme options. Design your questions to minimize these biases.
  • Cultural Differences. Keep in mind that people from different cultures might interpret questions differently. Consider these differences in your design and translation processes.

 

  • Technological Limitations. Online and mobile surveys can have issues like small screen sizes and varying levels of digital literacy. Make sure your survey is easy to navigate and understand across different devices.

Measurement is the backbone of effective survey design, ensuring the data we collect is accurate, reliable, and meaningful. One thing is for certain – it’s not for novices. By adhering to best practices and addressing potential challenges, we craft surveys that yield valuable insights and drive informed decisions. Ultimately, precise measurement transforms surveys into powerful tools for understanding the real world around us.

Sources for Additional Reading:

  • Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. John Wiley & Sons.
  • Fowler, F. J. (2014). Survey Research Methods. SAGE Publications.
  • Groves, R. M., et al. (2009). Survey Methodology. Wiley.
  • DeVellis, R. F. (2016). Scale Development: Theory and Applications. SAGE Publications.
  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The Psychology of Survey Response. Cambridge University Press.
  • Nardi, P. M. (2018). Doing Survey Research: A Guide to Quantitative Methods. Routledge.
  • Bradburn, N. M., Sudman, S., & Wansink, B. (2004). Asking Questions: The Definitive Guide to Questionnaire Design. Jossey-Bass.
  • Krosnick, J. A., & Presser, S. (2010). Question and Questionnaire Design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of Survey Research (pp. 263-313). Emerald Group Publishing
Recent Posts