In this short article, Ruth Stevenson of Ruthless Research considers whether it is ever acceptable to break the ‘golden rule’ of questionnaire consistency when designing tracking surveys.
Many arts organisations utilise research and evaluation to enable them to make evidence-based decisions, and this often includes the use of ongoing audience or member surveys, or feedback forms, to gather information which is ‘tracked’ over time.
Something everybody ‘knows’ about research is that if you’ve done a survey once and you want to run it again (and again, and again) you have to keep the questions the same. Consistency is vitally important and you cannot make comparisons between findings over time unless the questions are identical. I know everybody knows this, because my clients tell me it all the time. It seems it is the ‘golden rule’ of research and it cannot be broken.
Now those who know me will confirm that I’m substantially more idealistic than the average person, and in an ideal world it is true that you have to keep your questions exactly the same if you want to know for sure how much something has changed between one wave of a survey and the next. (Keep that in mind while you read this, I am absolutely agreeing that consistency is the best case scenario.) However, in practice, doing this is not as straightforward as you might imagine and like many things whether or not you should change your questionnaire between waves of a survey is a bit of a judgement call.
So first, can you really be sure that the second wave (or fifth wave, or tenth wave) of your survey is actually going to be identical to the first?
The thing is, if you’re going to keep a survey consistent you need to keep it all the same. This is not simply a case of using the same questions. Any element of inconsistency, no matter how small, has the potential to impact on the respondent experience and this can affect the way that questions are approached or interpreted. The survey also needs to be consistent in terms of:
- Not adding in or taking out any questions
- Consistency of whether questions were single answer or 'tick all that apply'
- Consistency of all lists of response options
- Consistency of the order of the questions
- Consistency of the layout of the questions
- Consistency of the wording of the introduction, close screen and any other non-question text used (or not used)
- Consistency of any logos or images used (or not used)
- Consistency of any incentive used to encourage response
- Consistency of distribution method (ie paper, online, telephone)
- Consistency of sampling method and profile of respondents
- Avoidance of externally influencing factors
There are some circumstances in which you can be sure that your survey is adequately consistent. If you used a research consultant or an off-the-peg methodology such as an omnibus for multiple waves, then you can be pretty sure it is adequately consistent. If you plan to literally copy the same questionnaire as last time (in paper or online format) it will probably be adequately consistent.
The problem for anyone running a subsequent wave of a survey is that getting on top of all of this consistency isn’t always as straightforward as it looks because people do not always keep full records of these things. ‘On file’ questionnaires that I am given typically do not include notes on layout. Half of the time they do not even include lists of response options. And where they do, I often find that the final version had a surprise ‘don’t know’ or ‘prefer not to say’ option that hadn’t been recorded. This is because people draft something up, then someone proof reads it and changes it, then someone else typesets it or sets it up online and adds in something new – but no-one takes a copy of the final version. So when someone comes to run it again it is surprisingly fiddly and time consuming to work backwards to exactly re-create the original.
So what am I saying? Well I guess I’m just throwing it out there that if you are honest with yourself you may find that your survey isn’t going to be as consistent as you thought it was anyway. So maybe making a few adjustments to the question wording isn’t the worst thing in the world.
And honestly, is making a statistically robust comparison of every last one of your questions really all that important?