In using these questionnaires, it is easy to think it is simply about the end product: the measurement of someone’s personality or attitude. But the reason most of these questionnaires are psychometric is also about the process of creating a tool to measure someone in this way.
How does the tool interpret and achieve that final measure? Psychometrics is therefore about understanding the tool itself and testing if it is valid and reliable.
Attitude to risk is often stated as a psychological trait, but I would argue it is more complex than that. If the former were true, an individual’s attitude to risk would be identical in different scenarios – for example, when contemplating a bungee jump – and you would be able to infer risk tolerance from other aspects of a client’s lifestyle.
But in that example, I know my attitude to risk when throwing myself off a bridge is very different compared to my attitude to risk concerning my finances.
Therefore, it is not that simple and as a result we need to ensure that we create a tool for measuring exactly what we want it to measure: specifically, someone’s attitude to risk concerning their pensions and investments, and their psychological preference for taking financial risk.
Designing psychometric questionnaires
There are a number of potential pitfalls to using a psychometric questionnaire that designers need to address:
- A questionnaire might not have been built with psychometric testing theory;
- The terminology in questions might be very complex, or assume prior investment or mathematical knowledge (for example, understanding and calculating percentages);
- Questions might ask clients to consider and compare themselves to their ‘peers’, which can be difficult to define;
- Questions might overlap and repeat themselves in some way, which can be frustrating for the respondent and invite problems, because the client naturally answers differently, assuming each question aims to elicit a fresh response;
- Finally, a question set might not capture and measure multiple dimensions of risk and, in particular, ignore the importance of emotions. How does someone feel about risk?
Poorly designed tests with ambiguous wording can lead to a range of measurement errors. If the questions used to make up an assessment are badly worded, inconsistent or confusing, the resulting test will be less reliable. Designers need to think about the questions themselves and how they are constructed: are they open or closed questions, for example?
It is also crucial that we avoid double-barrelled questions, which demand the client tries to answer two questions in one. For example, a poorly constructed question might ask: “On a scale of 1 to 5, how satisfied are you with your adviser and your investment returns?” which of course is asking for a single response to two very different questions.
We also need to think about the number of questions we include and reach a balance, between a suitably holistic understanding of a client’s complete attitude and emotions, but avoiding fatigue setting in when someone is completing the questionnaire.
However, even when a test has very clear wording, it might be designed in such a way that it is not actually measuring what is intended to assess.
For instance, a risk tolerance tool might aim to measure an individual’s willingness to take risk when, in fact, the tests measure something entirely different, such as a client’s investment time horizon or spending preferences.
