In Assessing Motivation, Rating Scales Are Far from the Best Choice
Researchers from HSE University and the Pushkin Institute have demonstrated that pairwise comparisons work better than rating scales for measuring motivation. The reason is that many people cannot rank their motives in a hierarchical fashion. The study findings are published in Frontiers in Psychology.
In psychometrics, personality attributes are often measured using the Likert rating scale, where respondents are asked to rate their level of agreement with a particular statement, eg on a scale from 1 to 5. Researchers from HSE University and the Pushkin Institute have demonstrated that pairwise comparisons of motives tend to be more informative than the Likert rating scale.
Using the Likert scale to measure motivation is based on the assumption that a person can understand the types of motives behind their actions and rank them by importance. For example, motivation is defined as internal when a person is interested in an activity and enjoys it. Motivation can also be external when a person expects a reward for what they are doing, and motivation can be absent when a person is doing something out of habit or because they have no choice.
It turns out, however, that not all people are aware of the hierarchical order of their motives. In fact, some 40% of respondents are incapable of ordering their motives by importance consistently. But if asked to compare three motives—A, B and C—pairwise, such respondents may rate motive A as being stronger than B, motive B as being stronger than C, and motive as C being stronger than A. This ordering is cyclical rather than hierarchical and resembles the game rock-paper-scissors. The lack of transitivity in the ordering makes the Likert scale irrelevant for such respondents.
The researchers had hoped that the quantitative responses on the Likert scale could indicate those incapable of ordering their motives in a hierarchical fashion. For example, the researchers assumed that someone likely to choose average scores or to rank different motives equally could be ‘non-transitive’. This, however, was not the case.
Yulia Tyumeneva, senior research fellow of the HSE Institute of Education
'Respondents incapable of ordering their motives remain “invisible” to the Likert scale. Therefore, we often end up with irrelevant quantitative results, because counting scores for "non-transitive" respondents does not make sense.'
The only way to know whether a respondent is capable of ranking motives hierarchically—ie in a transitive manner—and, accordingly, whether offering them a Likert scale makes sense, is to ask them to compare motives pairwise. This approach provides more information about the respondents than the Likert scale by immediately identifying 'non-transitive' respondents and by revealing personal hierarchies of motives for other participants. For example, external motivation may be the strongest for some respondents and the weakest for some others. The researchers find this kind of information on motivation more valuable than a Likert scale rating.