A power analysis was computed to determine the required sample size (the code is available at Weber & Röseler, 2022). Given α = β = 5%, average correlations between the scores as computed on the basis of the OpAQ data set (Röseler et al., 2022; version from August 27, 2021), and all Cronbach’s alphas = 0 except for one, which is at least .5, at least 79 participants were required.

Participants were recruited via SurveyCircle and SurveySwap, which are platforms that were specifically created for finding survey participants. They include a reward system in which a person can obtain points by participating in other researchers’ surveys. The more points one collects, the more participants one gains for one’s own study. Aside from gaining points, participants were given feedback on the extent of their susceptibility to anchoring effects as well as on the correctness of their answers to the survey, which was promoted as a general knowledge quiz. Before participating, they were also informed that the study is conducted for research about anchoring effects. There were no preselection rules; only access to the online survey was required for participation.

After data curation, 78 out of the original 96 participants remained. The planned sample size of N = 79 was not achieved because one of the participants later turned out to have completed the questionnaire two times without having admitted to doing so in the control question; therefore, the second trial had to be excluded manually after the initial data curation. Achieving a sample size of N = 78 instead of the planned N = 79 led to a minor decrease in power from T = .95 to T = .94.

The sample was predominantly female (67.9%), and participants’ ages ranged from 14 to 67 years (M = 28.05, SD = 9.01). Underaged participants were not excluded from our research to improve representativeness and because the contents of the study can be rated ethically safe as it was also approved by the ethics committee of the University of Bamberg. Overall, 76 participants came from SurveyCircle and two participants came from SurveySwap.

To examine whether there was an anchoring effect for at least two thirds of the items (anchoring hypothesis), a between-subjects analysis was applied. For every anchoring item, a one-tailed t test of the difference between the estimates with low versus high anchors was computed (direction: larger estimates for high anchors than for low anchors).

To check for whether at least one of the anchoring susceptibility scores had a different reliability than the others (reliability hypothesis), a within-subjects analysis was applied. Four different anchoring susceptibility scores were computed for every participant and item. The scores were as follows:

We tested whether the reliabilities of the four scores differed from each other by computing a chi-square test that compared the z-standardized Cronbach’s alphas. The significance criterion was set to .05 for both the t test and the chi-square test.

The study had been preregistered before the data were collected (Weber et al., 2021). Statistical analyses were computed with R version 4.0.2 (R Core Team, 2020) with the packages cocron (Diedenhofen & Musch, 2016) and ggplot2 (Wickham, 2016) and Microsoft Excel Version 2203 (Microsoft Corporation, 2022).

One of the 18 anchoring items (birth year of Alexander the Great) had to be omitted due to an error in the specification of the unit in the questionnaire. Estimates within anchoring items were excluded if, (a) participants stated that they had already known the correct answers, or (b) the estimates were more than three standard deviations away from the mean value. After applying the item-based exclusion criteria, participants who used external sources when answering the anchoring items along with participants who had nine or fewer valid items were excluded from the data set. If participants completed the survey more than once, we included only their first one.

There were significant anchoring effects for 14 of the remaining 17 items, meaning that high anchors resulted in higher estimates than low anchors. Among the items with significant anchoring effects, Cohen’s d ranged from 0.46 to 1.67 (see Table 1). Therefore, the data were consistent with the anchoring hypothesis, indicating that the anchors generally did influence the estimates in the expected way.

The reliabilities of the four anchoring susceptibility scores did not differ significantly from one another, χ²(3, N = 78) = 0.91, p = .824. Therefore, the data were not consistent with the reliability hypothesis. The Cronbach’s alpha values were set to zero for three of the scores because they had negative values. Only the restricted 0–1 score showed a small positive reliability (α = .15). Figure 1 illustrates these results.

Additional exploratory analyses were computed to investigate the influence of prior knowledge and motivation on the anchoring scores.

Anchoring effects were found in 14 out of 18 items as shown by the fact that lower estimates were given when a low anchor was presented and vice versa. Therefore, the data were consistent with the anchoring hypothesis, meaning that the intended manipulation worked and thus laid the foundation for further examination of the anchoring measurements.

What is also worth noting is that participants were informed about anchoring effects before performing the task. Not only was the effect explained to them, but they were also asked to avoid being influenced by the anchor when giving their estimates. The anchoring manipulation worked well anyway, which again underscores the robustness of anchoring effects.

Because there were no significant differences in the reliabilities of the four anchoring susceptibility scores, the reliability hypothesis was rejected. In fact, the most reliable score, which was the restricted 0–1 score, still showed very low reliability, whereas the reliabilities of the other three scores (adjustment, absolute adjustment, 0–1 score) were even negative. This result means that none of the four scores can be seen as a more reliable measurement of anchoring susceptibility than the others because none of them were actually reliable.

There are several possible explanations for these results. Certain characteristics of the present study could have led to these particularly low reliabilities. In particular, the high heterogeneity of the anchoring items is a likely reason for an inconsistently strong anchoring effect across the various items for one person. Because the items came from different domains (e.g., geography, history), prior knowledge could have impacted the anchoring effect sizes. When people have absolutely no idea about the true value of an item, they might just stick close to the anchor number, resulting in a seemingly stronger anchoring effect. When people know the exact value of an item, anchoring appears to vanish for this item. The most interesting case of anchoring is when people have to estimate the correct value on the basis of a vague idea of the possible scope. In the present study, after completing the anchoring task, participants had to indicate for every item whether they guessed the true value (i.e., they had no idea what the true value was), estimated it (they had a vague idea about what the true value was), or knew it. In the last case, the item was excluded from further analyses. As Table 2 shows only small to medium effect sizes for prior knowledge, adjustment strength does not seem to depend on previous knowledge about different items, meaning that the low reliabilities cannot be explained by prior knowledge. However, the role that prior knowledge about the items plays in an anchoring task should be further clarified in future research.

The range of plausible values is another factor that varied across items and could have led to different anchoring effect sizes and therefore to the low reliability of the susceptibility to anchoring scores. As an example: Relying on common sense, the average height of a woman in Germany can lie only within a relatively small range, that is, between about 160 cm to 170 cm. The smallest distance between Russia and the US—without previously knowing that the U.S. state of Alaska is so close to Russia—could also be estimated to be around several thousand kilometres, even though the true value is only 4 km, so even the high anchor was “only” 7 km. Differences between the estimate and the anchor can therefore be comparatively high in certain items, resulting in varying anchoring scores for one person and therefore in the low reliabilities of the anchoring susceptibility scores. In this respect, item heterogeneity can be seen as a limitation of this research.

Overall, for the present study, we chose to present a variety of anchoring items as part of our aim to obtain a valid measurement of anchoring. By not measuring estimation tendencies, which are based on the type of item, we limited the extent to which we artificially boosted the reliabilities of the anchoring scores. The low reliabilities in this study indicate that in studies in which the reliabilities of the anchoring susceptibility measurements are acceptable, the validities deserve particular scrutiny. In fact, recent examinations from the OpAQ data set, which already contains more than 50,000 anchoring trials, have shown that reliability was either low as it is in most tasks, or if not, the anchoring scores most likely lacked validity (Röseler et al., 2024).

The most plausible reason for the low reliabilities of the anchoring scores found in the present research as well as in more comprehensive meta-analyses is that there are no conditions under which anchoring susceptibility can be measured reliably. This can be explained either by the methodology employed in anchoring experiments or by the idea that anchoring susceptibility is not a stable personality trait. In this latter case, anchoring can be seen as an effect that underlies a range of situational factors but not as a general person-dependent susceptibility.

Participants in this study were recruited via platforms that are especially likely to recruit other researchers as participants (occupation was not recorded in the demographic questions). A more diverse sample would improve the representativeness and generalizability of the results. In these platforms, because participation in surveys is rewarded with points needed to gather participants for one’s own research, the motivation of the participants can also be questioned. The motivation to give correct answers and the motivation to avoid being influenced by the anchor were recorded. Figure 2 illustrates that participants generally indicated rather high motivation, and the motivation to give correct estimates was slightly higher than the motivation to avoid being influenced by the anchor. Exploratory analyses showed no systematic influence of motivation on the reliabilities of the anchoring measurements, as shown in Table 3. However, it should be considered that the results on the motivation scales could be influenced by social desirability. This is also the case for the replies to the control question about whether participants used external sources.

As already described, the choice of items and their heterogeneity can be seen as a strength of this study but also as a weakness in some ways. Some items are probably more suited than others for measuring anchoring and susceptibility to it.

Later exclusions of one of the original 18 items as well as of one of the original 79 survey trials are further limitations of the present research, as these are deviations from the preregistration and led to a minor decrease in power. Statistical analyses including all the original items and participants did not show any differences in the significance of the hypothesis tests.

The present study indicates that anchoring susceptibility cannot be measured reliably, independent of the type of the currently existing anchoring scores. The results suggest that the methodology of measuring anchoring susceptibility should be further examined and improved. If it turns out that this is not possible, existing theories that are based on the assumption that anchoring susceptibility is a stable individual parameter should be called into question. In future research, the psychological mechanisms underlying anchoring and its strength should be investigated by taking a closer look at person-independent factors rather than individual parameters.