From the perspective of the PPV, power is not only defined as the probability of finding an effect of a certain size given that it exists, but it also influences a finding’s post-study probability of being true. However, what is known about power when testing for interaction effects? Recently, several authors have argued that depending on the nature of an interaction, large sample sizes may be required in order to achieve sufficient power (Blake & Gangestad, 2020; Lakens, 2020; Simonsohn, 2014; Sommet et al., 2023). Furthermore, these authors have argued that many psychologists may not be aware of the real sample-size requirements when testing for interaction effects.

Generally, as with all kinds of effects, power, and thus sample size depends on the size of an interaction’s effect.³3

Generally, the considerations presented in this manuscript apply to both categorical variables (e.g., an independent variable in an experimental design) and continuous variables (e.g., age as a moderator). For sake of simplification, many authors refer to categorical variables in a 2 x 2 design (e.g., Lakens, 2020; Sommet et al., 2023). Particularly when referring to their work, we adopted this practice.

Different authors have approached interaction effects’ power requirements from different perspectives. Sommet et al. (2023) try to arrive at the most general recommendations based on generic assumptions regarding the plausibility of different effect sizes for interactions. Sommet et al. (2023) assume that disordinal interactions typically have the same effect size as a median-sized main effect, that fully attenuated interactions typically have half the size of a median-sized main effect (and thus of a typical disordinal interaction), and that partially attenuated interactions typically have between one fourth and one fifth the size of a median-sized main effect (and thus of a typical disordinal interaction). Based on these assumptions they calculate the sample sizes required for finding interaction effects of the respective sizes.⁴4

For details on these calculations and the underlying formulas please see Sommet et al. (2023).

Both Simonsohn (2014) and Blake and Gangestad (2020) avoid general assumptions regarding interactions’ effect sizes and instead compare interaction effects to main effects in the particular case of attenuated interactions. As attenuated interactions are defined in relation to a main effect, we can determine the sample size that is needed in order to find the attenuated interaction effect with the same power as the respective main effect, without making any assumptions about effect sizes of (ordinal) interactions in general. Based on both mathematical derivations and simulations, Simonsohn (2014) and Sommet et al. (2023) show that when researchers need a specific sample size for finding a certain main effect with a certain power, then they need four times that sample size for finding the respective fully attenuated interaction with the same power. When they are looking for a partially attenuated interaction, then the sample size needs to be even larger as compared to the sample size needed for finding the main effect (Simonsohn, 2014; Sommet et al., 2023). To the extent that published original effects did not follow these rather challenging sample size requirements, they were not adequately powered and are thus less likely to replicate. Blake and Gangestad (2020) provide further illustrations of attenuated interactions’ need for large sample sizes using real-world examples from published studies.

Lakens (2020) compares ordinal and disordinal interactions “where the largest simple comparison has the same effect size”. Simple comparisons refer to the effects of one independent variable separately for the levels of the second independent variable. Regarding the shape of interaction effects (or their “pattern of means”, Lakens, 2020), Lakens shows that “in two studies where the largest simple comparison has the same effect size, a study with a disordinal interaction has much higher power than a study with an ordinal interaction”. In a specific example, Lakens (2020) compares two hypothetical studies with the same sample sizes, however one with an ordinal and one with a disordinal interaction. In both studies, the largest simple comparison has the same size. In this example, the power for finding the disordinal interaction is nearly three times as high as the power for finding the ordinal interaction. It follows that when planning sample sizes for finding interaction effects, researchers are well advised to take into account the shape of the respective interaction.

However, it would be wrong to conclude that interactions always require large sample sizes in order to be detected with sufficient power. First, as noted above, power, and thus sample size depends on the size of the respective effect. Thus, if researchers can reasonably assume a large effect size, a correspondingly small sample size will be required. Second, as shown by Lakens (2020), effect sizes for disordinal interactions are larger than effect sizes for ordinal interactions, given a largest simple comparison of the same size. Thus, when a theory allows researchers to hypothesize a disordinal interaction, they need a smaller sample size in order to obtain a certain power than when the theory allows hypothesizing an ordinal interaction, again given a largest simple comparison of the same size.

In past publications, thinking about these issues has sometimes been obfuscated, because, a) researchers referred to interactions without specifying their size or their type, or b) because researchers explicitly or implicitly assumed that interaction effects can never be larger than main effects, or c) because researchers only referred to ordinal interactions in their considerations or examples (Lakens, 2020). For example, Gelman and colleagues (2021, p. 301) use an ordinal interaction of either the same size or half the size of a main effect when discussing why “[i]interactions are harder to find than main effects”. Likewise, Maxwell and colleagues assume that interactions are often small and ordinal (Maxwell et al., 2018). Thus, it is sometimes assumed that researchers always need large samples in order to detect interaction effects, because interaction effects must be smaller than main effects, either per se or because they are attenuating interactions. Having looked at interactions from the perspective of power and sample size planning, we will now turn to the next element of the PPV, namely their pre-study probabilities of being true.

Hypothesized interaction effects, like all other hypothesized effects, have different pre-study probabilities of being true (e.g., Button et al., 2013; Wilson & Wixted, 2018). In other words, they vary continuously on a dimension from zero (low R) to one (high R). Whereas R as a theoretical parameter has a clear definition (i.e., the base rate of true effects among all effects tested in a field), it is somewhat less clear which factors affect R and how researchers are supposed to estimate R. In a very general way, R depends on “established knowledge” (Wilson & Wixted, 2018, p. 191): The more prior knowledge we have in a specific field, the less false effects get subjected to empirical tests, and thus, the base rate of true effects among all effects tested in this field increases. In a more specific way, R is supposed to be higher to the extent that a hypothesis, a) “is guided by detailed, quantitative and well-supported theories” (Miller & Ulrich, 2016, p. 685), b) is more strongly supported by previous evidence, and c) is based on common experience (Miller & Ulrich, 2016; Wilson & Wixted, 2018).

Thus, interactions can be considered to be towards the low-R end of the dimension when they are, a) based on weaker theorizing, b) less strongly supported by previous research, and c) less in line with common experience. Interactions can be considered to be towards the high-R end of the dimension when they are, a) based on stronger theorizing, b) more strongly supported by previous research, and c) more in line with common experience. For example, when researchers find a main effect and then test for an interaction of their main effect with gender, without there being a theoretical reason to do so or prior research reporting such an interaction, their hypothesized effect would have a low prior probability of being true (i.e., it constitutes a low-R interaction). However, when researchers have a theoretical reason to expect an interaction effect, or when substantial prior research has reported similar effects, then their hypothesized effect would have a higher prior probability of being true (i.e., it constitutes a high-R interaction).

To the extent that low-R interactions make up a sizeable proportion of all interactions tested in a field, interaction effects in this field will on average have low PPVs and thus poor replicability. Thus, one reason for interactions’ poorer replicability may be that interactions tested in recent psychological research on average had lower Rs than main effects. More precisely, we will argue that some interactions are particularly prone to having low Rs (whereas others may have high Rs).

From the current perspective, interaction effects should have a lower replicability than main effects when, a) reported interaction effects in a field have on average lower Rs than main effects in this field, when b) reported interaction effects in a field are on average based on studies with lower power than main effects in this field, and when c) there is a combination of the previous factors. We are convinced that there is reason to believe that all of these factors came true in past research on interaction effects: As described above, sample size requirements based on interactions’ shape have only been described recently (Lakens, 2020; Simonsohn, 2014; Sommet et al., 2023). Thus, many studies were probably strongly underpowered, even more so than they were for finding main effects (Fraley & Vazire, 2014). Furthermore, we assume that more interactions than main effects had a low pre-study probability of being true (in other words, that interaction effects on average had a lower R than main effects) due to common research strategies outlined below.

First, so-called ‘control variables’ (e.g., gender, age) may be included as moderators into research designs based on sparse theoretical foundations and few previous findings, making them low R by default. Second, researchers may first establish a main effect based on theoretical or empirical grounds and then start looking for an interaction qualifying (or moderating) that main effect. In this case, the resulting effect will be low R and have an attenuating shape. This tendency may have been enhanced by the ‘hype’ around moderation and mediation in at least some subfields of psychological research, where finding evidence for mediation greatly increased a paper’s chances to be published in top journals (Fiedler et al., 2018). Although mediational analysis itself does not involve testing for interactions, interactions still play a role in mediational analysis (MacKinnon et al., 2007): First, because an assumption for tests of pure mediation is that the independent variable and the mediator variable do not interact. Thus, any test for mediation should include a test for an interaction between the mediator and the independent variable (MacKinnon et al., 2007). Second, researchers may test for moderated mediation or mediated moderation, both of which again require testing for interactions (as moderation is statistically modelled as an interaction in moderator analysis) (MacKinnon et al., 2007). Furthermore, it seems safe to say that many researchers regarded interactions as more interesting and more newsworthy than main effects, again increasing the incentive for researchers to search for these interactions (see Rohrer & Arslan, 2021, for a similar point). Third, another common research strategy may have been to try and combine two (more or less established) theories in a novel way. The resulting hypotheses were often interactional in nature, and, due to their novelty, low R.

We would like to strongly emphasize that we do not object towards any of the above-described research strategies per se. Quite the contrary, some of them may be essential for progress in psychology; however, only if performed correctly. Thus, both when planning studies and when interpreting them, researchers should be aware of the role of interactions’ shape and interactions’ R and they should proceed accordingly. To the extent that this was not the case in past studies, interactions may have had both lower power and lower Rs than main effects and thus should be less likely to replicate. For future research, we hope that researchers take the respective factors into account, starting from the choice of research strategy, continuing when planning studies and finally when interpreting their results. The following framework might be helpful for this end.

It seems possible to combine the shape of hypothesized interaction effects (with its implications for sample size planning) with their prior probability of being true, leading to a two-dimensional grid (Figure2). The y-axis represents the shape of interaction effects and depicts a continuum from ordinal to disordinal (or crossover). The x-axis represents the prior probability of being true of interaction effects and depicts a continuum from low-R to high-R.

Sector A appears to combine the best of both worlds: A high prior probability of being true with a shape that requires comparably small sample sizes in order to achieve sufficient power. Researchers who can predict a disordinal interaction on theoretical grounds can thus hope to find an interaction effect that has a high likelihood of being true and thus a high likelihood of being replicable based on comparably small sample sizes. Consequentially, it is precisely these kinds of interactions that readers should place most trust into.

Sector B contains potential interaction effects that appear to be important for theoretical and practical reasons, however that are hard to detect statistically: Both from a theoretical and an applied perspective it might be important to know whether a main effect is attenuated (e.g., when an intervention works better for women than for men). Thus, research on these effects seems worthwhile to conduct. However, researchers are well advised to utilize appropriate sample sizes and appropriate sample size planning. Sample size planning should be based on expected patterns of means (Lakens, 2020) and the exact nature of the expected attenuation (Simonsohn, 2014; see also Blake & Gangestad, 2020, for research on attenuated interactions). For this end, it might be helpful to use a software that allows entering hypothesized effects directly via their means (e.g., Superpower; Lakens & Caldwell, 2021; see Lakens & Caldwell, 2021, for more suggestions) or that allows drawing the shape of the expected interaction (INTxPower; Sommet et al., 2023). Depending on underlying effect sizes and the exact nature of the assumed attenuation, sample sizes may need to be very large in order to achieve appropriate power. Attenuated interaction effects found in small sample studies should be treated with care.

Sector C can be considered the ‘danger zone’ of research on interaction effects, as it combines the worst of both worlds: A low prior probability of being true with a shape that requires large sample sizes in order to achieve sufficient power. In fact, depending on underlying effect sizes and the exact nature of the assumed attenuation, these requirements may not only be challenging but sometimes hard to meet (Gelman et al., 2021; Simonsohn, 2014). Interaction effects fall in this sector when researchers test for attenuated moderation without having strong theoretical reason to do so. An example for this might be ordinal interactions with gender, that arise as the result of including gender as a control variable. If researchers insist on testing for such interactions, they are well advised to base their conclusions on sufficiently large sample sizes, taking both low R and the interaction’s shape into account. Again, it might be helpful to use a software that allows entering hypothesized effects directly via their means (Lakens & Caldwell, 2021; Sommet et al., 2023). Furthermore, when interpreting such interactions both authors and readers of the literature are well advised to consider that these interactions have a higher likelihood of being false positive and a lower likelihood of replicating, particularly when they are based on small sample sizes. Thus, neither theoretical conclusions, nor consequential real-world decisions nor the allocation of future research resources should easily be based on them.

Sector D contains potential interaction effects that seem to be less problematic than the ones in Sector C, however more problematic than the ones in Sector A. Although these effects also have a comparably low prior probability of being true, they are less challenging from a statistical perspective as they require smaller sample sizes due to their disordinal shape. Still, researchers might be well advised to treat them with care after they have been reported.

As sample sizes in order to investigate predictions from Sectors C and D can quickly need to be very large, researchers might want to ask themselves whether the potential benefits are worth the costs (e.g., Miller & Ulrich, 2016). For example, if investigating one potential interaction effect from Sector C requires a number of participants that would suffice for investigating several hypotheses from Sectors A or B, researchers need to decide whether the potential increase in knowledge regarding the effect is worth the effort.

Lakens (2020) cautions researchers not to shy away from “risky predictions”, just because these seem effortful to investigate. We fully agree. In the present framework, risky predictions can be found in Sectors C and D (i.e., risky predictions are predictions with a low prior probability of being true). The present framework allows deriving some recommendations on how to investigate risky predictions (see also Wilson & Wixted, 2018, for recommendations on low-R research; and Blake & Gangestad, 2020, for research on attenuated interactions). Generally, for given Rs, PPVs can be improved by increasing power and by lowering alpha. Thus, when investigating low-R hypotheses, researchers can, a) increase power by increasing sample sizes (while holding alpha constant), b) increase power by utilizing suitable experimental designs (e.g., within-participants designs have higher power than between-participants designs), if possible, c) increase power by increasing the reliability of the dependent variable⁵5

Whereas most of the literature on improving power in psychological research focusses on sample sizes, improving reliability seems to be a bit overlooked, potentially because researchers assume that their variables are reliable anyway. However, if there is room for improving reliability, the effects on power are substantial (see Sommet et al., 2023, for an illustration).

Researchers interested in planning their research project based on the PPV, R, power and α can use the algorithm depicted in Figure 3 as an orientation. The left-hand side of Figure 3 presents a heuristic approach for approximating the strength of R. The right-hand side of the figure shows what levels of power and α are needed in order to obtain a specific PPV, based on the approximations of R made on the left-hand side. Thus, instead of solving Equation 1 for individual values of R and power, Figure 3 presents a simplification for selected levels of R and power. Figure 3 shows the PPV for different combinations of R, power (1 – β) and α. We combine three values of R (.5, .25, and .1) with three values of power (.1, .5, and .8) and two values of α (.05 and .01). The impact of these estimates on the PPV is shown on the right-hand side of the flow chart. For example, for hypotheses with low R, a statistically significant result — even from high-powered studies — still leaves considerable uncertainty about the truth of the result. When α is lowered to .01, the PPV increases substantially, and particularly so for either low-power or low-R research, or for combinations thereof.

However, low levels of power mean that researchers only have a small chance of finding an effect given that it exists, even when the corresponding PPVs can become large. For example, even when the combination of R = .25, power = .5 and α = .01 means that a significant result represents a true effect in 93% of cases, still only 50% of true effects become significant in the first place.

In this approach, R is influenced by two key factors (left-hand side of Figure 3): (1) the plausibility of the hypothesis, and (2) indirect supporting evidence. Plausibility can be further divided into mechanistic plausibility and theoretical plausibility.

Mechanistic plausibility assesses whether we know of a mechanism supporting the hypothesis. For example, in medical research, a physiological mechanism may be established in mice, before respective hypotheses are tested in humans. Likewise, in psychological research, a hypothesis that is based on an established process may seem more plausible than a hypothesis which is not based on an established process. Theoretical plausibility assesses whether a hypothesis is aligned with a well-established theoretical framework. For example, in line with the Theory of Planned Behavior, it seems more plausible that subjective norms have a positive association with intentions than that they have a negative association with intentions (Ajzen, 1991). Plausibility is related to the definition of R as the base rate of true effects tested in a field, because a field that tests more plausible effects will on average have a higher proportion of true effects than a field that tests less plausible effects.

Besides the plausibility of the hypothesis, assessing the extent of indirect evidence in support of the hypothesis can serve to inform estimates of R. Indirectness is a well-established concept in evidence-based medicine (EBM) (Guyatt et al., 2011). Indirectness can refer to Indirectness of Population (e.g., when gender moderates the effectiveness of a psychological intervention in a specific population, it is plausible to expect a respective interaction in a related population), Indirectness of Intervention (e.g., when gender moderates the effectiveness of a specific intervention, it is plausible to expect a respective interaction for a related intervention), and Indirectness of Outcome (e.g., when gender moderates the effectiveness of a specific intervention with regard to a specific dependent variable, it is plausible to expect a respective interaction with regard to a related dependent variable).⁶6

In EBM, also the Indirectness of Comparator can be assessed. However, it seems difficult to find a psychological equivalent.

Researchers interested in planning their research project based on the PPV, R, power and α can use the algorithm in Figure 3 as follows: First, they need to arrive at an estimate of R based on the mechanistic and theoretical plausibility of their hypothesis and indirect evidence in support of their hypothesis. We suggest adopting some generic levels of R depending on the plausibility and the availability of indirect evidence: When a hypothesis seems implausible and there is no indirect evidence, we suggest adopting a generic R of .10. When a hypothesis is either plausible, or there is indirect evidence, we suggest adopting a generic R of .25. When there is both plausibility and indirect evidence, we suggest adopting a generic R of .5 Of course, these values do not represent the “real Rs” of the respective hypothesis, but they offer some orientation.

Second, researchers need to decide what level of power they want to achieve, both for optimizing power itself and with regard to the PPV. For this decision, it is helpful to consider what kind of interaction they expect (disordinal, ordinal, fully attenuated or partially attenuated), what kind of design they aim to realize (within- or between-participants), and the reliability of the dependent variable. Then, they can calculate an according sample size. When the necessary sample size seems too large, researchers can try to switch from a between-persons to a within-persons design or they can try to improve the dependent variable’s reliability. Third, they need to decide for an a priori α level in order to arrive at a PPV that they consider appropriate. The algorithm can also be used in reverse: When researchers read a paper reporting a significant result, they can try to estimate the relevant parameters and then they can decide how much trust to put into the result.

Finally, researchers might take into account the recommendations made by Rohrer and Arslan (2021), who show that researching interactions may often require more specific research questions than researchers may be aware of.