Why Aviation Researchers Often Should Eschew Manipulation Checks

Abstract

In many sciences, including aviation science, researchers often perform manipulation checks to demonstrate that their experimental manipulations work as hypothesized. And yet, manipulation checks can be problematic in that they can decrease generalizability; increase costs; and incur the risk that an experiment, that otherwise would result in an empirical victory for the researcher, will instead result in an empirical defeat. In addition, researchers overestimate the extent to which manipulation checks facilitate the elimination of alternative hypotheses. Consequently, the decision to use manipulation checks, or for journal editors to require them for publication, should be taken with more care than aviation scientists currently realize. Although there are exceptions, aviation researchers often should eschew manipulation checks.