Psychology’s Replication Crisis

Photo: Five Thirty Eight

The science of the mind has never been exclusively objective—consider the Oedipus complex. The psychological theories of today are not nearly as blatant in their errors. However, the discipline has had a bumpy ride since Freud. One example of a popular erroneous theory is behaviorism. Founded by John B. Watson, it dominated the field from the 1920s until about 1960. Behaviorism treats the human mind as if it were a blank slate at birth—a view that influenced Michel Foucault. Noam Chomsky has debunked Foucault’s view of human nature by examining language development in children. As Chomsky has shown, language is acquired without being explicitly taught—behaviorism cannot account for this. There are some mechanisms in our brains that we are born with: not everything comes from learning or culture. Many such mistaken beliefs were prevalent back when the field was still young. Surely, though, things aren’t as bad today.

Studies in fields ranging from economics to chemistry have recently failed to replicate. Psychology has been at the heart of this controversy. Since replicability—scientists’ ability to prove a hypothesis by reproducing the same results as the original experimenters, in their own labs—helps determine whether a study provides an accurate measurement of reality, this crisis has made the public more skeptical of the social sciences.

Studies have now been done to test the replicability of past research and the results do not look good. Over the last couple of years, one group of around 200 psychologists have been involved in checking the results of previous experiments and these have failed to replicate in 14 out of 28 cases. These findings don’t just involve popular myths. They include studies you will learn about in any social psychology class and even read about in top tier scholarly journals. A different group of researchers have attempted to replicate 52 studies published in the Journal of Personality and Social Psychology and Psychological Science. Using a variety of measures to ensure reliability, they found that only between 25% and 43% of the results successfully replicated. Neither group of researchers were able to replicate the findings of more than half of the psychology studies in question.

The list of inaccurate yet popular psychological concepts is long:

• Power posing
• Multiple intelligences
• Stanford prison experiment
• Implicit bias training
• Birth order
• Ego depletion
• Sex addiction
• 10,000 hours of practice
• Video games & aggression
• Stereotype inaccuracy
• Growth mindset
• Left/right brained
• Emotional Intelligence
• Attachment style stability
• Learning styles
• That there are no psychological sex differences

Take power posing. This is the idea that, before an important event, one can gain confidence by posing in a way that one associates with feeling powerful. The central findings of the original 2010 study— that power posing causes an increase in testosterone and a decrease in cortisol — did not hold up when replications were attempted. Nevertheless, several media outlets shared these findings as if they were true.

Another finding that has failed to replicate is ego depletion. Allegedly, we all have a limited supply of willpower, which  can become depleted by overuse. New data suggests that this is completely inaccurate: “results from the current multilab registered replication of the ego-depletion effect provide evidence that, if there is any effect, it is close to zero.”

Though the replication crisis is very important, it is unlikely that one will learn about it at a university or even through the media. The media is less to blame for this than the scientists themselves. According to two recent studies, one in 2014 and the other in 2016, academic press releases were often the beginning of a chain of misinformation.

Some have argued that the replication crisis has made science better. We now know that a lot of people think otherwise. According to the Pew Research Center, 44% of Republicans and 29% of Democrats believe that the scientific method can be used to produce any conclusion the researcher wants. Some of this doubt is warranted. While Republican skepticism towards science is partly related to climate change, it is likely that the replication crisis and academic political bias also play significant roles.

Left:moderate:right ratio among professors

Political bias is common in the social sciences, due to a lack of viewpoint diversity among professors. In psychology, the left-right ratio is 17 to 1 and most other disciplines in the humanities and social sciences have a ratio of more than 10 to 1. This can undermine the quality of both research and teaching. Politically controversial topics taught at universities are often left-shifted from the truth, as Jonathan Haidt explains in The Coddling of the American Mind.

Does this mean that we shouldn’t trust psychological science? I’d argue otherwise. As Stephen Jay Gould has pointed out, “In science, ‘fact’ can only mean ‘confirmed to such a degree that it would be perverse to withhold provisional assent.” In other words, it is irrational not to assume the truth of established knowledge until reproducible studies done by independent teams suggest otherwise.

Lee Jussim (et al) expand on this in their paper “A Social Psychological Model of Scientific Practices” which lists five conditions that should be met to confirm the validity of a phenomenon:

1. Something must be found.
2. It must be subject to rigorous, pre-registered attempts at replication by researchers independent of the team who made the discovery.
3. Most of those rigorous pre-registered replication attempts must succeed.
4. Pre-registered meta-analyses of those pre-registered studies must reveal that the phenomenon exists even after attempts to assess and remove biases in the literature.
5. The conclusions and interpretations regarding the phenomenon, even when its evidentiary basis is sufficiently strong to be considered “valid new knowledge,” need to be subjected to long and intense skeptical scrutiny by the scientific community.

Take the well-established data on personality. The big five model of personality has been studied endlessly, both within and outside western culture. Cross-cultural research of this kind tends to be more robust because it can be said to apply to the human population universally. This personality model has also been replicated countless times. Personality psychology doesn’t care about making profound points: it relies on objectivity and on the admission that the truth is sometimes boring — and that’s OK.

Professors, science writers and the media as a whole ought to abstain from presenting interesting findings as objective facts, unless they are established knowledge. We can learn from the science wars, as well as from methodological errors in our scientific methods. Science is the most objective way of measuring reality that we have, and it works pretty well most of the time. But we should work to improve it, and hold it accountable when its findings are incorrect. We ought to increase the amount of viewpoint diversity among professors. And we ought to teach our students about the replication crisis. This will strengthen public trust in academia and improve our methods of scientific inquiry.