Research Put to the Test: How Reliable Are Scientific Studies, Really?

Anyone reading a scientific study expects clear results. But how reliable the findings really are often only becomes apparent when they are independently verified. This was precisely the aim of SCORE(Systematising Confidence in Open Research and Evidence), the world's largest research project on the reliability of scientific findings. Researchers from almost 100 institutions - including Prof. Dr Jan Philipp Röer, Prof. Dr Johannes Michalak and Prof. Dr Thomas Ostermann from Witten/Herdecke University (UW/H) - tested 164 published papers from the social and behavioural sciences and repeated the original question with new test subjects and data. Did the same pattern of results emerge or not?

Science put to the test

About half of the analysed findings were confirmed, the other half were not. In addition, the results in the repetitions were usually weaker than in the original studies. This shows that The real challenge lies less in individual studies than in how their results are handled. They are often presented more clearly than they actually are - in science itself as well as in the media, politics and the public.

Self-criticism as a strength in science

"I don't believe that this is a problem for the credibility of scientific findings," says Prof Dr Jan Philipp Röer, holder of the Chair of General Psychology at the UW/H. "On the contrary. A failed replication does not necessarily mean that the original findings are wrong, but rather that the sum of the individual results is less clear than previously assumed." For him, SCORE is not a cause for resignation, but a mission: to understand under which conditions findings are stable and under which they are not. Failed replications are not a dead end, but a signpost: they show where further research is needed to paint the full picture.

What counts now: Transparency and open data

SCORE has not only tested - the programme has also set an example of how science should work. All data sets, analysis steps and codes are open, accessible to everyone, comprehensible and reusable. This is the promise of Open Science: not to practise research behind closed doors, but to understand it as a joint, verifiable project. "Transparency must become the standard," says Röer. "This is the only way science can create reliable knowledge that others can build on."

Further information:

The results of the SCORE study were published in "Nature", one of the most renowned scientific journals in the world. Nature is regarded as a seal of quality for research of particular relevance and methodological quality. The fact that a study on the limits of scientific reliability appears there is itself a signal: integrity and self-criticism in research are increasingly recognised as scientific achievements - not as an admission of weakness.