OWhen was the last time you saw a scientific article? Physics, I mean. An older scholar in my previous university department used to keep all of his scientific journals in recycled cornflake boxes. Upon entering his office, you’d be greeted by a wall of Kellogg’s roosters, occupying shelf upon shelf, on packets containing various issues of the Journal of Experimental Psychology, Psychophysiology, Journal of Neuropsychology, and more. It was a strange sight, but there was method: if you didn’t keep your journals organized, how could you be expected to find the particular paper you were looking for?
The days of cornflake boxes are over: we now have the Internet. Having been in print since the inauguration of the very first scientific journal in 1665, the overwhelming majority of research is now submitted, reviewed and read online. During the pandemic, he was often devoured on social media, a vital part of the unfolding Covid-19 story. Hard copies of journals are increasingly seen as curiosities – or not at all.
But although the Internet has transformed the way we read it, the overall system of how we publish the science remains largely unchanged. We still have scientific papers; we always send them to peer reviewers; we still have editors who give the final say on whether an article is published in their journal.
This system comes with some big issues. Chief among them is the issue of publication bias: reviewers and editors are more likely to give a scientific paper a good write-up and publish it in their journal if it reports positive or exciting results. Scientists therefore go to great lengths to boost their studies, rely on their analyzes to produce “better” results, and sometimes even commit fraud in order to impress these all-important gatekeepers. This radically distorts our view of what really happened.
There are possible fixes that change how journals work. Perhaps the decision to publish could be made solely on the basis of a study’s methodology, rather than its results (this is already happening to a modest extent in a few journals). Maybe scientists could just publish everything their default searches, and journals would organize, rather than decide, what results would be released to the world. But maybe we could go further and get rid of scientific papers altogether.
Scientists are obsessed with papers – in particular, with more papers published under their name, expanding the crucial “publications” section of their CVs. It may therefore seem outrageous to suggest that we could do without it. But this obsession is the problem. Paradoxically, the sacred status of a published and peer-reviewed article makes it more difficult to fully understand the content of these documents.
Consider the messy reality of scientific research. Studies almost always yield strange and unexpected numbers that complicate any simple interpretation. But a traditional paper – word count and all – forces you to make things dumber. If what you are aiming for is a large and important goal of a published article, the temptation is always there to file away some of the jagged edges of your results, to help “tell a better story”. Many scientists admit, in surveys, to doing just that – turning their findings into unambiguous and attractive articles, but twisting the science along the way.
And think about corrections. We know that scientific articles regularly contain errors. An algorithm that scoured thousands of psychology papers found that at worst, over 50% had a specific statistical error and over 15% had an error severe enough to override the results. With articles, correcting this kind of mistake is a chore: you have to write in the journal, get the busy editor’s attention, and get them to publish a new, short article that formally details the correction. Many scientists who request corrections find themselves blocked or ignored by journals. Imagine the number of errors that litter the scientific literature that have not been corrected because doing so is simply too much. problems.
Finally, consider the data. Back then, sharing the raw data that formed the basis of an article with the readers of that article was more or less impossible. Now this can be done with just a few clicks, by uploading the data to an open repository. And yet, we act as if we were living in the world of yesteryear: newspapers almost never have the data attached, which prevents critics and readers from seeing the big picture.
The solution to all of these problems is the same as the answer to “How do I organize my journals if I don’t use cornflake boxes?” Use the Internet. We can turn articles into mini-websites (sometimes called “notebooks”) that openly report the results of a given study. Not only does this give everyone a view of the whole process, from data to analysis to redaction – the dataset would be added to the website along with all the statistical code used to analyze it, and n Anyone could replicate the full scan and verify that they get the same numbers – but any corrections could be made quickly and efficiently, with the date and time of all updates publicly recorded.
This would be a major improvement over the status quo, where the analysis and writing of papers takes place entirely in private, with scientists then choosing on a whim to make their findings public. Of course, shedding light on the whole process might reveal ambiguities or contradictions in the results that are hard to explain – but that’s what science really is. There are also other potential advantages of this high-tech way of publishing science: for example, if you were conducting a long-term study on the climate or on child development, it would be a breeze. to add new data as they appear.
There are obstacles to big changes like this. Some are skill-related: it’s easy to write a Word document with your results and send it to a journal, like we do now; it’s harder to build a notebook website that weaves data, code, and interpretation together. More importantly, how would peer review work in this scenario? It has been suggested that scientists could hire ‘red teams’ – people whose job it is to poke holes in your findings – to dig into their notebook sites and test them until they are destroyed. But who would pay, and exactly how the system would work, is up for debate.
We’ve made amazing progress in so many areas of science, yet we’re still stuck with the old, flawed model of publishing research. Indeed, even the name “paper” refers to a bygone era. Some fields of science are already moving in the direction I’ve described here, using online notebooks instead of journals – living documents instead of living fossils. It’s time for the rest of science to follow suit.
Why trust science? by Naomi Oreskes
The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice by Chris Chambers
Rigor Mortis: How sloppy science creates worthless cures, crushes hope and wastes billions by Richard Harris