17. How science happens.
Messy science is still science. Meep Meep.
Somewhere around the eighth grade, most people learn that science starts with asking a question, evaluating collected facts and data, and eventually reaching a decisive conclusion.
But it often doesn’t work that way.
In the good number of years since I was in eighth grade, I have learned that sometimes scientists evaluate facts differently from one another and then reach different conclusions. It turns out that there are almost always contrarians, and objective scientists recognize that this is part of normal scientific discourse.
It’s a reflection of how confident scientists are in their conclusions. And it makes scientific findings stronger.
When science ignores some facts, but not others, that’s just not science.
Facts matter, but don’t always lead to conclusions.
A scientist collects sophisticated (and sometimes not-so-sophisticated) observations about some physical thing in the world. A biologist might sequence DNA, or a chemist would measure how long a chemical reaction might take. These observations can be of very different things: images of different structures in a cell, a list of numbers in a spreadsheet, or a computer model that calculates the trajectory of a rocket, for example. The kinds of data used and collected by scientists are really diverse, and often very different in each scientific field.
In my work, I collect measurements of air pollution concentration from all over the world - it’s something that lets me peek in to communities in dozens of different countries and see that they struggle with air pollution problems just like we do.
While scientists are usually quite skilled in collecting data, we often struggle to make sense of what our data is telling us. We try to shape our data into a conclusion that is consistent with all of the related science that we find in our field. It’s why scientific papers always end with a lengthy list of reference and citations - we have to acknowledge that our conclusions have been shaped by others. And it is evidence that a scientist conducting a study is using the most appropriate tools and techniques.
I would argue that one of the hardest parts of being a scientist requires you not only to interpret your own data, but you have to place it in the context of all of those scientists who came before you.
And sometimes what you see doesn’t match up with what everyone else saw. That is a lonely place to be.
But it is still science. Even if you’re the only one looking.
Incomplete Science is still not (yet) science
Count me a skeptic for those research findings that drive a media narrative without being published.
Normally, a scientific research paper is written and submitted to a journal, which sends the work to anonymous reviewers who provide objective and constructive feedback. This ‘peer review’ is then returned to the authors to help them make improvements - such as adding more detail, repeating key experiments, or clarifying their conclusions - and it almost always makes the science better. It’s a really important check-and-balance on scientific integrity, and requires scientists to be accountable for their conclusions.
But as we’ve seen over the past few years, there has been a lot of press on scientific papers that had not yet been through this review process.
Scientific findings are sometimes posted what are called pre-print archives, which are not proper journals, but a public repository of science in progress. But to the general public, you’d never know the difference. Can you find the fully reviewed paper? Is it this one? Or this one?
These pre-print archives are a sort of show-and-tell for scientists to put out their initial data and conclusions to seek feedback on their ideas and interpretations. It’s the Instagram of academia where anyone can weigh in on what you’re working on - which is really just a rough draft.
But to be clear, it is not yet accepted science.
The danger of these archives is that it becomes easy to lure media to amplify these results, to draw attention. And while dissenters may complain loudly about a pre-print paper, the paper itself remains posted to these sites without modification. It looks like a finished product; it is not.
Doing so undercuts scientific integrity when research is eventually reviewed and changes are needed because of minor technical problems or addition of more data. It diminishes the impact of science because we sometimes have to backtrack, correct, or withdraw a conclusion, which calls in to question our intent and integrity. This is not to say that most scientists are intentionally deceptive, or are seeking to publish nonsense. Mistakes can, and do, happen. It’s part of the process.
I worry about integrity the most for science that falls in the public interest. Scientists can not afford to jeopardize our duty to finding truth when we rush our conclusions out to the public when they are incomplete or perhaps with minor flaws. Too much is at stake.
In other words, one should not let the cat out of the bag, until we are sure that it is, indeed, a cat.
Science and the Media
It wouldn’t be much of a stretch to note that most scientists are awful at explaining their science to the public. I sometimes think of the antics of Dr. Bunsen Honeydew and Beaker (my two favorite Muppets, of course). Their incredible scientific inventions are surely good ideas, but entirely untested. Most of them wind up hilariously attacking or maiming poor Beaker, the trusty and loyal lab assistant.
It might make for good kids television, but no one would mistake this for real science. It plays well to the camera, but usually results in more damage than benefit.
Non-Muppet scientists all over the world arrive to conclusions in their field each day that are exciting and interesting and can change our understanding of problems. It is only natural they want to spread this message more widely and beyond their little corner of their scientific community. And it gives scientists an incentive to embellish, just a little, to jazz up their results.
I don’t mean they make stuff up. But maybe they wax poetically about how important their results are to the world. Or, compared to prior research, how their results are new! and exciting! and worthy of the attention!
While most scientists are seeking that eureka moment, the fact is, science can be slow. It’s methodical and incremental and needs repetition. I’d say most science is this way. It isn’t always exciting. Sometimes, science can be just boring.
And that’s the problem - boring science does not make for interesting news. It doesn’t tell a complete story, and it often doesn’t connect to people, which is a primary interest for media writers.
And to complicate things further, sometimes good scientific inquiry leads us to a failed study. In the field, we would call this a null or negative finding. These are especially boring, but are still science. And for the most part, it is especially difficult to publish articles on null findings.
“You can say that the entire history of dark matter research has been nothing but negative results, which makes it a little unusual”, Peter Meyers, Princeton University
Want to read a breaking news article about a research project where the data are entirely inconclusive? I didn’t think so.
Agree to Disagree and Other Lessons from 4th Grade
I always disliked having to show my work when I was in school - did I really need to work out that long division when everyone already knew that thirty five divided by seven was five? But it turns out that is what is required of any scientific inquiry - we show our work. As a scientist, I now know it is more important than I ever realized.
Why? Because it lets others peer in on how we conducted our science. And it lets others try to repeat what we just did. Not just our scientific colleagues, but pretty much anyone can take a look to see the recipes, the formulas, and the data sources.
There are some cases where confidential or proprietary information shouldn’t (or can’t) be shared. But these are exceptions, and not the rule. But that doesn’t stop some politicians from marginalizing science by pretending that scientists are hiding information, and according to this flawed logic, policies should be based on limited scientific input. That’s not science, and that won’t result in science-based policy.
Summing the Science
Once we have formulated our questions, conducted our research, shown our work, drawn conclusions, sought peer review feedback, and revised our conclusions, we arrive at a scientific result. We make sure to explain how this finding fits in with all of the other science, to what degree does the scientific community agree with the research. What, if anything, do the dissenters say?
Maybe its an research finding that has broad public interest. But probably not. And then we think about the next study.
Curiosity and integrity are the only prerequisites to being a scientist.. Ask your question, collect your data, drawn a conclusion, and have it reviewed by others.
Ask any question you want.
But always remember: your results may vary.