According to recent studies 2% of researchers admit that they have committed fraud in their work, but they thought that 14% of their colleagues had also committed fraud… fraud is basically lying…. Also in a study of 20,000 biomedical research papers 2% contained deliberately falsified images… this was reported on BBC Inside Science - Science Fraud & Bias, Immunity to COVID-19. [My wife told me about this radio broadcast she heard whilst cuddling our two new Cats Protection foster kittens (Toes and Splodge)!]
How do you publish a scientific paper?
In the pre-pandemic era having your paper published in a scientific journal was difficult. It wasn’t enough to just write it up, submit it to the editor and then sit back waiting to see your name in print. There was a rigorous process to go through in order to satisfy the editor that your paper was worthy of publication. I have both been published in journals and also rejected, as well as being a reviewer for journals!
If the editor is interested in the subject (and most papers are rejected at this stage) then it would be sent to at least 2 experts in the field of study for a peer review. These reviewers would scrutinise your work and decide if it was good enough for publication; they would also provide feedback on how the paper should be re-written or adjusted before it should be released. This was known as the “peer-review” process BUT it could take 1-2 months.
The peer-review process is based on trust. The editor trusts that the authors aren’t lying or falsifying data and the authors trust that the editorial team and reviewers will give their paper a fair chance at being published. The process isn’t perfect by any means but it is the best we’ve got at the moment.
However, this process seems to have broken down in the Covid-19 pandemic era, and clearly with a rapidly spreading disease 1-2 months to go through the peer review process in order to get new information out there is just too long. There has been a massive surge in publications about Covid-19 and in order to get research out to the World as fast as possible many of these papers are being published on the internet in PRE-peer review formats with open-access for everyone. This is very worrying as this research hasn’t had the same level of scrutiny before it has been released. As a result there has been a lot of bad evidence used to support patient care!
When I look at various scientific journals now for new information on Covid-19 I see hundreds of corrections, amendments and withdrawals of papers which just goes to show how bad some of this research was.
Only time will tell the extent of the bad scientific papers published during the pandemic.
Press statements
Another worrying trend during the pandemic is the release of clinical trial information in the form of press statements prior to, or instead of, following the normal peer-review process. Those responsible say it has been done in order to get the information out as fast as possible but I don’t believe this; yes I’m a self-confessed cynic BUT, I think it is possibly more about making headlines and supporting self-interest. I believe that if the information in the press release was really important it would reach those who need to know it however it was reported. The problem with the press release method is that the outcomes become “known” unquestionable “facts” without proper scientific scrutiny.
There have been three big examples of science–by-press-release relating to drugs for treating Covid-19.
The partial results of the clinical trials of Remdesivir, steroids and (hot-off-the-press) SNG001 for the use in the treatment of Covid-19 were released to the media and politicians BEFORE they were published and scrutinised in the scientific literature. My wife knew about SNG001 before me from the 1 o’clock BBC news! We were being told by the press that these were “game-changers” in the management of Covid-19 before doctors and scientists had seen the data, or the data even being published. Often based on single studies, one of which was funded by the manufacturer Gilead, we were being told to prescribe these drugs to our patients. This is surely wrong?
If we are going to believe the claims made from these clinical trials we should all be allowed to see the data; not just the scientific community but everyone. The data supporting these claims may have been wrong, biased, misrepresented or incomplete but we don’t know because it hadn’t been published… we only got to see the headlines. Remember being back at school and doing maths questions… you got most of the marks for how you “got to your answer”, not just the answer itself!
If we look at what has been published so far we see that for Remdesivir if it is started early in a patients illness, before they become really unwell, then it may reduce the duration of their illness. This isn’t really as dramatic as the press statement would have us believe but it’s been good for the company Gilead as the USA has bought nearly all of the World’s stock of Remdesivir!
For steroids and Covid-19 the press statement “we should be giving steroids to everyone who is mechanically ventilated or requiring oxygen” is borne out, at least to some extent, by the data that has subsequently been published. However, in order to properly assess a study a paper should give data comparing treatment versus standard care showing the patient characteristics for each group of the study e.g. this study would show steroids not requiring oxygen, requiring oxygen and requiring ventilation compared to standard care not requiring oxygen, requiring oxygen and requiring ventilation. Without detailed split data how do we know variables have been properly controlled and the differences in mortality between two treatments are due to the treatments and not just differences in the types of patients and their comorbidities within each group.
In my opinion the statement that all ventilated patients should be treated with steroids is an over-statement as it may actually only be a proportion of those patients that benefit from steroids; I’d love to see the raw data for this study as I think it has real potential but what is presented is too vague for my clinical practise.
The final drug called SNG001, an inhaled version of interferon-beta produced by a company called Synairgen, has also been reported in the media without any data or peer review. Apparently it reduced progression to severe covid-19 or death by 79%, but in whom, how exactly is it given, what are the side-effects?? We don’t know but in the pandemic era that doesn’t seem to matter… we just have to take the company’s word for it and feel grateful for the headline grabbing information!
What do I do, as a Microbiologist, when I am “presented” with these statements be it “discovered” by my wife, via a press headline on a mail server ad or delivered by the Prime Minister? Should I do as Boris Johnson would have me do and give steroids to all of my hospitalised patients…?
So why does it matter that studies aren’t peer reviewed or that the full data to support the claims made in press statements are not released to the scientific community? Well the basic answer is that there are lots of things that can go wrong in research, errors can occur but also bias becomes a risk. Peer review and scientific scrutiny is the process by which these errors can be corrected and addressed before they lead to patient harm.
What is bias?
Bias is a trendy subject in science, and it seems that the list of types of bias gets longer and longer with time. According to the pinnacle of learning (Wikipedia) bias is “the disproportionate weight in favour of or against an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair”. Basically, bias screws up the way you interpret what you are seeing.
There are lots of different types of bias but in scientific research I think the key types of bias are:
- Confirmation bias – only believing information that supports already held beliefs and ignoring information that contradicts or disproves that belief e.g. assuming that a raised procalcitonin (PCT) in a patient with Covid-19 means they have a secondary bacterial pneumonia and need antibiotics because that is what a raised PCT means in other patients even though the evidence (e.g. here and here) says that a raised PCT is a marker of severe Covid-19 and not just secondary bacterial infection
- Anchoring – the tendency to believe the first piece of information you have been given and then ignore information that contradicts this e.g. the initial reports that Hydroxychloroquine prevented Covid-19 has been very hard to reverse even when later evidence shows there is potentially even an increased mortality with this drug
- Selection bias – only choosing participants in a study that would support your perceived outcome e.g. only giving Remdesivir to patients with mild Covid-19 and comparing to those with severe Covid-19 and then saying that Remdesivir reduces mortality (don’t sue me, I might be showing my bias here!)
- Reporting bias – only selecting results that support a specific message e.g. steroids reduce mortality from Covid-19 in patients on oxygen, without reporting what happens to those not on oxygen
- Cultural bias – only believing information from within your own culture e.g. a study from the UK is correct whereas a study from China, Italy or USA is wrong
- Self-serving bias – only believing something that leads to a personal benefit e.g. only publishing something that leads to further funding of your own research, which can be affected by conflicts of interest
I would add another bias here, it is less relevant to science BUT very relevant to Covid-19:
- Framing – how the way information is presented influences the message that is perceived e.g. if the news starts with the death rate from Covid-19 saying “100 more people died yesterday from the deadly coronavirus”, then we all assume the World is ending whereas if they were to say “only 100 people have died in the last 24 hours, down from 300 yesterday”, then we feel more positive and see this as a good news story. The basic fact that 100 people died hasn’t changed, just the way it has been presented. This has been a major tool in the armament of the media and politicians and has been used to manipulate the way we ALL perceive their messages… these days we have another name for this type of bias and that is “spin”.
Publication bias
This type of bias occurs when the outcome of an experiment or research study influences the decision whether to publish it; it is a form of reporting bias. It is often used in relation to the tendency to only publish positive outcomes in research e.g. Drug A lead to a 30% reduction in mortality compared to Drug B. But why does this matter, surely we only want to know what helps our patients? Evidence suggests that 5x more publications report positive results than negative… but does this really reflect the outcome of all scientific research? Without all studies being published how do we know? We don’t! Does it matter?
Well, what if there were 5 studies comparing Drug A to Drug B and only one study showed a difference, whereas the other 4 showed no difference or even harm? If only the positive study is published we would get a false representation of the true effect of Drug A. Added to this, if someone performed a meta-analysis combining all trials for Drug A and only the positive papers had been published the meta-analysis would draw false conclusions. Everyone would start using Drug A even though it was actually of no benefit or even caused harm, as confirmed by the other unpublished studies.
Publication bias is thought to occur because there is a perception that we are all only interested in the positive story and that we will only read journals that show the new, best treatments for the diseases we treat. I actually don’t think this is true. I think most doctors just want to know the truth most of the time, and if some fantastic study proves that the antibiotic I’ve been using for years doesn’t actually work I for one would really like to know so I can stop using it!
The main approach to trying to reduce publication bias at the moment is to insist that all clinical studies are pre-registered. The aim is to have a database of all these studies so that someone (but who??!) can make sure that they are all reported and published. It’s a nice idea but I’m not sure how effective it is in practice; there are an awful lot of studies to police and are they all really going to pre-register?
Conflicts of interest
This is a big annoyance of mine, especially when it comes to published guidelines; it is a form of self-serving bias.
Okay, I understand that when a pharmaceutical company produces their new wonder-drug they have to test it and they have to pay to do that. This creates an understandable conflict of interest as they have invested lots of time and money in having a fantastic new effective treatment. If they declare this in the research paper and everything is transparent then I’m happy. However…
In the past I have worked with Doctors who have done research that has been sponsored by a Pharmaceutical Company, and when the study has shown no benefit the Pharmaceutical Company blocked the publication. That’s just downright dishonest.
What also gets up my nose is when I see a National or International guideline for the treatment of X, Y or Z and the conflicts of interest statement is almost as long as the guideline itself! For example, the 2015 European Society for Cardiology endocarditis guidelines are 42 pages long (excluding the 12 pages of references) and the conflict of interest paper for the authors is a whopping 35 pages! Now there may be no influence at all from the people/companies who gave money to the authors, and at least they are being transparent, but can people who have been paid really say it doesn’t influence their decisions? I find conflicts of interest very worrying….
So what research am I reading?
Firstly, I am being very careful about what I read and how I read it. I am looking at papers in detail and not just concentrating on the headlines. If I think an important study might be flawed, I take note of it and look to see if I can find any other evidence to either support or refute their conclusions. I also look to see if they are quoting their own or others research papers or if this “potentially flawed study” is being quoted again and again in other papers.
All the time I am keeping in mind the quote from Chris Hadfield the NASA Astronaut “there is no problem so bad that you cannot make it worse” to which I have added the reverse of an old saying so that it now says “don’t just do something, stand there”.
When I see something “new” I PAUSE, I evaluate it, consider it and look for further support of it and only if it continues to look correct then I start to use it… but then perhaps that’s good old fashioned paper reading practice anyway… but then I’m getting old and might be biased?!
Further information
How to read a paper; the basics of evidence-based medicine by Trisha Greenhalgh
I think this is an excellent book; a previous winner of the BMA book awards. It gives easy to understand ways to approach scientific literature and I think it should be essential reading for every health professional… and now that we’ve seen what happens in a pandemic I think every politician should also have to read it!
NB I have no affiliations with the book above or the research mentioned, my only conflict of interest is an association, as a volunteer fosterer, with Cats Protection.