SurgiSphere refuses to share data on which hydroxychloroquine study is based, forcing retraction of papers in Lancet and NEJM: bad science damages reputations of high impact journals.

Coronavirus studies by Engin Akyurt via pixabay.com

The study published in both Lancet and New England Journal of Medicine has been retracted because the primary source of the data in the study, Surgisphere, refuses to make the raw data available.  It turns out that the aggregated data is implausible for several reasons, not least of which is the large number of patients it covers.  Worst of all, the study failed to comply with previous agreements as to data and coding, nor to standard practices in statistics.  The following is extracted from a post on the blog RetractionWatch, updated June 4.

From Lancet: “They were unable to complete an independent audit of the data underpinning their analysis. As a result, they have concluded that they “can no longer vouch for the veracity of the primary data sources.” The Lancet takes issues of scientific integrity extremely seriously, and there are many outstanding questions about Surgisphere and the data that were allegedly included in this study.”  The problem being, as the retraction notice specifies, that Surgisphere refused to give the authors of the paper access to the raw data so that an independent audit could be performed.

From the retraction notice:

After publication of our Lancet Article, several concerns were raised with respect to the veracity of the data and analyses conducted by Surgisphere Corporation and its founder and our co-author, Sapan Desai, in our publication. We launched an independent thirdparty peer review of Surgisphere with the consent of Sapan Desai to evaluate the origination of the database elements, to confirm the completeness of the database, and to replicate the analyses presented in the paper.

Our independent peer reviewers informed us that Surgisphere would not transfer the full dataset, client contracts, and the full ISO audit report to their servers for analysis as such transfer would violate client agreements and confidentiality requirements. As such, our reviewers were not able to conduct an independent and private peer review and therefore notified us of their withdrawal from the peer-review process.

This does not mean that the raw data is necessarily wrong (or made up out of whole cloth), simply that it could not be independently analyzed.  Why would Surgisphere’s data be questioned?  Well, according to an open letter to Lancet: 

This scrutiny has led to both methodological and data integrity concerns. The main concerns are listed as follows:
1. There was inadequate adjustment for known and measured confounders (disease severity, temporal effects, site effects, dose used).
2. The authors have not adhered to standard practices in the machine learning and statistics community. They have not released their code or data. There is no data/code sharing and availability statement in the paper.  The Lancet was among the many signatories on the Wellcome statement on data sharing for COVID-19 studies.
3. There was no ethics review.
4.There was no mention of the countries or hospitals that contributed to the data source and no acknowledgments of their contributions.  A request to the authors for information on the contributing centres was denied.
5. Data from Australia are not compatible with government reports (too many cases for just five hospitals, more in-hospital deaths than had occurred in the entire country during the study period).
Surgisphere (the data company) have since stated this was an error of classification of one hospital from Asia.  This indicates the need for further error checking throughout the data base.
6. Data from Africa indicate that nearly 25% of all COVID-19 cases and 40% of all deaths in the continent occurred in Surgisphere-associated hospitals which had sophisticated electronic patient data recording, and patient monitoring able to detect and record “nonsustained [at least 6 secs] or sustained ventricular tachycardia or ventricular fibrillation”. Both the numbers of cases and deaths, and the details provided, seem unlikely.
7. Unusually small reported variation in baseline variables, interventions and outcomes between continents (Table S3).
8. Mean daily doses of hydroxychloroquine that are 100 mg higher than FDA recommendations, whereas 66% of the data are from North American hospitals.
9. Implausible ratios of chloroquine to hydroxychloroquine use in some continents.
For example, in Australia 49 received chloroquine and 50 received hydroxychloroquine.  However, chloroquine is not readily available in Australia and administration requires authorization from the Therapeutic Goods Administration.
10. The tight 95% confidence intervals reported for the hazard ratios appear inconsistent with the data. For instance, for the Australian data this would imply about double the numbers of recorded deaths as were reported in the paper.  The patient data were obtained through electronic health records, supply chain databases, and financial records. The data are held by the US company Surgisphere. In response to a request for the data Professor Mehra replied: “Our data sharing agreements with the various governments, countries and hospitals do not allow us to share data unfortunately.”

In other words, an implausibly large number of patients and opaque record-keeping made the data difficult to swallow.  The provenance of the data, Surgisphere, is also in question: an article in Wikipedia summarizes the skepticism that such a tiny company could have access to such a large database of patient records.   In conclusion, while the results of the study are entirely possibly correct, the data is extremely suspect for several reasons.  The subsequent publication of a paper which found no benefit for prophylactic use of hydroxychloroquine against SARS-COV-2 infection makes the conclusions of this paper more likely, but does not help us reach these conclusions with much confidence.

This is depressingly open-ended.  So far, we have not had any reliable studies showing any benefit from hydroxychloroquine.  This study, retracted, leaves us up in the air as to any possible benefits.  It also damages our confidence in the ability of these two widely respected journals to avoid publishing bad science.