Transparency in the reporting of in vivo pre-clinical pain research: The relevance and implications of the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelines

2.1 Consolidated standards of reporting trials CONSORT

Probably the best known example of clinical reporting standards is the Consolidated Standards of Reporting Trials CONSORT (www.consort-statement.org). CONSORT governs the style and minimum information sets for the reporting of randomised controlled clinical trials [2,3]. CONSORT provides a template for the reporting of clinical trials and consists of a twenty-five item checklist (design, analysis and interpretation of the trial). CONSORT also includes an informative flow diagram which displays the path which all subjects enrolled in the trial took at each of the four key stages of a trial (enrolment, intervention allocation, follow-up, and analysis). Over the past two decades CONSORT has had a major positive impact on the design, conduct and reporting of clinical trials and the ability to conduct robust systematic review and meta-analysis across all disciplines of medical research (www.consort-statement.org/about-consort/impact-of-consort).

CONSORT is also an example of the iterative nature of such standards, having been through a number of revisions since the original inception in 1993. Quite properly, it should now be nearly impossible to conduct and publish a clinical trial without adhering to the CONSORT ethos.

2.2 Preferred reporting items for systematic reviews and meta-analyses guidelines PRISMA

The example of CONSORT has been replicated in other clinical research disciplines, for example the reporting of systematic reviews and meta-analyses are governed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA (formerly QUOROM) – www.prisma-statement.org) [4] and for epidemiological studies MOOSE: Meta-analyses of observational studies in epidemiology (www.consort-statement.org/Initiatives/MOOSE).

2.3 SPIRIT, MIAME, MIAPE

Similarly, the SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials) sets out the standards for clinical trial protocols [5]. Minimum information sets have also become established in the culture of some areas of pre-clinical research, for example: Minimum Information About a Microarray Experiment (MIAME)(www.fged.org/projects/miame) [6] and Minimum Information About a Proteomics Experiment (MIAPE) (www.psidev.info/node/91) [7].

3.1 Responsibilities of editors and funders for transparent reporting of animal research

Scientific journals and their editors have a vital implementation duty in ensuring the adoption of transparent reporting criteria; initially by publishing the guidelines, often in co-ordination with other journals, and then by formally including, and enforcing, such guidelines in their instructions to authors. The Scandinavian Journal of Pain have recently taken a crucial step in this regard in becoming the first specialist pain journal to adopt the ARRIVE format [1] for the reporting of pre-clinical studies which use experimental animals (www.scandinavianjournalpain.com).

Since their original publication [1], the ARRIVE guidelines have been adopted by a steadily increasing number of journals, including the prestigious Nature and Public Library of Science (PLoS) families of journals (www.nc3rs.org.uk/ARRIVEjournals).

Funders of research also have a duty to ensure responsible and transparent dissemination of research that they fund. In the United Kingdom, for example, public sector (e.g., Medical Research Council and the Biotechnology and Biological Sciences Research Council) and charity (e.g., The Wellcome Trust) funders of research are amongst those endorsing ARRIVE. In May 2012 the heads of these funding bodies instructed the leaders of universities and research institutes at which they fund research that compliance with ARRIVE is now a condition of their funding (www.nc3rs.org.uk/ARRIVEfunders). In the United States the National Institute of Neurological Disorders and Stroke have recently drawn attention to the widespread deficiencies in current reporting standards and pressed the case for transparency in reporting of in vivo studies [8].

3.2 Reducing bias and improving quality of reporting animal research: need for ARRIVE

The need for robust reporting standards for pre-clinical pain research have been previously highlighted [9,10], although a suggested pro forma [9] was not adopted into the instructions for authors of pain journals. This pro forma has now been superseded by the publication, and wide adoption, of ARRIVE as a generic reporting guideline for animal studies across the spectrum of biomedical research. ARRIVE has its origins in the work of the United Kingdom National Centre for the Replacement, Refinement and Reduction of Animals in Research (www.nc3rs.org.uk). The stimulus for ARRIVE has its roots in a survey that revealed that most publications reporting animal research lacked key information on how the experiments were designed, conducted and analysed [11].

This survey revealed that essential information about bias reduction tactics were not included, with 87% of publications not stating adequate information about random allocation of animals to groups and 86% not reporting details regarding observer blinding. Similar figures have been revealed for pain research [9,10,12]. Given that a great deal of in vivo animal experimental work essentially consists of “clinical trials” of development compounds in non-human species, it is unsurprising that aspects of ARRIVE bear resemblance to CONSORT. Thus, ARRIVE consists of a 20 essential item checklist concerning aspects of title, abstract, introduction, methods, results and discussion. The methods section highlights details of bias reduction tactics such as sample size calculation, random allocation to groups and observer blinding.

It is important to note that ARRIVE are intended as generic reporting guidelines to provide a standard format for authors to indicate to reviewers, editors and readers exactly what they did and how they did it. Whilst, the aim of ARRIVE is to improve the reporting of animal research, it is explicitly not intended as a rigid dictat for experimental design. Nevertheless, the checklist does de facto serve as a useful aide mémoire in the design of animal experiments, as highlighted by the UK funding agencies in their letter referred to above.

3.3 Impact of ARRIVE and “Good Laboratory Practice”

We cannot be certain of the impact of experimental bias in the results of the existing pre-clinical pain literature because of the poor standards of reporting [9,10,12]. Widespread adoption of ARRIVE by journals which publish pain research will, in time, permit meta-analyses to estimate the impact of bias in, for example, the overestimation of efficacy of drugs being developed.

In the meantime, we can learn and apply the lessons from other closely related fields such as stroke: Macleod, Sena and colleagues in the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (CAMARADES) consortium (www.camarades.info) have conducted ground-breaking work in demonstrating the clear detrimental impact of experimental bias in overestimation of efficacy in experimental stroke research [13,14,15,16,17,18,19,20]. Exploiting this information they have drawn up a code of “Good Laboratory Practice” intended to reduce the impact of bias in the design, conduct, and reporting of animal experiments modelling human stroke [20], which could easily be adopted for pain research. The essential domains of “Good Laboratory Practice” relevant to pain research are set out in Table 1, although it is accepted that the term “Good Laboratory Practice” does have alternative meanings in other fields. Sena et al. have demonstrated the cumulative impact of inadequate methodological quality in each of these domains in overestimating efficacy of experimental compounds [16].

3.4 Other areas where the predictive validity of preclinical pain research can be improved

Although implementation of robust experimental bias reduction methods and transparent reporting thereof, for in vivo pain studies is one of the easiest changes to implement, there are several other challenging aspects of animal models that need to be addressed in order to improve their overall relevance to human painful disease. Top of this list is establishing a portfolio of conditions in rodents which, to that degree which is possible, reflect the range of clinical states where pain is a feature and the development of validated, ethologically relevant, outcome measures which reflect human pain-related clinical signs [9,10,21,22,23,24,25,26].

There are also other aspects which will be even more challenging to address, such as replicating the temporal aspects of chronic disease, accounting for co-morbidity seen in patients, heterogeneity of clinical presentations, predicting adverse effects and pharmacokinetic variables of drugs and a better reflection of human “pain” outcome incidence for various diseases.

The usual choice of healthy, young, male, genetically similar rodents for modelling the complexities of human chronic pain is questionable.

3.5 Biased animal research: ethical and societal implications

Returning to the question of pre-clinical bias reduction methods, and the transparent reporting thereof: the impact of methodologically flawed animal studies is not trivial nor an issue for mere academic cogitation and jousting, but has wide ethical and societal implications. Some examples: firstly, animal “pre-clinical trials” are often used to justify the conduct of early stage human clinical trials; allowing inefficacious medications through this barrier exposes the humans participating in such trials to unnecessary risk. Indeed, such prediction of efficacy is a justification for the large scale use of animals in drug development and poorly designed animal studies are probably the cause of much wasted time and money in the pharmaceutical industry. Only 37% of highly cited animal research is translated at the level of human randomised trials [27]. Secondly, the use of animals in studies which do not have the methodological capability to adequately test the hypothesis is unethical. Thirdly, the use of scarce and valuable research funding for poorly designed studies is unjustifiable. Finally, inadequate reporting of trial methods prevents adequate systematic review and meta-analysis of the literature, which in the clinical domain has revolutionised how we critically examine evidence.

3.6 Caveats, reevaluating metrics of success and future directions

There are important caveats to unquestioningly accepting the premise of the universal adoption of robust bias reduction tools and reporting transparency into pre-clinical research. A potential consequence of this approach is that less “positive” or “breakthrough” pre-clinical papers will be published. Of course, this artificial division of the literature into “positive” and “negative” is a false dichotomy and arguably a paper reporting robustly conducted studies in which the hypothesis was not proven, is of greater worth to the evidence base than an inadequately designed and conducted “positive” experiment. Nevertheless, metrics of success would need to be re-examined: in academia, publication record, especially of “trophy” papers, is a major criterion for success in professional performance appraisal, resource allocation, personal remuneration, grant funding, job security, and promotion.

Indeed, this judgement extends beyond the single academic to the overall assessment of departments and universities. For instance, in the United Kingdom the Research Excellence Framework (www.ref.ac.uk) is the main instrument by which state university funding is determined. A subjective value judgement of the best four publications of individual academics in a specific time window accounts for 65% of the overall REF score.

At the current time, academic success is rarely assessed beyond the number of publications, the citation index, the perceived standing/impact factor of the journals concerned and the grant funding track record. Attention has been drawn to the dangers, despite the inevitable attention which they attract, of overemphasising the importance of early, as yet un-reproduced, pre-clinical studies with small sample sizes reported in high impact factor journals [28,34]. If methodologically flawed animal based data are highly rated in this process it would introduce a systemic error into the assessment.

Similar perverse incentives exist in industry, which extend beyond performance appraisal of individual R&D workers, with an apparently promising pre-clinical pipeline contributing to the stock value and survival of companies. When ARRIVE is widely adopted, then it is essential that academia, and industry, seize the opportunity that transparent reporting provides; namely the ability to systemically appraise reports for the likelihood of methodological flaws and the potential for bias confounding the results of an experiment. There is also a need to revise professional performance criteria to include longer term metrics, such as the eventual confirmation of pre-clinical discoveries in randomised controlled clinical trials and a demonstrable change in clinical practice. There are promising developments on the horizon, for the first time the 2014 REF will assess, alongside publication output, the long term impact of research-although this only amounts to 20% of the overall assessment. The dangers of not measuring the long term impact of research has obvious parallels with the distortions inherent in the short term bonus culture for performance reward in the financial and management sectors.

Transparency of reporting and elimination of bias are relatively easy to deal with compared with some other issues which are facing pre-clinical research and which the clinical evidence field is already addressing.

Firstly, direct replication of important pre-clinical discoveries is rarely undertaken and attempts to document the reproducibility of high profile papers has not been reassuring [29,30,34]. Transparency of reporting will facilitate replication and there is a need to recognise the value of replication studies by funding agencies; encouragingly replication is already becoming part of the culture in genetic research.

Secondly, holistic assessment by systematic review of an evidence base requires unrestricted access to all experimental data, published and unpublished, “positive” and “negative”. The confounding problem of publication bias is well recognised in clinical circles and increasingly so for the pre-clinical literature not least to prevent unnecessary repetition of previous experiments [18,31,32,33]. Linked to this is the importance of publishing all data gathered in an experiment and the avoidance of selective publication of results. The clinical field is addressing these issues by prospective registration of clinical trial protocols in publically accessible databases, linked to portals for publication of results that may not be attractive to mainstream peer review journals. However, the challenges of introducing and enforcing such an ethos to pre-clinical in vivo research are Herculean.