Design, conduct, and analysis of biomedical and public health research form an interdependent continuum. Some specialties have more efficient mechanisms than others to optimise the design, conduct, and analysis of studies, providing the opportunity for different specialties to learn from successful approaches and avoid common pitfalls. The rapid introduction of new biological measurement methods involving genomes, gene products, biomarkers, and their interactions has promoted novel and complex analysis methods that are incompletely understood by many researchers and might have their own weaknesses. Additionally, biomedical and public health research increasingly interacts with many disciplines, using methods and collaborating with scientists from other sciences, such as economics, operational research, behavioural sciences, and informatics,1 heightening the need for careful study design, conduct, and analysis.
These issues are often related to misuse of statistical methods, which is accentuated by inadequate training in methods. For example, a study2 of reports published in 2001 showed that p values did not correspond to the given test statistics in 38% of articles published in Nature and 25% in the British Medical Journal. Prevalent conflicts of interest can also affect the design, analysis, and interpretation of results. Problems in study design go beyond statistical analysis, and are shown by the poor reproducibility of research. Researchers at Bayer3 could not replicate 43 of 67 oncological and cardiovascular findings reported in academic publications. Researchers at Amgen could not reproduce 47 of 53 landmark oncological findings for potential drug targets.4 The scientific reward system places insufficient emphasis on investigators doing rigorous studies and obtaining reproducible results.
Recommendations
- 1
Make publicly available the full protocols, analysis plans or sequence of analytical choices, and raw data for all designed and undertaken biomedical research
- •
Monitoring—proportion of reported studies with publicly available (ideally preregistered) protocol and analysis plans, and proportion with raw data and analytical algorithms publicly available within 6 months after publication of a study report
- 2
Maximise the effect-to-bias ratio in research through defensible design and conduct standards, a well trained methodological research workforce, continuing professional development, and involvement of non-conflicted stakeholders
- •
Monitoring—proportion of publications without conflicts of interest, as attested by declaration statements and then checked by reviewers; the proportion of publications with involvement of scientists who are methodologically well qualified is also important, but difficult to document
- 3
Reward (with funding, and academic or other recognition) reproducibility practices and reproducible research, and enable an efficient culture for replication of research
- •
Monitoring—proportion of research studies undergoing rigorous independent replication and reproducibility checks, and proportion replicated and reproduced
Problems related to research methodology are intricately linked to the training and composition of the scientific workforce, to the scientific environment, and to the reward system. We discuss the problems and suggest potential solutions from all these perspectives. We provide examples from randomised trials, traditional epidemiology studies, systematic reviews, genetic and molecular epidemiology studies, so-called omics, and animal studies. Further reading for each section is provided in the appendix.