Medicina Intensiva  (English Edition) Medicina Intensiva  (English Edition)
Med Intensiva 2018;42:180-3 - Vol. 42 Num.3 DOI: 10.1016/j.medine.2017.12.007
Series in Intensive Medicine: Methodological update in Medicina Intensiva
Methodological update in Medicina Intensiva
Actualización en metodología en Medicina Intensiva
J.L. García Garmendia
Servicio de Cuidados Críticos y Urgencias, Hospital San Juan de Dios del Aljarafea, Bormujos, Sevilla, Spain
Received 13 December 2017, Accepted 15 December 2017
Abstract

Research in the critically ill is complex by the heterogeneity of patients, the difficulties to achieve representative sample sizes and the number of variables simultaneously involved. However, the quantity and quality of records is high as well as the relevance of the variables used, such as survival. The methodological tools have evolved to offering new perspectives and analysis models that allow extracting relevant information from the data that accompanies the critically ill patient. The need for training in methodology and interpretation of results is an important challenge for the intensivists who wish to be updated on the research developments and clinical advances in Intensive Medicine.

Resumen

La investigación en el enfermo crítico es compleja por la heterogeneidad de los pacientes, por las dificultades para alcanzar tamaños de muestra representativos y por la cantidad de variables que intervienen de manera simultánea. Sin embargo, se beneficia de la cantidad y calidad de registros, así como de la relevancia de las variables utilizadas, como la supervivencia. Las herramientas metodológicas han evolucionado ofreciendo nuevas perspectivas y modelos de análisis que permiten extraer información relevante de la riqueza de datos que acompaña al enfermo crítico. La necesidad de formación en metodología y en interpretación de resultados constituye un importante reto para los intensivistas que deseen estar al día en las líneas de investigación y en los nuevos avances de la Medicina Intensiva.

Keywords
Critically ill patient, Methodology, Statistical analysis, Statistical interpretation, Peer review, Methodological errors
Palabras clave
Paciente crítico, Metodología, Análisis estadísticos, Interpretación estadística, Revisión por pares, Errores metodológicos

The series of methodology we start in Medicina Intensiva tries to bring the clinician closer to different methodological issues that we have considered relevant in the research setting of critically ill patients.

Critical disease has some special conditions when it comes to generating scientific knowledge.1 On the one hand, the amount of monitoring and the availability of tests in these patients provides a great deal of analyzable information. Also, the high impact of physiopathological aspects allows us to go deep into applied basic research. Due to the patients’ fast progression and rates of survival, variable mortality is commonly used, generating high interest and clinical relevance to clinical researches. On the other hand, the velocity of changes, complexity, and interrelation of variables and simultaneous treatments, working schedules, and difficulties trying to find homogeneous enough patients in one single center elevates the requirements of study designs, and resources needed.2

All these challenges lead to two (2) fundamental issues. On the one hand, the need to find tools for statistical analysis adjusted to data, and possible designs with critically ill patients, which is why new and diverse approaches are needed. On the other hand, it is important to improve methodological training for research purposes and also for the interpretation of results. In this series we do not wish to write a statistical treaty, but draw an interesting picture including general aspects of result interpretation as advanced topics among which we will find new strategies of analysis, or big data.

In one comparative analysis of the statistical techniques used in 3 journals dedicated to the critically ill patient (Medicina Intensiva, Intensive Care Medicine, and Critical Care Medicine), we analyzed the methodological procedures used in one recent sample of the original manuscripts published during 2016 and 2017 (Table 1). Other than the differences seen in the number of original manuscripts published per issue, and in the presence of Spanish authors in the three (3) journals, we found significant differences in the type of studies conducted and in the complexity of the methodology used. Although the three journals mostly use observational studies, compared to the other two journals, Intensive Care Medicine includes a higher percentage of clinical trials, being such trials rare in Medicina Intensiva (4.6%). In Medicina Intensiva, the percentage of multicenter studies is lower than in the other two journals, and there are not too many estimates of the size of the sample either, probably due to the fact that fewer clinical trials are published.

Table 1.

Comparative analysis of the statistical methodology used in the original manuscripts published in three (3) journals within the intensive care setting.

  Medicina Intensiva  Intensive Care Medicine  Critical Care Medicine  p 
  n=65  n=67  n=80   
Period analyzed  2016–2017  2016  January–May 2016   
Issues studied (n)  18  12   
Originals per issue, average (SD)  3.6 (0.8)a,b  5.6 (1.6)b,c  16 (2.8)c  p<0.001 
Impact factor, 2016  1231  12,015  7050  Non-applicable 
Originals with Spanish authors, n (%)  56 (86.2)a,b  5 (7.5)c  6 (7.5)c  p<0.001 
Type of study, n (%)  46 (70.8)a  34 (50.7)b,c    p<0.001 
Observational      57 (71.3)   
Quasi-experimental  8 (12.3)  4 (6.0)  2 (2.5)   
Clinical trial  3 (4.6)a  21 (31.3)b,c  13 (16.3)   
Diagnostic test  8 (12.3)  8 (11.9)  5 (6.3)   
Simulation  0 (0)  0 (0)  3 (3.8)   
Multicenter, n (%)  22 (33.8)a,b  38 (56.7)c  46 (57.5)c  p=0.007 
Analysis of mortality, n (%)  27 (41.5)  41 (61.2)  47 (58.8)  p=0.045 
Estimate of the sample size, n (%)  5 (7.7)a,b  28 (41.8)c  20 (25.0)c  p<0.001 
Statistical analyses
Comparison of percentages, n (%)  52 (80)  56 (83.6)  57 (71.3)  p=0.176 
Comparison of quantitative variables, n (%)  55 (84.6)  55 (82.1)  60 (75)  p=0.315 
Comparison of repeated measurements, n (%)  7 (10.8)  8 (11.9)  9 (11.3)  p=0.977 
Kaplan–Meier, n (%)  11 (16.9)  15 (22.4)  13 (16.3)  p=0.591 
Simple regression, n (%)  1 (1.5)  4 (6.0)  7 (8.8)  p=0.173 
Multiple linear regression, n (%)  1 (1.5)b  7 (10.4)  13 (16.3)c  p=0.013 
Logistics regression, n (%)  26 (40.0)  27 (40.3)  38 (47.5)  p=0.577 
Cox regression, n (%)  7 (10.8)  15 (22.4)  12 (15.0)  p=0.182 
Propensity score, n (%)  1 (1.5)  4 (6.0)  7 (8.8)  p=0.173 
ROC curves, n (%)  12 (18.5)  9 (13.4)  14 (17.5)  p=0.706 
Correlation analysis, n (%)  5 (7.7)  5 (7.5)  11 (13.8)  p=0.345 
Concordance analysis, n (%)  5 (7.7)  3 (4.5)  5 (6.3)  p=0.743 
Other tests,d n (%)  1 (1.5)a,b  23 (34.3)c  21 (26.3)c  p<0.001 
Number of tests per original manuscript, media (SD)  2.9 (1.2)a,b  3.5 (1.4)c  3.5 (1.4)c  p=0.007 

SD: standard deviation. ROC: receiver operating characteristics.

a

p<0.05 (Bonferroni correction) versus Intensive Care Medicine.

b

p<0.05 (Bonferroni correction) versus Critical Care Medicine.

c

p<0.05 (Bonferroni correction) versus Medicina Intensiva.

d

Methods used: Bayesian analysis (1), qualitative analysis (1), clusters analysis (2), analysis of the main components (2), multiple concordance analysis (1), mediation analysis (1), factorial analysis (1), fractal analysis (1), decision tree (1), generalized estimating equations (2), phi statistics (1), gradient boosted machine (1), G-Study (1), Inverse probability treatment weighting (IPWT) (2), jointpoint regression (1), K-nearest neighbors (1), Locally weighted scatterplot smoothing (LOWESS) (2), Modified early warning systems (MEWS) (1), negative binomial model (1), generalized linear model (12), generalized mixed linear model (2), structural marginal model (4), simulation models (2), Montecarlo (2), network maps (1), neural network (1), Cochran-Armitage trend test (2), random forest (1), binomial regression (1), Poisson regression (5), competitive risks (2), temporal series (4), support vector machine (1) and Cuzick's test (1).

On the statistical analyses, the number of tests used is much lower in Medicina Intensiva than it is in the other two journals, although in the most common methods we did not find any significant differences, except in the multiple linear regression analysis. Where we did find a clear difference is in the use of more advanced statistical methods, almost inexistent in the original manuscripts from Medicina Intensiva, but present in one third of the originals from Intensive Care Medicine and Critical Care Medicine. The greater use of advanced tests has become a common practice and is a clear improvement compared to similar assessments made in high-impact factor journals some 20 years ago.3

In sum, we found a significant difference in the level of design complexity and statistical analysis in the manuscripts submitted to Medicina Intensiva compared to the other two (2) journals.

On the other hand, in one review conducted on a sample of 62 original manuscripts submitted for publication to Medicina Intensiva from 2016 through 2017, we found a series of methodological problems that are shown in Table 2 being the most common ones, per frequency of appearance, the inadequate definition of variables, the mistakes made in the description of results, the fact that conclusions did not match the results, the lack of statistical analysis (multivariate models, above all), the existence of not very clear designs, the inadequate presentation of the goal, selection biases, and the use of inadequate statistical tests for analysis. These problems have also been found in other publications4 and, in this sense, the use of reviewers trained in statistics during the review process has proven a useful tool to improve the quality of the articles published5 and implies some sort of ethical commitment.6

Table 2.

Main methodological problems found in the first review of original manuscripts submitted to Medicina Intensiva from 2016 through 2017.

Methodological problems  Frequency 
Goal
Poorly expressed goal  21.0% 
Variables
Poorly defined variables  48.4% 
Lack of variables of interest  9.7% 
Design
Poorly defined design  22.6% 
Inadequate design  6.5% 
Inadequate definition of the study type  6.5% 
Target population
Poorly defined population  17.7% 
Old study  3.2% 
Period of study missing  1.6% 
Sample
Selection bias  21.0% 
Insufficient sample size  17.7% 
Observer bias  8.1% 
Size of the sample not estimated  8.1% 
Sample error  1.6% 
Lack of flow chart  1.6% 
Results
Descriptive with errors  41.9% 
Lack of statistical analysis  25.8% 
Inadequate statistical test  21.0% 
Poorly used interquartile rangea  19.4% 
Contradictory data  11.3% 
Lack of comparisons of interest  11.3% 
Lack of standardized mortality ratio  8.1% 
Lack of confidence intervals  4.8% 
Inadequate subgroup analysis  3.2% 
Unnecessary comparisons  3.2% 
Interpretation
Interpretation of non-significant resultsb  16.1% 
Errors interpreting the analysis  9.7% 
Conclusions
Conclusions not matching the results  27.4% 
Lack of conclusions  3.2% 

Number of originals analyzed=62.

a

The interquartile range (IQR) is the difference between percentiles 75th and 25th.

b

Erroneous interpretation of non-significant differences: “Differences are found, but they happen to be non-significant”.

It is, therefore, evident that there is margin for improvement in methodological design, and in the analysis and interpretation of results in all research studies submitted to our journal. The Editorial Committee of Medicina Intensiva, directed by Dr. Garnacho Montero, understands that this journal needs to draw and promote studies of high methodological quality that, in time, will improve the impact factor and promote the submission of new quality studies.7 Within this virtuous circle setting, this series is born in an attempt to improve the quality of our investigations.

This series begins with a general reflection on the particularities that affect the studies of critically ill patients, and we will be discussing the difficulties, different approaches, and new strategies defined to make scientific knowledge move forward. Also, we will be analyzing the interpretation of statistical results in its most basic and advanced aspects, trying to facilitate an adequate understanding of the terminology used in the most common scientific literature. This paper is a review of common methodological errors made in clinical research, some of which can be more or less innocent, but all will undoubtedly help the reader keep a critical view of the information presented here. We will also be reviewing a fundamental aspect known as the bioethics of clinical research–very important for the editorial team,7 a historical summary will be presented here, followed by an analysis of the different ethical requirements that are mandatory in the clinical research of critically ill patients.

When we open up the arsenal of statistical tools available, three (3) reviews can be conducted: one review on the techniques used to determine the causality of observational studies with an emphasis on propensity score analysis; a second review on new techniques of alternative analysis such as hybrid studies, nested case–control studies, recursive partitions, or competitive events, one article discussing the development of systematic reviews, and meta-analyses as sources of knowledge. Lastly, we have addressed a popular issue these days by including an update on big data–the tool that will probably revolutionize the way we understand the clinical information available, and, ultimately, condition the way we make decisions.

We hope that the readers of Medicina Intensiva find this series of methodology interesting, and a source of inspiration to keep moving forward in the fascinating world of clinical research in the critically ill patient.

Conflict of interest

The authors declare no conflict of interests associated with this article whatsoever.

References
1
J.L. García Garmendia
Conceptos de bioestadística en el paciente crítico
Manual de Medicina Intensiva, 5.ª ed.,
2
N. Pattison,N. Arulkumaran,S. Humphreys,T. Walsh
Exploring obstacles to critical care trials in the UK: a qualitative investigation
J Intensive Care Soc, 18 (2017), pp. 36-46 http://dx.doi.org/10.1177/1751143716663749
3
L.C. Silva Ayçaguer,C. Pérez Nieves,I. Cuellar Wong
The use of statistical methods in 2 medical journals with high impact factor
Gac Sanit, 9 (1995), pp. 189-195
4
S. Fernandes-Taylor,J.K. Hyun,R.N. Reeder,A.H. Harris
Common statistical and research design problems in manuscripts submitted to high-impact medical journals
BMC Res Notes, 4 (2011), pp. 304 http://dx.doi.org/10.1186/1756-0500-4-304
5
E. Cobo,A. Selva-O’Callagham,J.M. Ribera,F. Cardellach,R. Dominguez,M. Vilardell
Statistical reviewers improve reporting in biomedical articles: a randomized trial
6
D. Poole,G. Nattino,G. Bertolini
Overoptimism in the interpretation of statistics: the ethical role of statistical reviewers in medical journals
Intensive Care Med, 40 (2014), pp. 1927-1929 http://dx.doi.org/10.1007/s00134-014-3510-6
7
S.R. Leal-Noval,R. Amaya-Villar,J.L. García-Garmendia,F. Gordo-Vidal,J. Garnacho-Montero
Editorial policy of Intensive Medicine
Med Intensiva, 41 (2017), pp. 63-66 http://dx.doi.org/10.1016/j.medin.2017.02.001

Please cite this article as: García Garmendia JL. Actualización en metodología en Medicina Intensiva. Med Intensiva. 2018;42:180–183.

Copyright © 2018. Elsevier España, S.L.U. and SEMICYUC
Med Intensiva 2018;42:180-3 - Vol. 42 Num.3 DOI: 10.1016/j.medine.2017.12.007
Cookies Policy
x
To improve our services and products, we use cookies (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here.