Meta-research metrics matter: letter regarding article “indirect tolerability comparison of Deutetrabenazine and Tetrabenazine for Huntington disease”
Journal of Clinical Movement Disorders volume 4, Article number: 19 (2017)
Here we discuss the report by Claassen and colleagues describing an indirect treatment comparison between tetrabenazine and deutetrabenazine for chorea in Huntington’s disease using individual patient data. We note the potential for discrepancies in apparently statistically significant findings, due to the rank reversal phenomenon. We provide some cautionary observations and suggestions concerning the limitations of indirect comparisons and the low likelihood that good quality evidence will become available to guide clinical decision comparing these two agents.
To the Editor,
We read with interest the report by Claassen and colleagues describing an indirect treatment comparison between tetrabenazine (TBZ) and deutetrabenazine (DEU) for chorea in Huntington’s disease (HD) using individual patient data .
DEU is a form of TBZ, chemically-modified to optimize its pharmacokinetic properties. Both are vesicular monoamine transporter 2 (VMAT2) inhibitors and each was tested successfully against placebo for chorea associated with HD [2, 3]. No double-blinded head-to-head comparison has been performed or is planned to compare the efficacy and safety profiles of these compounds.
Indirect treatment comparisons are useful meta-research tools when little or no data directly comparing treatment are available . In meta-research, the use of individual patient data instead of aggregate data has many potential advantages, such as the power to study subgroups and to control for confounding factors. To some extent Claassen et al. used individual patient data, since raw patient data from the FIRST-HD trial testing DEU was incorporated into the analysis. This partially overcomes possible reporting bias from the literature, such as adverse events frequencies not recorded in primary reports, and provides the opportunity to use more complex and complete statistical models adjusted to important covariates. It is a shame that individual patient data from the TETRA-HD trial  were not included, especially since both trials were performed by the same study consortium (the Huntington Study Group). In addition, it would have been both possible and interesting to undertake exploratory analyses to find out whether subgroups of patients with different genders, CAG repeat lengths, baseline levels of functional ability, motor symptoms and quality of life differed in regards to safety profile.
We also think it unfortunate that the authors limited their report to safety data, while it would be extremely relevant to learn how the efficacy profiles of TBZ and DEU compared using the individual patient data available to them. We recently performed an indirect treatment comparison using all published aggregate data from the same trials, and found no difference between the primary efficacy outcomes of both trials: the total maximal chorea score change from baseline mean difference was between TBZ and DEU was −1.00 (95% confidence interval: −3.04 to 1.04) .
Interestingly, in contrast to Claasen and co-authors’ findings, our safety outcomes did not demonstrate any difference between odds ratios of TBZ over DEU (Table 1). However, when converting our raw dataset to risk differences instead of odds ratios, as presented by Claassen et al., our results were in line with theirs, in regards to direction and magnitude of effect. This is a well-described statistical phenomenon called rank reversal. It stems from the fact that different measures (e.g. risk differences, risk ratios, and odds ratios) are affected differently by dissimilar baseline risks . Bucher’s model of indirect treatment comparisons was originally designed for odds ratios, but others have applied it to risk ratios and risk differences, with proper adjustment.
When indirectly comparing treatments, the choice of presented metric matters. We believe it is important for readers to be made aware of the potential for apparently discrepant findings that may arise from the rank reversal phenomenon . Undoubtedly, risk differences increase interpretability, but odds ratios are the only measure that guarantees the avoidance of impossible predicted event rates when extrapolating the results for real populations (for example, applying a risk difference of 0.1 to a population with a risk of 0.05 would give rise to an apparent population risk with a value less than zero, which is implausible) and relative measures are known to be more consistent than absolute measures [7,8,9].
Choice of data and outcomes aside, an overarching concern is that none of the included clinical trials has been appropriately powered to investigate the safety of these compounds, and an indirect treatment comparison of one trial per agent (i.e. TBZ versus placebo, and DEU versus placebo), cannot improve the precision of the results. Therefore, the results of these comparisons [1, 5] should be interpreted with caution; and although regarded as best available evidence, they are nonetheless of low quality according to the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework .
Given all this, only a direct comparison between TBZ and DEU would be able to rigorously test whether the efficacy profiles of these compounds significantly differ, and our sample size calculation (over 600 participants) suggests that such a trial is unlikely to take place . Further post-authorisation safety studies or other observational studies will be required to provide robust evidence on the safety profile of DEU to inform safe prescribing decisions.
Grading of Recommendations Assessment, Development and Evaluation
vesicular monoamine transporter 2
Claassen DO, Carroll B, De Boer LM, Wu E, Ayyagari R, Gandhi S, Stamler D. Indirect tolerability comparison of Deutetrabenazine and Tetrabenazine for Huntington disease. J Clin Mov Disord. 2017;4:3.
Huntington Study Group. Tetrabenazine as antichorea therapy in Huntington disease: a randomized controlled trial. Neurology. 2006;66(3):366–72.
Huntington Study Group. Effect of Deutetrabenazine on chorea among patients with Huntington disease: a randomized clinical trial. JAMA. 2016;316(1):40–50.
Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50(6):683–91.
Rodrigues FB, Duarte GS, Costa J, Ferreira JJ, Wild EJ: Tetrabenazine versus deutetrabenazine for Huntington's disease: twins or distant cousins? Mov Disord Clin Pract 2017:n/a-n/a.
Norton EC, Miller MM, Wang JJ, Coyne K, Kleinman LC. Rank reversal in indirect comparisons. Value Health. 2012;15(8):1137–40.
Walter SD. Choice of effect measure for epidemiological data. J Clin Epidemiol. 2000;53(9):931–9.
Deeks JJ. Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes. Stat Med. 2002;21(11):1575–600.
Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Stat Med. 2000;19(13):1707–28.
Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the journal of clinical epidemiology. J Clin Epidemiol. 2011;64(4):380–2.
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
All the authors of this manuscript performed and published an aggregate data indirect treatment comparison between tetrabenazine and deutetrabenazide for chorea in Huntington’s disease. The authors declare that they have no other competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rodrigues, F.B., Duarte, G.S., Costa, J. et al. Meta-research metrics matter: letter regarding article “indirect tolerability comparison of Deutetrabenazine and Tetrabenazine for Huntington disease”. J Clin Mov Disord 4, 19 (2017). https://doi.org/10.1186/s40734-017-0067-x