I saw this on RenalWeb and then read the abstract, but I can’t access the whole thing because I’m a mere patient and have to pay for information that could save my life. I was wondering if you could access it, read it and then comment because the abstract seems to fly in the face of all we have come to believe. Thank you.
http://www.nature.com/nrneph/journal/vaop/ncurrent/pdf/nrneph.2011.140.pdf
Hi Moosemom. I can’t access this article either. Nature isn’t a journal we get. But, knowing Peter Kerr, who, like Dr. Agar, is an Aussie, I’d be VERY surprised if the answer to the question he asks in the title isn’t a resounding YES. One “negative” study that wasn’t able to recruit enough people does not negate the decades of science and common sense that have established the benefits of nocturnal HD.
When the FHN trial was first started, the plan was to do two randomized, controlled studies. One compared IN-CENTER short daily (5-6 days a week) with IN-CENTER standard HD. This study was able to recruit enough people. Its results were resoundingly positive for each measure the researchers looked at.
The other study was intended to compare nocturnal home HD with standard in-center HD. But, there was a problem. Once people were educated about the options , they refused to be randomized to standard in-center HD! They wanted the better treatment. The study protocol had to be changed to nocturnal home HD vs. standard home HD (3x/week). Even then, the researchers were still not able to get enough people to take part. And, the differences between the two groups was not big enough to be statistically significant–though they did trend in the right direction.
The thing is, a paper by Saner has compared folks doing standard in-center HD with those doing standard HOME HD. Those on home HD lived about twice as long–even though they were doing the same amount of treatment. So, just the fact that both groups were at home means that there was less difference between them. And, I suspect that, once home, the “standard HD” folks–already educated about the benefits of MORE dialysis–may have dialyzed longer than they were supposed to for the study, or even did every other day. This would further reduce the differences between the groups and lead to a “negative” trial.
So, it’s important to take these study results with a grain of salt. Failure to find a significant difference doesn’t mean that there isn’t one.
There is a Peter G Kerr nephrologist in the Melborne area, is he who wrote this? I’d be very interested in your thoughts Dr. Agar.
Thanks for your reply, Dori. Since I am not able to access the entire article, I don’t want to make assumptions about its contents. And there are plenty of studies that I’ve come across that I take with a grain of salt or greet with a resounding, “Well, duh.” (There was one study that postulated that the anxiety and depression suffered by most CKD patients just may interfere with their sex lives. Well, duh.)
NHD critique in Nature Reviews reviewed
I have been asked to pass comment on a review by Peter Kerr to be found in the pre-publication electronic release of Nature Reviews: Nephrology.
It goes by the title “Nocturnal haemodialysis – should we still provide this therapy?”
Peter is a particularly close personal friend, a much-valued colleague and a like-minded dialysis expert … though I fancy he is more the expert while I am more the evangelist – both for home haemodialysis. He is the head of one of Australia’s largest renal services and a staunch supporter and proponent of home-based care and extended hour and frequency home dialysis – as am I – and, in this country, that means nocturnal home haemodialysis. He oversees one of the largest nocturnal programs in our country.
I had actually been asked to write this review but was, at the time, on vacation in Canada and the request clearly passed to Peter. I am glad it did. He has written almost exactly as I would have written – though he has likely been more eloquent than I could have been – and, as such, I am thrilled to read his critique and agree with it, lock stock and barrel.
Here I am inserting a direct quote from Dori Schatell’s previous post (I hope Dori won’t mind) … as it encapsulates what I will then make more detailed comment on …[B]
[I]When the FHN trial was first started, the plan was to do two randomized, controlled studies. One compared IN-CENTER short daily (5-6 days a week) with IN-CENTER standard HD. This study was able to recruit enough people. Its results were resoundingly positive for each measure the researchers looked at.
The other study was intended to compare nocturnal home HD with standard in-center HD. But, there was a problem. Once people were educated about the options , they refused to be randomized to standard in-center HD! They wanted the better treatment. The study protocol had to be changed to nocturnal home HD vs. standard home HD (3x/week). Even then, the researchers were still not able to get enough people to take part. And, the differences between the two groups was not big enough to be statistically significant–though they did trend in the right direction.
The thing is, a paper by Saner has compared folks doing standard in-center HD with those doing standard HOME HD. Those on home HD lived about twice as long–even though they were doing the same amount of treatment. So, just the fact that both groups were at home means that there was less difference between them. And, I suspect that, once home, the “standard HD” folks–already educated about the benefits of MORE dialysis–may have dialyzed longer than they were supposed to for the study, or even did every other day. This would further reduce the differences between the groups and lead to a “negative” trial.[/I][/B]
The brief from Nature Reviews was to comment on the recent paper arising from the Frequent Hemodialysis Network (FHN) group, led by Dr Rocco, which compared the outcomes of frequent nocturnal haemodialysis with conventional haemodialysis. The trial was reported as a negative trial – but this was, in our view, rather misleading for a number of reasons. Nature Reviews – if my memory serves me correctly from the approach to me – set the title. There was no offered opportunity to add a ‘YES’ to the end of their pre-formed review title. In addition, the abstract does not convey any of the caveats to the FHN trial conclusions that many believe should have been given more emphasis. The body of Peter’s text nicely lists and discusses the many short-comings of the trial.
Statistics can be and often are misleading – even those produced from the hallowed ‘RCT’ … and my personal views on the certainty of the ‘RCT’ as the gold-standard for all trials and decisions that arise from trials can be found in a paper I wrote in NNI in 2008.
Some (myself included) believe that the frequent nocturnal arm of the FHN trial should not have been reported – certainly not as a ‘negative’ trial – and if reported, it should only have been regarded as a ‘mis-trial’ or as a ‘null-trial’. (And here, as an aside to Dori - maybe you could ask Carl Kjellstrand to comment on this comment!)
I should start out by saying that I should (and do) congratulate the FHN trial team for at least trying! They tried to do what has often been said couldn’t be done … doing an RCT of a major lifestyle intervention. BUT … the reason it has been so hard to do is that is almost impossible to take patients and randomise to an in-centre treatment (with shorter time, travel, institutionalisation … all the ‘goods’ and ‘bads’ of a centre program) or to home (with longer time, isolation, training time, home comforts … all the ‘goods’ and ‘bads’ of a home program) - and then expect the two to be comparable groups. This will always be the ‘flaw’ in life-style RCTs.
Dori correctly pointed out Saner’s paper - but the critics of this paper will always raise (correctly) the issue of ‘referral’ or ‘placement’ bias … that the ‘best’ patients go home and thus might be ‘expected’ to do better. This criticism of the results of home therapies is always hard to toss … and is true, to an extent. It muddies the water of every attempt to prove this is not the only reason for better home outcomes. Rocco’s effort with the FHN trial was laudable - but then failed to reach the power needed to ‘conclude’ - one way or the other though the ‘trend’ … and they were strong trends … was always to the NHD arm being the better outcome.
Sometimes its nice to mutter 'damn the statistics, I’ll follow my gut conclusions …", and just go home!
Now … from here on in, I will roughly paraphrase and/or partially quote Peter Kerr’s paper for you (in lay terms and as best I can) … I am sure he would not mind me doing so …
The trial set out to enroll 250 patients. The investigators had great trouble in reaching the numbers required to allow a likely statistical outcome to be achieved (in ‘stats’ terms, this is referred to as the power of a study).
The FHN trial of frequent (6x) NHD v conventional (3x) CHD trial was, in the end, quite severely and unfairly underpowered - and this applied to both ‘arms’ of the trial - the NHD arm and the CHD arm.
Only 87 patients in all could be recruited. Thsi is less than 35% of the number required to reach a statistical power to answer the 3 end-point questions of the trial … and these were to determine if was there a statistical difference between the NHD and CHD groups and did NHD show
Furthermore, at 12 months … a very short time to compare any dialysis-related outcome … of the already sadly underpowered 87 patients recruited, only 76 had data points for left ventricular mass!
As only 3 deaths occurred in the study period, it is very hard to make a statistical argument (positive or negative) from a very small number of deaths within a similarly small number of total patients. Yet a conclusion was made that as there was no statistical difference, that one therapy could not be supported above the other on the basis of survival … ie: it was reported as a negative trial for survival benefit for NHD.
Left ventricular mass did fall (this should be regarded a good outcome) – but it only reached a ‘p’ value – a level of significance – of 0.095. The confidence levels were -23.7 to +1.8.
What do these stats mean? … well, to be ‘significant’, the ‘p’ value should be less than 0.05 … or a more than 95% chance that the difference between the two groups is significant. A ‘p’ value of 0.095 means that there is only a 90.5% chance that the difference is significant. But, when the numbers in both groups are small, it is difficult to reach a significant ‘p’. That is what is mean by the ‘power’ of a study. The ‘power’ predicts the number fairly needed to detect significant differences between two groups. 250 patients split into two groups of 125 patients were needed, 76 split into two groups of less than 40 in each was achieved.
Let’s think further on ‘power’. If two patients are assigned, each to a different treatment, and the one assigned (A) feels better and the other assigned to (B) feels worse … is this proof that (A) is a better treatment than (B)?
No … this might just be by chance. You need lots of patients in each group to discern a ‘true’ benefit from a ‘chance’ benefit.
Now, assign 1000 patients, 500 to (A) and 500 to (B). Let’s say that in group (A) 490 feel better and 5 feel worse but in group (B), 450 feel worse and only 50 feel better. In this example, you might reasonable conclude – and applied stats would confirm – that (A) has a better-than-chance likelihood of being the superior treatment
A statistician would (fairly) say that this particular study – the FHN NHD vs CHD trial – was simply far too underpowered to show the statistical difference required to determine whether the two groups were truly different – or just different by chance.
The correct outcome, and more reasonable report - if indeed a report at all? … a ‘mis-trial’ and a ‘null’ report. In my view – and that of many others – that the results were reported at all, and then, as a negative study, was, correctly, incorrect!
Left ventricular mass reduction (a good thing) was found in the NHD group when compared to the CHD group and was measured as a 19.6% reduction. This was a similar dimension of LV mass reduction as had previously been shown in the other arm of the FHN trial (short daily HD) … where the reduction was reported as ‘significant’ - because of the larger number of patients.
A similar reduction - also reported as ‘positive’ or ‘significant’ was found in Bruce Culletons RCT from Calgary reported several years ago now and regarded (still) as a seminal trial - both in dialysis broadly and in NHD in particular.
The small ‘n’ (number of patients) again ‘muddied’ the conclusion in the FHN v CHD trial, leading to a ‘negative’ outcome where, not due to the of size of effect but due simply to the size of trial, it showed a ‘positive’ outcome in the (separate) report of the SDHD arm that was released at the ASN last year in Denver.
In addition, the amount of dialysis delivered also did not meet the required separation between the two groups.
The CHD group dialysed for quite a bit longer (in the US, the mean CHD run time is 213 minutes for CHD but the far-too-small CHD group in this reportedly ‘negative’ trial dialysed for a mean of 256 minutes).
The NHD group dialysed for quite a bit shorter than most NHD patients fro around the world are reported to commonly use (the mean 6x NHD run time more commonly 8hrs+ providing >40 hours per week while the also far-too-small NHD group in this reportedly ‘negative’ trial dialysed only for 6 hour runs and a mean 30 hours per week).
In addition, less than 75% (to be exact, 72.7% of NHD patients completed their allotted number of dialysis sessions.
As a result, there was less dialysis delivery difference between the two groups, already under powered by patient numbers, to attempt to prove a statistical difference between the amount of dialysis delivered.
In terms of the (always) soft and hard-to-determine physical composite health score results – the ‘do you feel better or worse’ questions – the NHD score showed a 7.4 point improvement against the CHD group.
Again … the trial reports that this ‘trended to’ but ‘did not reach’ significance. This is likely (again) a function of inadequate study power … ie: too few patients followed.
Better blood pressure control (I think quite important), and better phosphate control (also quite important) and both among the other ‘benefits’ that were seen in the NHD group, were not listed among the 3 primary endpoints and, as a result, didn’t not count in the assessment of ‘positivity’ or ‘negativity’ of the trial as a whole.
Peter’s critique reaches many other telling and correct conclusions beyond these few. His conclusion is 'that although the FHN trial reported by Rocco et al is a technically negative trial, it is unlikely to deter NHD (enthusiasm or) enthusiasts’. He concludes that “the trial may be dismissed (by the enthusiasts) as a negative trial, which is possibly the correct interpretation, but we should be encouraged that home haemodialysis is a safe and effective therapy.”
My own conclusion might have added to an otherwise thoughtful and spot-on critique - and well might I say “well done, PK - the much sadder thought that … … “the reporting of this study as ‘negative’ when, indeed, if probably better deserved to be either un-reported or described as ‘inadequately powered’ and ‘non-statistical’ is, in my view, regrettable– it is a truly ‘null study’ … and that it is “also regrettable, becuase it unfairly empowers the dialysis status quo” (short, unkind dialysis) while “it dis-empowers the impetus to better dialysis through longer and more frequent therapies”.
In my own view (and this is clearly a personal comment) it is a report which might better have never been published – and I am surprised it was, given the statistical weaknesses. It will set back what I beleive has been enlightened progress to a better dialysis future by entrenching the poor dialysis regimens of our past.
Well, I personally will not let this underpowered study deter me from getting the dialysis I want. Thank you for explaining the article to us because the title and the abstract were very counterintuitive. I agree that it probably should not have been published. We can’t allow anything to stop us from our goal…“Optimal Dialysis for All!” Yay!
Hope your trip to China went well.
Thanks for you note(!) and your comment above. Much appreciated (both).
Yes, China was wonderful, thanks, and the meeting was useful, too.
When I say it might have been better if the FHN NHD v CHD trial had not been published … it is not that a negative trial should not be published: indeed one might say that not enough are. It is actually quite hard to get negative trials accepted for journals. But, on this occasion, it was negative because it was woefully underpowered. It was not because there was no difference between the modalities.
It was simply that not enough patients were in the trial. You can’t make stats out of 2 patient. Nor out of 4. Nor 6, or 10 (though, as the numbers rise, it is getting more likely all the time that a big difference is a true difference). The stats rule for a +ve result is a >95% chance. You can’t reach that with 2, or 4, or 10 patients. You need lots. They didn’t get lots - though not for want of trying, mind you.
The authors knew this. The journal knew this. The stats (for what they were) showed ‘trends’ - and strong ones too, given the tiny numbers - towards a positive result that I think all accept would have likely produced a positive trial had the numbers been enough … and yet, despite all, it was published as a ‘no’. But a ‘trend’ is a fuzzy stat … its the stat you claim if you don’t get ‘significance’ but think there might (should) have been a significance if the numbers in the trial (‘n’) had been sufficient. A trend, here, is actually a fair claim - for ‘n’ was only 35% of the ‘n’ predicted to be needed to show a significant difference. But, in the end, a ‘trend’ isn’t a stat!
What irks me most is that this publication didn’t fairly fess up to the under-powered nature of their results. And, in my view, it should have.
I rest my case.
I understand the difference between a negative trial and one that is underpowered, and I agree that “negative trials” can be valuable if they are not underpowered. This is not an urgent question, but I would be interested in hearing why you think this particular article was published at all since everyone seemed to agree that it was underpowered. Are journals of this nature desperate for anything to publish?
I explained re the stats as best I could and in that depth, mainly as some who read here may not be as clear on what stats mean and/or how they can be (mis)used.
I doubt it would be wise for me to speculate on how/why this trial was published … and I don’t want to get myself into too much hot water with others by doing so publicly … remember, I publish stuff too, and want to keep doing so! IHowever, I would say that it might have been better to place some different emphases (plural) on some of the interpretations.
It was, after all, a trial that fell short - indeed, far short (to <35%) of the number predicted to be needed to reach statistical significance. One could be forgiven for thinking that if a p value of 0.095 could be reached with an ‘n’ of <35% of the required ‘n’, how powerful might have been the positive outcome been if the results had reached the required 100% of ‘n’.
We will never know. It is a trial unlikely to be ever attempted again - though the ACTIVE trial from Australia might yet unseat the FHN conclusions - if its’ ‘n’ can be reached and if its’ outcomes are positive (and they = two big ‘ifs’). Again, we will not know - at least for a long time yet.
Why was this trial published? I can’t answer that except that I am surprised it was. One thing should be said - it is not the authors fault … an author has and always should have the right to submit a paper - no matter what the paper says or concludes. It is an editorial and reviewer responsibility to ensure that a submitted paper upholds the standards of medical reporting and statistical analysis required by the journal.
Further, among the authors are some close friends and like-thinking (to me) nephrologists - who contributed many of the NHD group patients. As major contributors, they are (and should be) co-authors. Like me, I think they are (likely) disappointed by the outcome. However, when one does the work - regardless of whether the study shows what you’d hoped (or not) - it’s important to see the work published. My only concern - and, possibly, theirs too - is that the language of the final write-up did not more reflect the power issues and less reflect the negativity.
When a paper is submitted, it is sent for ‘review’ to 3 independent reviewers to pass comment, to suggest changes, to point out weaknesses that need to be addressed or corrected before publication is agreed. The authors have the right to suggest who those reviewers might be (though the editors also have the right to over-ride the authors suggestions and send the paper to 3 reviewers of their choosing). Clearly, authors may select suggested reviewers who may feel sympathetic to the conclusions of the paper and thus provide most likely ‘acceptance’ pathways, the easiest passage to print … and least revision work for the authors!
There is much heat in the ‘CHD vs longer/more frequent dialysis’ argument … between those who favour the status quo and those who seek to change it. While I am for change - and as there is no disputing that, my cards are laid out for all to see - this is not the view of all: and I must (and do) respect the counter views of others. But, entrenched views, careers, practices, funding models and financial returns are powerful animals to challenge … and home dialysis concepts do challenge all these.
Contrary views are healthy! It drives debate, and debate drives rigorous assessment. But, it is kind of a low blow if those who support one side get in a free punch on the back of poor stats. I think that’s what may have happened here.
Perhaps the reviewers of this particular trial report were in the status quo camp, perhaps not … I don’t know. What has happened, I think, was that a trial that had - to me - a number of glaring short-comings, has ‘made it through to the play-offs’.
Enough said on this paper, I think.
Excellent summary of basic stats!
Many forget that the accepted thresholds of .05 or .095 etc. are essentially arbitrary, i.e, CHOSEN points at which to define
significance (proportionally tied to the number of standard deviations).
Unfortunately, if an experiment measures a difference, but only achieves say 89% probability (that it was NOT due to chance)–we call it “insignificant” and then go on to weak and confusing language such as “trending” (toward significance)
–All the while forgetting that in these situations it is still MORE likely that the observation is real than just random chance. (In this example 89% vs. 11%).