Design and Analysis of Individually Randomized Group Treatment Trials in Public Health



good morning my name is Marie Rienzo and I want to welcome you to the NIH office of disease prevention Lima gap webinar series this series explores research designs measurement intervention data analysis and other methods of interest to prevention science our goal is to engage the prevention research community and thought-provoking discussions to promote these the best available methods and support the development of better methods before we begin I have some housekeeping items submit questions during the webinar there are two options first to meet submit questions via WebEx by clicking on the question mark in the lavash toolbar please direct your questions to all panelists second you may participate by Twitter and submit questions using the hashtag NIH MTG I think at the conclusion of today's talks we will open the floor to questions that have been submitted via WebEx and Twitter lastly we would appreciate your feedback about today's webinar I'm closing the WebEx meeting you will be directed to a website to complete an evaluation we would appreciate your feedback as it will help us improve this webinar series at this time is like introduce dr. David M Murray associate director for prevention and director of the office of disease prevention Thank You Murray today's speaker is dr. sherry pals dr. pals is a statistician at the Centers for Disease Control and Prevention she had turned obtained her PhD in Experimental Psychology with the concentration and research design and statistics in 2002 she also got a master's degree in mathematical statistics in 2003 both degrees from the University of Memphis she joined the division of hiv/aids prevention at CDC in 2002 after discovering a love for international work dr. Powell's moved to the division of global HIV and tuberculosis in 2010 in her work in domestic and international HIV AIDS and tuberculosis she is designed and analyzed numerous group randomized trials she has authored several manuscripts on design and analysis clustered studies including group randomized trials and individually randomized script read the trials in fact she published a paper in 2008 in which she invented the label individually randomized group treatment trials and it has stuck in the field it's my great pleasure to welcome dr. pals Thank You dr. Murray both for that introduction and for inviting me to speak on this topic it's one that I've been interested in for a number of years and so I'm going to talk about design and analysis of individually randomized group treatment or IRG T trials and just a quick overview of what I'm going to talk about I'll first define what is an i RG t trial and distinguish it from some other designs that share characteristics with it I'll talk about intraclass correlation and its implications for design and analysis of IRG T trials and then talk about sample size and optimal allocation of participants to conditions then I'll talk about analytic methods for two types of IRG T trials the reporting requirements for these trials I'll present on a couple of reviews of published IRG T trials and some future work that's desperately needed in this area so first what is an ir GP trial these are child in which individuals are randomly assigned to conditions and then interventions are delivered to groups and i want to say a little bit about terminology here i'm going to refer to the treatment that participants are assigned to as study condition and then the groups that participants receive their treatment in so subgroups within conditions as groups some people refer to conditions as treatment arms and some people refer to treatment conditions as groups so I just want to be clear about that within these treatment groups there is the potential for correlation among individuals to develop over the course of the intervention and that happens for a variety of reasons so one can be therapist or facilitator effects some therapists or facilitators are more effective than others there's also group interaction the group members interact with each other and may promote better delivery of the intervention that way and the other and this is not actually an IR gt trial but participants where we're participants select themselves into a group based on their preference for a day or time so the implication implications of intraclass correlation in these studies the variance of test statistics can be larger in an ir GP trial than in a trial in which no correlation between participants is expected due to the additional between group variations so you have the typical between participant variation and then an additional component of variance that's due to differences between groups and so the inter class correlation coefficient or ICC is the measure of how much of the variation is attributable to groups so the percentage of the total variation due to group membership and like any correlation this varies between negative 1 and 1 with those two extremes indicating perfect prediction of the outcome based on group membership and so next the variance inflation factor this is the increase in variance that you would expect due to the correlation within groups and this is a very important quantity so it is 1 plus M minus 1 times the ICC where m is the average number of members per group so it's really important to notice when looking at this formula that the variance inflation can be impacted both by the ICC and by the number of members per group so you can have what looks like a fairly small ICC but if your group sizes are large it can still result in quite a substantial variance inflation so what this does is to reduce the sample size and here's a little formula the effective sample size is basically the sample size divided by the variance inflation factor some other authors also call that the design effect so I our GG trial study designs vary they may or may not have a baseline measurement it is better if they do for a variety of reasons an intervention delivered within groups would be compared to another group intervention or a waitlist or other individual intervention or some combination of these a small proportion of these studies have more than two arms and the number of and timing of the post intervention measurements can vary there could be a measurement right after the conclusion of the intervention and then there could be additional measurements to determine how the intervention effect is maintained over time so I want to talk a little bit about the history of IRG T trials the recognition of the issues of clustering in individually randomized trials really began in the psychology literature with a paper by Martindale in 1978 that was titled the therapist as fixed effect fallacy in psychotherapy research and so this article was the first to recognize that though individuals were randomized there could be clustering within groups or within therapists and that had important implications for analysis however in public health there was little mention of the potential foreign implications of clustering in individually randomized trials until the early 2000s at the same time the methods literature on group randomized trials or gr T's was growing rapidly there was also a mention of clustering in gr T's in the late 70s and then in the 90s as sample size and analysis software became available the literature expanded rapidly also at that time the computing power available made simulation studies possible and so investigators were able to look at how different analytic methods performed you know using simulation one of the first papers in public health or medicine to detail the issues of correlation in individually randomized trials so it was this one by Donald Hoover in 2002 and he referred to the between group differences as heterogeneous teaching subgroup effects he also derived the true type 1 error rate of tests that ignored correlation with the varying numbers of groups and members per group and found that that ranged from the nominal 0.05 to almost 0.5 so really discouraged the ignoring of correlation in studies with the potential for it within groups he presented a Satterthwaite unequal variance t-test for studies with two group treatment conditions and the difference between this and the standard methods that had been used for group randomized trials was just allowing that variance to differ across study arms or treatment conditions he also presented sample size formulas and an example for this approach so that was helpful and then at the very end of the article he acknowledged that this approach couldn't really take into account covariance and so he recommended mixed models to adjust for covariance so the next article to appear dealt with just that approach a mixed models approach that was a Robertson Roberts article in 2005 so they introduced the use of mixed models to account for clustering and presented formula for different allocations of participants to study arms based on different variances across arms so the idea there was to have optimal power you want to allocate participants evenly across arms but when you have different variances across arms different correlations within groups that can reduce the effective sample size in conditions differently so they presented a formula to deal with that and I'll talk about that a little more later on they also examined the performance of a mixed model that allowed variants to differ across arms and found that it performed well and lastly they dealt with an issue that has been coming up in group randomized trials for a long time the idea that you can test the icc for statistical significance and say if the ICC is insignificant then you don't have to take the correlation into account in the analysis and they argued against this for a couple of reasons and one is that these studies aren't really powered for the testing of that hypothesis and the second is that these studies are designed to deliver treatment in groups so the correlation is really part of a study design so it should be included in the analysis so next there were a couple of papers that dealt with what they called partially clustered data and that was the Baldwin in 2011 Bower sturby and Hal FERS in 2008 and so these studies had clustering in some conditions but not others so they're the situations where you'd have a group based treatment compared to a wait list or even bibliotherapy or something like that and they recommended models that allowed for a between group component of variants only in the conditions with group treatment and the Baldwin paper actually has some supplemental material that presents SAS code to be able to do this and it's a very nice example so if you're working in SAS you may want to look that up they also dealt with another argument that comes up in group randomized trials a lot the idea that you can add group as a fixed effect so this basically amounts to creating dummy coded indicator variables for the treatment group nested within condition and adding those to the model as a fixed effect the problem with that is that it actually inflates the type one error or even worse than ignoring the group altogether so it's really not a good approach and they also discuss some additional situations especially in the Bowers urban health ours article such as adjusting for baseline having outcomes with dichotomous or other distributions and multiple treatment conditions so they really started expanding the methods work there and then there's finally a very recent paper with an interesting twist the androgen all paper expanded the previous work to apply IRG T trials to studies where participants belong to more than one group and so this could be a situation where participants get components of the therapy in different groups or a situation where they get some therapy in a group and also belong to a naturally occurring group like say a classroom within a school so the Android paper talked about using SAS proc glim extenda m'f ECTS for both groups a participant belong to and they found out that you really have to do that to take into account correlation that is within each group to preserve the nominal type one error rate and they use ten word Rogers degrees of freedom and the model yielded nominal type one error rate in good power they also tested a generalized estimating equation approach or GE approach and that didn't perform well but they did have a limited number of groups in one part of the simulation and so they didn't apply a small sample correction for these so that might help the type one error rate okay I'm going to move on to talk a little bit about sample size for I our GP trials and this formula is a pretty standard one for to study conditions and it solves for detectable difference which is a smallest difference that we can detect with eighty percent power so smaller is better where detectable difference is concerned you want you want to be able to detect a small effect many behavioral interventions produce a small to moderate effect so it's very important to be realistic in terms of this detectable difference so this is the square root of and the Sigma squared Y is the normal between person variance so that's what you would have in a randomized clinical trial and then these two components here next in the equation are the two variances for the two treatment conditions notice we have the variance inflation factor in the numerator so one plus M one that's the average group size in condition 1 and Rho 1 which is the intraclass correlation coefficient for condition one so in the denominator of this variance we have n1 which is a sample size for condition 1 now pay attention here to the fact that you have the members per group in the numerator in the M and in the dot denominator in the total sample size so what this means is that if you add additional people to existing groups you're not going to increase the power that much because it's present in the numerator and denominator if you add additional groups that will increase power more because it doesn't appear here you don't change the M you only change the n ok so we have the same term over here for the condition 2 so this is going to allow for those rows to differ and for the MS to differ by condition and then at the end of the formula we have the critical values from the T distribution T alpha which is usually 0.05 and beta which is usually point 2 which corresponds to a power level of 80% and then we square that ok so I want to make a plug here for using the T distribution a lot of these formulas use the Z distribution to avoid having to change this every time you change the number of groups or yeah the number of groups per condition but using the Z distribution can really underestimate the number of groups you need so the second formula is for an I our GT trial with only one group treatment so that would compare a group treatment to a waitlist control for example so the first condition term here is the same we have the variance inflation factor over the total sample size but the second one we just have one in the numerator so if you multiply that out taking the regular betw person variance term you'd multiply that out and you would have that divided by the sample size which is just the usual variance in a randomized clinical trial so and one more thing about these formulas there it's fairly straightforward to you algebraically manipulate these so that you can solve for power or the number of groups per condition instead of the detectable difference it just depends on how you want to approach this and that's also fairly straightforward to modify them for different distributions so Alan Donner and Neil Carr have a book on cluster randomized trials that presents these formula for dichotomous variables it's just it would just be the usual group randomized trial formula where it pools these two terms so and I want to talk about the allocation ratio I mentioned earlier from the Roberts paper so this is simply the square root of the ratio of variance inflation factors so this is the variance inflation factor for condition 1 this is a variance inflation factor for condition 2 and so you divide those and take the square root and that tells you the proportion of participants that should go in condition 1 versus condition 2 now again you need to add those if you're trying to increase this to optimize power you need to add additional groups not additional members and that formula just reduces in the case of having one treatment that's group based in one treatment that is individually administered this formula reduces to just the numerator okay so this is a question that I get asked a lot in group randomized trials how do I choose an ICC you want to pick an ICC that's derived from a study as similar as possible to the study that you're planning and that means similar in the study design and importantly for IRG T trials the duration of the group interaction if your group is meeting for six months you don't want to use an ICC from a group that meets for four sessions because I would worry that the intraclass correlation would be a lot greater for the group that met much longer ICC's have been shown to differ by outcome variables with the correlation much higher for attitudinal and B Haverhill variables and for physiologic variables so keep that in mind there are a couple of good papers that report a lot of ICC's for group randomized trials so you can get some indication about the factors that affect ICC's so but what if just such an estimate is not available for an ir GT trial one thing that you can do is to look at unpublished data so does someone else have a similar study they might have published the results of a trial but not the ICC so it's possible to contact folks and ask about you know can we just estimate an ICC based on your data I used to recommend getting an estimate for a cluster randomized trial and using it for an IR GT maybe using it as an upper bound but I'm no longer convinced that that is necessarily the best idea at least without a lot of caution a lot of thinking about how the ICC arises in the cluster randomized trial and you know would that be similar for the IR GT trial or could the ICC be larger so it might be a place to start but I think carefully about that and here are a few of the ICC's that have been published I'm not going to read through all of these but I do want you to notice that they differ quite a lot so this first one is from a 12-week PTSD intervention for veterans and so these ITC's 0.042 0.13 are not that large and combined with groups of only six to eight it would not result in a huge amount of variation variance inflation but as we look at this next one the Hertzog paper those icy seas were point three two and point four so they're quite a bit larger and in the Baldwin 2011 paper they examined ICC's for psychotherapy studies and compiled a database of these now a number of those are not grouped treatment studies I don't think but they still may provide some interesting examples so Roberts & Roberts reported ICC's for a study of psychotherapy for schizophrenia and those were also fairly large and then this last paper that I talked about earlier the ICC for a behavioral measure for youth was 0.06 so there may be others out there that I didn't list but at least these will give you a start I want to shift now to talking about the analysis of IRG T trials and I'll start with those that compare two group treatments and so for these we can pull from the group randomized trials literature for some ideas about how to analyze and first I would say that a flexible classical approach is the mixed model approach this approach can incorporate covariance which is important you know because you could have imbalance across study conditions you know that's that's a typical thing to do at the beginning of a study is to look at table 1 and check for your balance of important prognostic indicators across conditions so these models if you do find imbalance these models can incorporate covariance and that includes a baseline measurement and that is a good reason for a couple of reasons it can adjust for pre-existing differences but it also may actually reduce the icc and improve power and it estimates those covariance parameters and can allow those to vary across conditions so you know that's sometimes the covariance parameters are of interest in themselves and so you want to be able to estimate those generalized estimating equations or GE may be an approach that can be used in IRG t trials and it can take the correlation into account in variance estimation but unlike mix models it doesn't explicitly model the group variants so it doesn't partition that variance into group and individuals it just fixes the standard errors for that additional correlation the most important thing to know about GE II though is it is an asymptotically robust method what does that mean so that means that with a certain large sample size ge is robust to miss specifications of the correlation matrix so you don't have to know exactly what that correlation looks like between your group members or over time but if you have a large enough sample that's okay so how large is large enough studies have shown that that's about 20 groups per condition so that's ni rgt trials that's 20 therapy groups or treatment groups within conditions um if you don't have that many and the vast majority of IRG T trials do not have that many you may need a small sample correction there have been several of these proposed in recent years and they're now actually incorporated into standard software including staffs so that's convenient another approach that's been used allotting in group randomized trials particularly early on where computing power wasn't extensive the permutation tests these are also called exact tests and they are based on the idea that under the null hypothesis of no effect of the treatment groups are exchangeable meaning they could be swapped back and forth in either treatment condition and you shouldn't see a big difference in the effect so that would be the difference between treatment means and so you can get a p-value by placing the observed effect in the distribution created by all possible arrangements of those group effects across conditions and so these are really the gold standard or testing for significance the only thing is their inadvisable if the group sizes differ across arms so it may be something that you can't use for a lot of ir GT trials but if you have the same group sizes and amount of contact across conditions then those may be a good approach and finally Bayesian methods may be more flexible in allowing the incorporation of prior information on the ICC and other parameters of interest you can actually allow the random effect to have different distributions here in the Bayesian approach which you can't do in the classical approach and so analysis of IR GT trials that compare a group treatment to individual treatments these are the studies that others referred to as partially clustered and they're more complicated than studies with group treatments in both arms or both conditions so you can use the Satterthwaite t-test approach and compute variants for each conditions separately and then just construct a t-test dividing the difference between means by the variance allowing those variances to differ across conditions you can fit mixed models that allow the correlation structure structure to vary across arms and this is where you take the Baldwin approach in the model correlation in arms with group treatment only so you only fit a group component of variance in one arm and then again the Bayesian methods that allow incorporation of prior information about covariance parameters I want to make a comment about degrees of freedom because when you use mixed models you have to select a degree of freedom method and the default methods often don't deal with complicated models very well and so degrees of freedom should be based on the number of independent units in the analysis so this is easy for group randomized trials and randomized clinical trials the groups are the independent units in a group randomized trial and individuals in a randomized clinical trial but in IRG t trials it's less clear so the Hoover paper reported a Satterthwaite method Baldwin examined Satterthwaite and Kenward Rogers method and found little difference and the Android paper recommended kenra Rogers degrees of freedom so both of those methods take into account the correlation in determining the number of independent units and both have performed well in multiple simulations oh I should say about degrees of freedom if you're using staff fast do not use the between within methods because those four degrees of freedom because those don't perform well and so I want to talk about reporting requirements for IRG T trials because as you'll see later on when I present the results of reviews of these studies very often important aspects of the trial are neglected in reporting so there was an extension of the consort statement a few years ago to non pharmacologic interventions and so this wasn't specific to IRG T trials and it's possible that a statement is needed for IRG T trials but it did point out a number of important things that are needed in reports of these studies so for each study condition you want to talk about whether treatment was administered in groups or not and fully describe the intervention that was given you want to report the number of groups and members per group report the sample size calculations in detail including the ICC that was used or the variance inflation factor and then also report the ICC's for all trial outcome variables and by study condition if applicable so if if those varied by study condition you want to report them for both study conditions so that means you're reporting to ICC's the one used in sample size calculation and the one that actually resulted from the data and then described in detail all analytic methods used including degrees of freedom okay so David mentioned the 2008 review that we did we've reviewed all published IR GT trials that appeared in six journals and we have the journals listed here and so most of them were in preventive medicine medicine health psychology obesity addictive behaviors and just a couple in aids and behavior and American Journal of Public Health the vast majority of these had to study conditions but eight and three had three and four study conditions and when you think about the impact of power the impact of the inter class correlation coefficient on power you have to wonder if these studies had sufficient power to have that many study conditions um the number of group treatment conditions most studies had one or two group treatment conditions a few had three or four and so then the baseline sample sizes the majority were less than 200 participants many of them less than 100 participants and so that raises questions about power again whether they were adequately powered so these are it says review methods that this is actually the results 32 of the 34 articles reported analysis at the individual level ignoring the group entirely so this means they made no mention of the group at all conveyed no awareness that delivering an intervention in groups could mean correlation within groups to reported mixed model analyses and one reported structural equation modeling and we determined by review of three of the authors for each paper that only one article reported appropriate analyses 32 reported in appropriate analysis and one didn't have enough information to judge the analytic methods and so that gets back to the reporting requirements I just went over you need to have enough detail on the analytic methods for the reader to judge then we reviewed in 2011 hiv/aids focused group randomized trials and IR GT trials published in seven journals and I'm only going to talk about the IR GT part of the review here we identified 25 I our GT trials and none of the 25 reported sample size calculations taking intraclass correlation into account to reported appropriate analytic methods and 21 reported at least one significant trial outcome so what that means is that many of those significant trial outcomes could happen from studies that use inappropriate analytic methods and had a much higher type 1 error rate than the nominal I also want to mention this paper on the Baldwin Murray and shadows paper this was a really great paper that looked at 33 group psychotherapy treatments they're designated as empirically supported treatments by the American Psychological Association so these treatments have been disseminated widely of the 33 they found none appropriately analyzed their trial data and when they applied a post hoc correction to look at how many would be significant when they corrected for the correlation they found that between 6 and 19 of the treatments would no longer be significant if appropriate methods were used but they did highlight the fact that they were using ICC's from papers that may report on studies very different from the ones in this compendium so they recommended compiling a database of ICC so that corrections to prior studies could be made with better precision and also so that investigators would have these ICC's when planning future studies so to present some conclusions although work on group randomized trials has been plentiful and recognition of the impact of correlation has increased the same can't be said for IRG two trials we really only have a handful of methods articles at this point and virtually no recognition of these issues in published trials and the methods work have been clear that ignoring the ICC and some in some cases different group sizes and differing variants across conditions can inflate the type 1 error rate sometimes substantially and sample size and analytic methods are now available for a variety of study designs and article reviewers and journal editors really should require appropriate methods for these trials I want to highlight some future work that I said earlier was desperately needed for IRG T trials one thing is an ICC database for public health outcomes we need data from IRG T trials with a variety of study designs intervention durations outcome variables etc and really since any one study has limited power and a limited number of groups sampling error for these ICC estimates is going to be large so really only when we get a few studies for NIACC can we have some confidence that in the value of that ICC so we also need to determine the impact of covariates on IRG t trial ICC's this is work that's been done for group randomized trials and we have a pretty good idea in some areas of what covariance will reduce ICC s and by how much and so we need the same for IRG T's we need sample size estimation software not everybody is able to take the formulas that I presented and do that algebra algebraic manipulation and come up with the exact sample size estimation formula that they need and employ that properly so we need to have packaged software that enables people to do that we need further reviews to demonstrate weather awareness and use of the proper design and analytic methods improves so far we see very poor recognition of this issue so and we also need further reanalysis of IRG t trial data that was originally analysed improperly now this isn't always possible I've tried a couple of data sets from that I've gotten here and unfortunately the group indicator variable for which a group which group a participant received treatment in isn't always in the data set so hopefully we will correct that going forward okay so last I'm going to ask for questions and offers of IRG t trial data or ICC's if you have such trial data I can you can either send me the data and I can do the analysis and we can work together for publishing that or I can guide you if you can't release the data I can guide you through estimating the ICC's and that's been done before was done in the baldwin paper so reach out to me and let me know what you have and i'm going to stop there and ask questions I thank sherry very much terrific presentation very clear I'm going to ask one clarification question and then I've got a whole just slew of substantive questions the clarification question you might want to go back to your slide 11 or slide 12 if that's possible to put your formulas up sorry that's all right yes so yeah here we go 11 so the the in there you've got m1 and m2 and you say that's the sample sizes for conditions 1 & 2 is that the number of groups small groups that where the interventions being delivered in each of those 2 study conditions or is that the total number of people in each of those 2 study conditions that's the that's the total number of people and I know you use mg there to be clear so it's the number of members per group times the number of groups okay your intention is it is the total number it's the total so yeah this is how it was presented in the baldwin paper and so i since i was referring to people referring people to that paper so much I wanted to be consistent but I agree that it is a little confusing if you're used to notation that breaks those out separately all right thanks for thanks for clarifying so some more general questions how somebody has asked how do I know if the ICC or variance inflation factor is small enough to ignore so that's a question that comes up in group randomized trials a lot and I've actually seen people report in papers the ICC was only 0.01 so we did not fit a mixed model or take into account the correlation the problem is when I presented that variance inflation factor and you can see it here on this slide that that ICC is multiplied by the number of members per group again so you know even a small ICC you know an ICC like 0.05 with 30 members per group can result in nearly a 50% increase in the variance inflation so for in the variance relative to independent observations so I really would say that I wouldn't ignore the any ICC it's part of the design and I would incorporate that ICC just as a plan from the very beginning since a positive intraclass correlation reduces power and we can see that in the formula because it it makes the detectable difference larger and we want usually a small one what can you do to increase power in an individually randomized group treatment Ron so there's a couple of things and one is something I pointed to a couple of times and that's to include covariance to reduce the ICC and what we want to do there is to think about what leads people to respond similarly within groups and is it anything that we can statistically adjust for are there more women than men in a group and is that related to the outcome age other demographic factors but the most important thing is adjusting for a baseline membership baseline the measurement so that's been shown in group randomized trials to reduce the ICC more than anything else and so including that baseline measurement reduces or or removes that variability that was existing before the intervention so that's a really great way to improve power and of course you you increase the degrees of freedom by having another measurement too so then the other thing is and people laugh at me here when I say this but make the intervention more impactful so think about is there any way we can make this intervention more powerful can we increase the length of time can we include other components that other people have found to be successful at improving whatever our outcome is and finally including more measurements in addition to a baseline measurement maybe think about including more post intervention outcomes I don't know if I left anything out sounds good to me okay we have questions about are there websites or software packages that will help someone do sample size calculations for an individually randomized group tree to try so there's one that I like if you're working in Stata I forgot to say that I have a couple of reference slides here and some of these papers have good examples of the baldwin paper 2011 has good examples of SAS code and he goes through each of the options for you know things that you can include in the model and how you can fit more than one or more than two treatment conditions and things like that um chris roberts who wrote the 2005 paper has a page that's got a lot of helpful resources on it but one of them is a sample size calculation procedure called CL sampai so it's easy to download and install that and his page has instructions on how to use that and then there's also the nih page which i assume you guys go over at the end of here but here's also the link it has resources on group randomized trials including a sample size calculator and I believe if I'm not letting the cat out of the bag early that you all are developing an ir GT trial sample size calculator that's the plan it's not there yet okay yeah so and then if you have the capability of doing simulation studies you might get to the point if you have multiple measurements and a you know an outcome with a less standard distribution you might need to do simulation studies to look at power and that's where if you don't know how to do those yourself it's a good idea to consult with a statistician full employment for biostatisticians yes other questions here is the design effect the same thing as the effective sample size no the design effect is the effective sample size is a function of the design effect so you would take the sample size the raw sample size and divide it by the variance inflation factor to get the effective sample size and that's sort of a crude way that you can do sample size calculations by dividing and getting the effective sample size and then putting them into a naive power calculator that ignores the clustering so I do that sometimes when I want a really rough estimate and you know need it quick that doesn't take the degrees of freedom into account no it doesn't it doesn't it's so it works better if you have large studies but yeah that's an Allen Donner in his book uses the deedsy distribution and then rounds up and it's a great book but that particular thing I don't love that because in really small studies you can underestimate the number of groups needed if you're going to estimate an inter class correlation from preliminary data that you have or you're reaching out to another investigator who's done a study something like yours and and has data and is willing to analyze it do you get an estimate of a crude intraclass correlation unadjusted in other words or or do you try to get an intraclass correlation estimate from a model that includes covariance which one of those is is preferable I would do both so in fact I'd do a series of them and I've published a couple of papers for group randomized trials ICC's doing this presenting the crude ICC and then one adjusting for a very limited set of demographic covariance and then one adjusted for expanded covariance and that's so that both so that you can get an idea of how the covariance help in reducing that ICC and for me to be able to present iccs in a paper and give people a range of ICC's to choose from if you're going to include the baseline measurement in your analytic model so let's say you have a pre post pretest post-test design and you've so you've got some data you've recommended adjusting for it because it can reduce the intraclass correlation are you talking about including it as a covariant or are you talking about including that pretest and posttest as two levels of time in a repeated measures mixed model so in the group randomized trials literature there are a couple of papers that show that most of the time including it as a covariant is the most powerful approach and that's actually true of randomized clinical trials as well so I would recommend that but it's a good idea to see those papers and dr. Murray's a co-author on those papers so they're easy to look up and and consider your specific situation it's helpful if you have parameters like the correlation over time for your outcome variable and obviously numbers of groups and members and that will help you consider which approach is likely to be more powerful um what if part of the effect of the treatment that you're delivering the intervention that your delivery is to increase the similarity and responses of the participants in other words it creates correlation within the small group it effectively creates or increases the intraclass correlation aren't the analytic methods that you're talking about going to get rid of that and effectively get rid of some of your treatment effect you know this is a question that we got years ago I think when I presented this at a conference and I think your your goal may be to promote interaction among the members of a group in order to improve their outcomes but I doubt that your goal would be to increase between group variation because that would mean that some groups do better than others so I don't think that would be the goal and even if that group interaction is the goal it doesn't get around the statistical issue that you have to take that into class correlation into account in having appropriate variance so if you were to leave that if you were to ignore that correlation because that's part of the out part of the results of your intervention you'd be reporting a p-value that's it's inappropriate or yes absolutely I think I think at this point we have three decades of research on that and if you're a statistician all you have to do is derive the variance of those condition means and you can easily see that you'd be inflating the type 1 error rate if you ignored it we have a question completely different kind of question how do you recruit schools or work sites or communities or other such groups to participate in a study that's going to be a group randomized trial or an individually randomized retrieve the tribe oh wow ok that is not my specialty but I've been involved in some of it as studies have gone on and I think the key is making contact with that school administrator or the group leader in my case it's very often in villages in Africa and it involves getting buy-in from the community leaders like tribal leaders and such and so it's really sitting down and communicating how important the study is and what it's going to do for the population that you're targeting and further than that you probably have to ask a behavioral scientist yeah I think that's a really good answer though within a traditional randomized clinical trial we're randomizing individuals the the pitch or the recruitment effort is aimed at the individual and it's their decision whether to participate or not when you're expecting to randomize groups in a group randomized trial then you really need to talk to the leadership of the groups and persuade them that it's an important scientific question and it's worthwhile in an eye rgt since you're randomizing individuals isn't the the pitch or the recruitment still going to be aimed at those individuals even if they end up getting some of their treatment in small groups yeah I think so so you're still trying to enroll individuals and doing the same thing that you would do in a randomized clinical trial one thing I realize I forgot to mention and it has nothing to do with this question but when I was talking about reducing ICC's there's some indication that manual izing treatments spelling very clearly out in a manual how the treatment is to be administered and ensuring fidelity to that treatment manual can reduce ICC's but then again that's not for every situation because some treatments involve really adapting to the participants you have so that's not applicable in every situation I just wanted to put that out there so you've made a you've offered an invitation to members of the audience to share data with you or collaborate with you on papers can you go back to the slide that shows how people can get in touch with you and say a little bit about what sort of information they would need to send you if they want to pursue this invitation okay so I would need to know the detailed information about the study so if there's a study protocol or a publication that describes the study the intervention and those details I would need that and then I would also need data that most importantly has an indicator for the group a participant belong to and then covariance of interest and we can talk about what those might be I think a first step might be reviewing the protocol or study description and then talking with the investigator about what covariance might influence the outcome and also what Co various are in the dataset and then we could come up with a reduced data set from the entire trial data set or like I said I can guide the investigator through doing this analysis and that can be done in SAS data are I'm not completely sure about SPSS is capabilities but there's definitely a mixed model procedure I'm just not sure it allows the variance to vary across arms um one of the things you mentioned in that list of pieces of information that you would need was a an indicator variable to identify the small group that a person received their treatment in you cited the and rige which talked about more than one group and we also could imagine the situation where people change groups so maybe they go to the Tuesday evening session to get their weight loss treatment for a while and then their schedule changes and they have to go to the Thursday evening session is it important to keep track of all the different groups that a person is in where they receive treatment and do you need to take all of that into account 'ok it could get rather complicated yeah absolutely i would say keep track of it and we can deal with the analytics complexity in a variety of ways and see now you definitely the Android's paper pointed out that you definitely do have to take into account multiple groups that a participant belongs to but what we don't know is how large are those ICC's if if a participant belongs to a classroom and then goes and gets one session of a group treatment how big is that ICC how big is the variance inflation factor and those are the things that we don't know we have enough methods work now to say we have to take it into account but but how big are those ICC's that's a pretty big mystery okay terrific discussion and lots of great questions from the audience I appreciate it very much and sherry I want to thank you for a great presentation I'm going to turn this back over to Marie and we're going to close off the session as we hit the one hour mark thank you dr. Marie and thank you to everyone substituted miss Kay's webinar on the mind the gap website to mention nih.gov / – s you'll find several resources for this talk including supplies and a wrist a list of references they'll also be posting a recording of today's webinar on our website next week you'll receive an email with the link to the recording when it's available in yes you you

Leave a Reply

Your email address will not be published. Required fields are marked *