Friday, August 03, 2007

Autism and ABA in the UK: A controlled trial

The ABA non-randomized controlled trial presented by Patricia Howlin at the International Meeting for Autism Research this year has recently been made available online. You can find the abstract of this in press paper here [edit: If you follow the link, you'll see that this paper has now been published]:

Magiati, I., Charman, T., & Howlin, P. (in press). A two-year prospective follow-up study of community-based early intensive behavioural intervention and specialist nursery provision for children with autism spectrum disorders. Journal of Child Psychology and Psychiatry.
As I wrote earlier, Dr Howlin's study is a prospective, community-based study comparing the outcomes of 28 autistic children receiving home-based intensive ABA-based interventions (the ABA group) with a control group of 16 autistic children receiving "autism specific nursery provision" (the nursery group). The interventions are described as being "typical" of ABA and nursery services available in the UK. The children were between 23 and 54 months and had a very wide range of IQs (16 to 138) at intake. They were followed up after ~26 months.

The ABA group received intensive 1:1 Lovaas-type ABA, with two children receiving additional Verbal Behaviour services. For 20 ABA group children, ABA services were provided by "recognised ABA organisations in the UK, Norway or USA", while the remaining 8 children had independent providers. The nursery group was distributed across 10 different schools. They received what would be called "eclectic" intervention, where the "most common named practices" included TEACCH, PECS (an ABA-based approach), Makaton, and SPELL (from a 2001 guide published by the UK National Autistic Society), as well as "other developmental and behavioural teaching methods".

And--as reported in the abstract--after a bit more than 2 years, there were no significant differences to report between the ABA and nursery groups in a multitude of outcome measures. On only one measure was a difference approaching significance found in favour of the ABA group (Vineland Daily Living Skills, where p=.06). In contrast, there were major individual differences among the children in progress made, regardless of which group the children were in.

As happens in non-randomized controlled trials, the two groups were not matched on many variables at intake. The authors accounted for only one of these unmatched variables--a significant difference in mean intake IQ which favoured the ABA group (83 for the ABA group; 65 for the nursery group). Other significant differences at intake (e.g., Vineland socialization, Vineland composite, parental education) favoured the ABA group and were not accounted for, while other unaccounted for differences favouring the ABA group were just short of being significant (e.g., the ABA group was younger at intake). None of the between-group differences at intake favoured the nursery group.

The authors also made no attempt to account for gender. The nursery group had a higher percentage (33%) of females than the ABA group (4%). In my view (and apparently, in the authors' view), this isn't a major issue and if anything--according to the lore that autistic females are at a disadvantage--this difference would again favour the ABA group.

The ABA group also had the advantage of receiving a significantly more intensive intervention than the nursery group, both at the outset of treatment and at follow-up, with the ABA group at ~33hrs/wk (range 18-40) and the nursery group at ~26hrs/wk (range 15-30). Within each group, intensity of intervention did not change from the outset to follow-up. That is, the children's need for services did not decrease over the course of about two years. The nursery group received little in the way of 1:1 intervention (6hrs/wk average, compared to the ABA group, where the full ~33hrs/wk was 1:1).

Because they received a more intensive intervention, the ABA group also received a significantly higher total number of intervention hours than did the nursery group--an average of 3415 hours versus 2266 hours per child. This is a difference of 1149 hours favouring the ABA group.

Using the average intensity of intervention for the ABA group, the ABA group received an equivalent of 35 weeks more in the way of intervention time than the nursery group children. Using the average intensity of intervention received by the nursery group, the nursery group received an equivalent of 44 weeks less in the way of intervention time than the ABA group children. That is, for the amount of intervention to be equivalent in both groups, the nursery group children would have had to receive an additional 44 weeks of intervention (at their average intensity of intervention).

On the other hand, this study found no relationship whatsoever between intensity of intervention and any outcome measure. This is keeping in mind that the range of intensity in the ABA group was 18-40hrs/wk. In my IMFAR poster this year, I mentioned the failure of the ABA literature to relate higher intensity of ABA-based interventions with better outcomes, and in Dr Howlin's study, that failure continues.

Another factor that had no effect on outcomes was age at intake. Whether the children were younger or older at intake, across the range from 23 to 54 months, made no difference to how well they did. The popular if not ubiquitous contention that, when it comes to autism interventions, "earlier is better" has failed--again--to be supported by evidence from a controlled trial (for a previous failure, right up to intake at age 7, see Eikeseth et al., 2002, 2007).

And regardless of the ABA group starting with an average IQ well within the "high-functioning" range (indeed, the intake IQ here for the ABA group is the same as the follow-up IQ in Lovaas, 1987), after ~2 years of intensive ABA, all children in this group were still receiving ABA services and none was in a mainstream school without 1:1 assistance. This is also regardless that almost one-third of the ABA group did not have the specific diagnosis of autism at intake, instead being assigned a subthreshold "ASD" diagnosis.

On average, neither group did well. They both made progress in age-equivalent scores but, as the abstract reports, standard scores changed little. Individuals either did well or did poorly, and this was unrelated to which kind of intervention they received, the intensity of intervention, or the age at which intervention began. This is similar to what was found in a recent Canadian observational study (Eaves & Ho, 2004).

Dr Howlin's study, like all studies, has weaknesses and limitations (and of course I want a lot more information about practically everything), some of which are addressed by the authors. I find it ironic that the authors argue for autism-specialized services by citing Cohen et al. (2006). While it is difficult to compare across studies, it could be argued that the control group in Cohen et al. (2006), which received completely inadequate services that no one should recommend (generic segregated special education of low intensity), fared better by the usual standards than both groups--the ABA group and the nursery group--receiving autism-specialized services in Dr Howlin's study. Dr Howlin and colleagues also fail to point out that when unmatched intake variables are accounted for in Cohen et al. (2006), the few significant differences between groups (ABA vs generic segregated special education) in all but one outcome measure (classroom placement) disappear.

Dr Howlin and colleagues conclude that:

Our data support the growing consensus that no one intervention for children with ASD is universally superior to all others (NIASA, 2003).
My own conclusion is one I've arrived at before: after more than 60 years of autism research, encompassing the colossal existing autism intervention literature, researchers still cannot scientifically claim to know how to help autistic individuals (Volkmar et al., 2004). The best adult outcomes reported in the autism literature continue to belong to individuals who grew up before the current era of early interventions and who as children met the narrowest, strictest autism diagnostic criteria ever devised (Dawson et al., in press).

I'm prepared to agree with Dr Howlin that autism-specific services are important. But even the evidence Dr Howlin and colleagues drum up in support of these services glaringly reveals how poorly autism research has served the interests of autistics. Even if you leave out ethical concerns that would be paramount with any non-autistic population, the major currently-popularized and -marketed autism educational interventions (as opposed to non-popularized non-marketed science-based approaches; see Aldred et al., 2004; Gernsbacher, 2006) leave little to choose from. Autistics deserve a whole lot better.


References:

Aldred, C., Green, J., & Adams, C. (2004). A new social communication intervention for children with autism: pilot randomised controlled treatment study suggesting effectiveness. Journal of Child Psychology and Psychiatry, 45, 1420-1430.

Cohen, H., Amerine-Dickens, M.S., & Smith, T. (2006). Early Intensive Behavioral Treatment: Replication of the UCLA model in a community setting. Journal of Developmental and Behavioral Pediatrics, 27 (S2), 145–155.

Dawson, M., Mottron, L., & Gernsbacher, M. A. (in press). Learning in autism. In J. H. Byrne (Series Ed.) & H. Roediger (Vol. Ed.), Learning and memory: A comprehensive reference: Cognitive psychology. New York: Elsevier.

Eaves, L.C., & Ho, H.H. (2004). The very early identification of autism: outcome to age 4 1/2-5. Journal of Autism and Developmental Disorders, 34, 367-378.

Eikeseth, S., Smith, T., Jahr, E., & Eldevik, S. (2002). Intensive behavioral treatment at school for 4- to 7-year-old children with autism: A one-year comparison controlled study. Behavior Modification, 26, 49–68.

Eikeseth, S., Smith, T., Jahr, E., & Eldevik, S. (2007). Outcome for children with autism who began intensive behavioral treatment between ages 4 and 7: A comparison controlled study. Behavior Modification, 31, 264-278.

Gernsbacher, M.A. (2006). Toward a behavior of reciprocity. Journal of Developmental Processes, 1, 139-152.

Lovaas, O.I. (1987). Behavioral treatment and normal educational and intellectual functioning in young autistic children. Journal of Consulting and Clinical Psychology, 55, 3–9.

Volkmar, F.R., Lord, C., Bailey, A., Schultz, R.T., & Klin, A. (2004). Autism and pervasive developmental disorders. Journal of Child Psychology and Psychiatry, 45, 135-170.

49 comments:

mcewen said...

I cannot tell you what 'balm' this is.[to an unscientific person such as myself] As a biased parent, two things jump out at me 'eclectic' approach [which is the best I can manage on a good day] and that 'early intervention' isn't the be all and end all.
Cheers

daedalus2u said...

Excellent post.

The only legitimate way to measure the "outcome", when it involves a human being is to ask them.

This can be problematic because there are compensatory mechanisms that humans use when subjected to trauma to survive and mitigate the injury. Capture bonding and Stockholm syndrome being examples where trauma induces attachment to those doing the abuse.

mcewen, from what I can tell from your blog, the interventions you are doing are of fabulous, virtually mythic quality. If a trial could be done comparing your "eclectic" approach to ABA, the trial would be stopped early because your "eclectic" approach is so obvously superior.

notmercury said...

Hi Michelle,
Is PECS an ABA-based approach or did I misunderstand the parenthetical reference?

Michelle Dawson said...

Hi Not Mercury,

PECS is a form of ABA.

There's a PECS chapter (written by Bondy & Frost) included in Dr Lovaas' most recent ABA manual (Lovaas, 2003).

Coincidentally--Dr Howlin recently published a PECS randomized controlled trial (Howlin et al., 2007).

notmercury said...

Thanks Michelle,
I'll add that to the long list of things I've learned from you.

David N. Andrews MEd (Distinction) said...

Wonderful, Michelle.

"My own conclusion is one I've arrived at before: after more than 60 years of autism research, encompassing the colossal existing autism intervention literature, researchers still cannot scientifically claim to know how to help autistic individuals (Volkmar et al., 2004). The best adult outcomes reported in the autism literature continue to belong to individuals who grew up before the current era of early interventions and who as children met the narrowest, strictest autism diagnostic criteria ever devised (Dawson et al., in press).

I'm prepared to agree with Dr Howlin that autism-specific services are important. But even the evidence Dr Howlin and colleagues drum up in support of these services glaringly reveals how poorly autism research has served the interests of autistics. Even if you leave out ethical concerns that would be paramount with any non-autistic population, the major currently-popularized and -marketed autism educational interventions (as opposed to non-popularized non-marketed science-based approaches; see Aldred et al., 2004; Gernsbacher, 2006) leave little to choose from. Autistics deserve a whole lot better."

These two paragraphs say it all, really, don't they?!

Of all that you have done on the issue of autism and the associated research, I think this has been your most salient (not sure if this is the right word entirely, but at 7.45am my brain is no good without tea) post.

You've surpassed yourself.

HortenseDagle said...

I'm curious if side-effects were gauged at all in this study? Side-effects or negative outcomes such as elevated stress levels, fatigue, sensory overload, sleeping disorders or eating difficulties? Both groups had, imo, intense "work weeks" and I'd be surprised if that didn't affect the children.

Michelle Dawson said...

Hi Hortensedagle,

You asked, "I'm curious if side-effects were gauged at all in this study?"

The answer is no, and this study is typical in this respect. Here's a recent quote from a famous behaviour analyst:

"Side effects are defined as unintended effects. No intervention is likely to be side effect free, yet this topic has largely been ignored in the intervention literature on children with ASD." (Matson, 2006).

The overwhelming concern in the literature (and elsewhere) has been for the satisfaction and well-being of parents (and sometimes other family members) of autistics receiving ABA services. E.g., there has been and continues to be a lot of concern about whether putting an autistic child in an intensive ABA program is stressful for that child's parents.

HortenseDagle said...

Thank you, just as I figured. Yeah, let's study the side effects of ABA on the parents. [/sarcasm]

It just seems to me that if your child is prone to stressing out easily, you would want to reduce intense activity and not give them an adult work week.

Suzanne said...

I too, find the connection of PECS and ABA confusing. In our house, PECS is a form of communication, not really a therapy. There are rules governing the use, such as he must point to/push each word in the sentence, and I speak it, mimicking(and in prep. for) certain communication devices.
I also attempt to use the cards to express language to him (BUS!)

Suzanne said...

so I went looking... found this
http://www.autismusaba.de/whyaba.html

"The problem is that PECS without ABA only teaches a child what communication is. It takes a firm understanding of the ABA principles for PECS (and sign language) to be useful as a step towards effective verbal communication."

oh

Jennifer said...

Michelle,
I know that there are other studies that have looked at the difference between ABA and "eclectic" interventions, and they have found ABA more effacatious. For instance, there is Howard et al, Res Dev Disabil. 2005 Jul-Aug;26(4):359-83, and Eikeseth et al, Behav Modif. 2007 May;31(3):264-78, which seems to be a follow-up to Eikeseth et al, Behav Modif. 2002 Jan;26(1):49-68.

I carefully read Howard when it first came out, and as I recall that there was quite a difference between the educational levels of the parents in the two groups.

Since I am sure that you are much more familiar with these texts than I, I wonder if (when you have time) you could give us a quick rundown of the issues with the randomization in those studies. Thanks.

Michelle Dawson said...

Hi Suzanne,

Something PECS-like is sometimes used in informal ways with autistic kids (by parents and others), in ways which might not adhere to the principles of behaviour analysis.

E.g., I've been told about one non-speaking autistic girl who got the idea way ahead of the steps you are rigidly required to take in PECS, and started ripping images out of magazines to communicate things no one had ever considered she might want to communicate.

But mostly I write about research, and PECS in the scientific literature is an ABA-based intervention.

Michelle Dawson said...

Hi Jennifer,

You wrote, "I wonder if (when you have time) you could give us a quick rundown of the issues with the randomization in those studies."

Neither Eikeseth et al. (2002, 2007; I referenced this two-paper study in my post) and Howard et al. (2005) is a randomized controlled trial. There is no randomization, so I'm not sure how to answer your question.

Suzanne said...

Thanks Michelle!

Jennifer said...

Michelle,
Sorry, I had forgotten that Howard (2005) was not randomized. Would it be possible for you to comment on how the way the children were assigned to the two different groups might have had impact on their outcomes? Thanks.

Michelle Dawson said...

Hi Jennifer,

There are three groups in Howard et al. (2005), and the only factual answer to your question (how did group assignment affect outcomes?) is, who knows? The problem with non-random assignment to groups is that you can only speculate about how some group differences (those not accounted or only partially accounted for by the authors) may or may not have affected group outcomes.

I find it interesting that parents had the final say in the services their children were assigned to in Cohen et al. (2006), while having influence but not the final say in Howard et al. (2005; where IEP and IFPS teams decided). I don't know enough about how things work in California to know the reason for this apparent discrepancy. In Eikeseth et al. (2002, 2007) assignment was according to availability of services, so parents had no say at all.

But there is no way of knowing in any of these studies what effect the method of group assignment in itself might have had on outcomes. In all three studies (and in every non-randomized trial), the authors were free to choose which variables (of many, many available) they thought were important to measure and account for. The rest are unaccounted for and can only be speculated about.

Lili Marlene said...

"The best adult outcomes reported in the autism literature continue to belong to individuals who grew up before the current era of early interventions and who as children met the narrowest, strictest autism diagnostic criteria ever devised.."

What are you saying here, Ms Dawson? Are you saying or hinting that autistic people who met older and stricter autism diagnostic crtieria had, as a group, greater intellectual potential than kids who are being diagnosed as "Autism Spectrum Disorder" these days?

Are you pointing out that these autists with good outcomes met the strictest definition of autism so that one can't counter-argue that the autists with good outcomes were "only Asperger syndrome" or only had a little bit of autism?

I've heard that Kanner's definition of autism explicitly excluded intellectually disabled people and people who clearly had some kind of genetic syndrome, while these days people who fall into these other categories are also being given autism spectrum diagnoses. I wish I had time to read Kanner myself!

Michelle Dawson said...

Hi Ms Marlene,

I passed along what is reported in the literature (this can be verified by reading the literature). Also, I did not refer to "intellectual potential."

At least two of Kanner's original 11 developed epilepsy. While this is a very small sample, this near-20% rate is much higher than expected in an idiopathic autistic population. Most of Kanner's original 11 were reported by Kanner to have been treated or diagnosed as "feebleminded" at some point. Several autistics were "introduced to us as idiots and imbeciles" (Kanner, 1943).

Kanner (1943) reported IQ scores for only two of his original 11. He added a third on follow-up (Kanner, 1971). None of these three scores was below 80 and one was 140. None of these three autistics was reported to have a good outcome.

laurentius rex said...

Michelle I hope you will pardon me posing this question on your blog, but like it or not I am a bona fide academic researcher now, and I am intersted in your opinion on certain things ABA related

What is your opinion of Nikopoulos

who has written of behaviour modelling using video. My interest is purely with the video aspect of this, as I am attempting at the moment to ground the educational uses to which video has been put in the context of autism.

In case you are not aware of Nikopoulos here are a few references. He wrote his thesis on the subject at the Uni of Ulster in 2003

Several papers on these lines :

Effects of video modeling on social initiations by children with autism
Authors: Nikopoulous,Christos K.; Keenan,Michael

Video modelling and behaviour analysis: A guide for teaching social skills to children with autism
Authors: Nikopoulos,Christos; Keenan,Mickey
Source: (2006).Video modelling and behaviour analysis: A guide for teaching social skills to children with autism.205 pp.London, England: Jessica Kingsley Publishers, 2006
Source: J.Appl.Behav.Anal., 2004, 37, 1, 93-96

Enhancing the ability to reflect on self and other : a qualitative study using video during social situations with pupils with autism
Authors: Nikopoulos,Christos K.
Source: 2002., University of Birmingham, Birmingham

Promoting social initiation children with autism using video modeling
Authors: Nikopoulos,Christos K.; Keenan,Michael
Source: Behavioral Interventions, 2003, 18, 2, 87-108

write to me privately if you will.

Michelle Dawson said...

Hi Mr Rex,

I only know Nikopoulos's 2004 JABA paper, which is based on the one-way-street model of social reciprocity in autism (see Gernsbacher, 2006). Here's an exemplary quote from Nikopoulos & Keenan (2004):

"Reciprocal play was defined as the child engaged in play with the experimenter using any toy in the manner for which it was intended." (p. 93)

I don't know Nikopoulos' 2003 paper (the abstract is here). I haven't read his book, or his 2007 JADD paper (not on your list).

I know some of the other video modeling papers in the ABA literature (these go back into the 1980s). E.g., video modeling was used in conjunction with in vivo modeling and a token economy to train autistic children to emit the "right" "affective behaviour", regardless of their actual emotions (Gena et al., 2005).

I suggest reading through the video modeling studies in the ABA literature in autism (if you haven't already). There aren't too many. Mostly, short videos are (and occasionally aren't, as in Nikopoulos & Keenan, 2004) combined with reinforcement, and used in limited ways to train limited behaviours in single subject designs with mixed results (according to the usual behaviour analytic standards).

Video modeling is now mentioned as a component of some (but not all) comprehensive early ABA programs (e.g., Cohen et al., 2006; Sallows & Graupner, 2005).

I have no idea if any of that answers your question(s).

laurentius rex said...

Well I have to admit my bias towards what I want to find in my research, that video is effective because that is what I do, therefore I could cite Nikopoulos and his antecedents and contemporaries as evidence that in one particular aspect video modelling is effective, however being super critical I can't accept that because all the studies are small scale and not what I would call controlled. It all predicates back to Bandura really and all of those numerous pundits who have posited the harmful effects of watching too much TV on behaviour.

Usefull for a literature review and that is about all.

I would be really interested in looking at the cognitive side of how autistic's and others interpret the perception of and the semiotics of video. What elements of a video, fire up what in the brain, and whether that is a lasting effect. will need some collaborators for that though as this is not the kind of research that Birmingham goes in for.

All part of the triangulation as I intend to fit all this into classic film thery going back to Eisenstein, as I figure, that ideology or otherwise he knew what he was doing, the essential grammar of the edit all goes back to the pragmatics of what has worked over the past century, the popularity of Hollywood being a testimony to that.

Namrata, Vedabhyas said...

Hi,

Greetings from Deepstambha, the virtual organization working towards an inclusive society! I like to congratulate for a very heart rendering blog.

To highlight the inspiring stories and experiences of how socio-medically disadvantaged people across the world with indomitable courage, faith and determinations are volunteering to create a better world, Deepstambha has launched a blog, http://deepstambha-lightforever.blogspot.com. Deepstambha is a virtual organization and believes in the need to promote an Inclusive Societies with equal opportunities for all.

To understand the issues and concerns of the socio- medically disadvantaged being highlighted in the blogosphere; Deepstambha is doing an online study. The study will help better understanding of the contents related to the socio- medically disadvantaged in the blogosphere, the types of related blogs, the issues highlighted in these blogs and whether they can lead to the promotion of an inclusive society.

We have a few questions which will help us conduct the study. We request you to spare a few moment of yours to respond to this questionnaire. As soon as we are able to compile the findings we will share our findings in our blog. We request you to support us in this endeavor of ours and answer the questions.

With warm regards,

Namrata Bansal,

Team leader,

Deepstambha

Following are the questions we request you to reply:

1) Do you think the blogosphere can contribute towards highlighting the issues and concerns of the persons with disability and the terminally ill? If yes, how?

2) Can blogs be part of efforts to promote an Inclusive Society?

3) What inspired you to start your blog?

4) What are the challenges you are facing in promoting your blog?

5) Do you consider the blog as one of your achievements? If yes, how?

6) Can blogs help raise resources? How can blogs be sustained over a period of time?

7) Is blog an effective medium through which you can connect to the persons with disability across the world? If yes, how?


"Stand up, be bold, be strong. Take the whole responsibility on your own shoulders, and know that you are the creator of your own destiny. All the strength and succor you want is within yourselves. Therefore, make your own future."-Swami Vivekananda

Interverbal said...

Fair points Michelle.

I am looking forward to see how you deal with my own behavior analytic forray into the science. :)

Chaoticidealism said...

If only they'd managed to put the variables straight! I'd really like to see a well-designed study that determines whether or not ABA helps autistic kids--especially because the newer play-based, reward-based type might not cause the worrisome psychological damage that the older, punishment/reward kind of ABA caused.

(There's still the BIG risk of learning things without learning why you do them, though--the "robot" problem--and the problem of eliminating functional autistic behaviors simply because they look autistic. Those can be damaging even if the ABA itself isn't.)

But you'd have to differentiate ABA, specifically, from friendly attention from an adult who isn't using ABA--because it could simply be the repeated opportunity for interaction that causes results. After all, if you have trouble learning communication, you will be more likely to learn it if you have a lot of opportunity to practice...

Still no proper studies on ABA. This irks me.

KeithABA said...

I don't even know if you'll see this comment because it's pretty late! But anyways...

Three things:

1. Did the article specify who ran the ABA program? I think it is very important to consider that the article did not say an ABA approach. It said, a home based intensive early intervention approach. You mentioned it was consistent with the Lovaas ABA method. What does that mean? They bought the ME Book? So were these programs designed and monitored by a Board Certified Behavior Analyst?

2. I very highly doubt the article specified what the ecclectic approach did every day and for every interaction. Do the authors assert that the "control group," was never prompted? What about hugging or tickling them for a job well done?

Were these variables controlled for? A great deal of the treatments probably overlap in both groups. How then could this be a control group, if the independent variable was present in both groups? I am hoping you don't make the argument that ABA was the independent variable, but if you'd like to, we could discuss that.

3. I think the Analysis that this article somehow show's that ABA doesn't work, or is harmful, is a pretty faulty argument. At best you could say that this article supports that the children who got this "Intensive Early Intervention Package," did not prove to make more significant gains than the ecclectic group.

I think you really have to seperate the effects of ABA based strategies on specific targeted behaviors, rather than as a "Cure," for autism. While Lovaas and some individuals may make that claim, that is hardly the concensus among behavior analysts.

Strategies such as shaping, prompting and fading, Functional communication training, Functional Analysis, Escape Extinction, and Differential Reinforcement are not somehow "invalidated" by this experiment.

This article does not prove those methods as innefective, or weaken the well designed within subject designs that have proved them to be effective with certain individuals.

Michelle Dawson said...

Hi Keith ABA,

1. I suggest that you read the paper, though I did describe the interventions received by both groups in my post. Your arguments about treatment fidelity and BCBAs would invalidate many existing ABA controlled trials.

2. Your argument re the control condition would invalidate almost every existing ABA controlled trial, including the solitary RCT.

3. My comments about this article are in my blog post and don't resemble those you attribute to me.

I'm not sure, but you seem to be denying the usefulness or effectiveness of intensive ABA-based autism interventions.

In my view, single subject designs are useless unless the knowledge acquired through these studies contributes to better outcomes in controlled trials of intensive ABA-based interventions (the kind of intervention that is promoted and funded as the only "effective" autism treatment). And so far, this has not happened.

The hundreds (or is it thousands) of single subject designs published in the 37 years since intake started in Lovaas (1987; where intake started in 1970) have not resulted in any improvement in autistic outcomes reported in controlled trials of early intensive ABA-based autism interventions. More the opposite.

KeithABA said...

Michelle:

If you can summarize the whole paper to make your points, don't you think that it would be fair to accurately describe the treatment group? It seems unfair to use some sort of transitive property of EIBI to ABA. Your post is "Autism and ABA in the UK: A controlled trial." Clearly your goal was to associate the lack of statistically significant difference with ABA, not EIBI.

1. Then let's invalidate them. I am all for that. There aren't good studies becuase this is something really hard to study.

2. Only in reference to group design studies. With most well controlled group design studies, the IV is only given to one group. The control comes in that the IV is easy to control. Real pill vs placebo. Saline vs. medication injection. That seems really hard to do with a treatment package.
As for within subject studies, I don't see how you could make this argument. The methodology is completely different.

3. I should have stated that more clearly. You don't directly make some of those arguments, but many other people use your posts and comments as a reference.

"I'm not sure, but you seem to be denying the usefulness or effectiveness of intensive ABA-based autism interventions"

I don't think there is great evidence at this point about ANY treatment in reference to long term outcomes when the outcome is measured by IQ tests.

In my opinion, interventions that are effective produce socially significant changes in behavior. To scientifically validate the effectiveness of a given procedure, that procedure must be studied with a strong methodoligical design.

Even if an IQ doesn't change over time, that does not mean that there were no changes in behavior. Same thing for a Vineland, or any other standardized test.

If a child has no signs, then an intervention teaches them to use signs, and they are using them spontaneously, and without prompting, how is that useless?

Michelle Dawson said...

Hi Keith ABA,

Sorry to take so long to get back to you.

So far as I can tell, you've taken the position that Lovaas-type ABA programs (received by all the experimental group children in Magiati et al., 2007) are not behaviour analytic. You are free to take this position (I can just about imagine what would happen to me if I did), but you've provided no evidence to support it.

1. "There aren't good studies becuase this is something really hard to study." Many interventions and treatments can be argued to be "really hard to study." This is one reason why there is good experimental design. I suggest that autistics deserve good experimental design.

Also, seeing as "there aren't good studies," shouldn't behaviour analysts and others stop making science-based claims re early intensive ABA-based autism interventions? And shouldn't behaviour analysts and others do something about all the false or exaggerated claims that have been made?

2. I mentioned controlled trials of ABA-based autism interventions, as well as the solitary RCT. Uncontrolled designs in autism have a record of not being informative. Also, see above point, re the availability of good experimental design.

3. I don't understand the point you're making here, sorry. My point remains that my comments "don't resemble those you attribute to me."

Re the girl who learns to sign without prompting, the certainties (meaning, biases) of those who invest in, provide, and/or market interventions or treatments is another reason why good experimental design (including the RCT; and then there's meta-analysis, to deal with publication bias, etc.) exists and is essential.

As Lord et al. (2005) wrote,

"RCTs protect the public from deliberate false claims and exploitation and from well intentioned and sincerely-believed views that may nonetheless be false."

KeithABA said...

Michelle,

Thanks for the response.

"So far as I can tell, you've taken the position that Lovaas-type ABA programs (received by all the experimental group children in Magiati et al., 2007) are not behaviour analytic."

I'd have to read it. My question was more, why did the article call it "EIBI," and you called it ABA in your review. Again, I'd have to read it, but it is fair to say that not all EIBI is ABA.

"And shouldn't behaviour analysts and others do something about all the false or exaggerated claims that have been made?"

We would have to look at each of those claims, and decide how false or exaggerated each one was. I do not beleive that behavior analysts should advertise to parents that with an ABA program their kid has a 50% chance of recovery. From my work in the field, and this is my opinion, I'm not even sure "Recovery," is a valid goal. What I do beleive is, that 100% of kids in a competent ABA program, (with or without therapists) make socially significant changes in behavior.

A big rift that will never be closed is the difference in our selections of dependent variables and acceptance of within subject methodology. I do not beleive you need an RCT to demonstrate weather a child learned signs due to an intervention.

http://www.pubmedcentral.nih.gov/
articlerender.fcgi?artid=1279571

A multiple baseline across subjects is enough evidence for me to say that this procedure was responsible for the increase in spontaneous requests, and decrease in problem behaviors.

The way in which the dependent variable is measured, is very different and cannot be compared other studies. In other within subject studies the dependent variable was not explicitly defined, repeatedly measured, and interobserver agreement was obtained during measurement.

Ex. Kids were better once on the secretin diet.

The methodology of all within subject designs (which are not always only 1 person) are not the same. There are good and bad within subject designs, just like there are good and bad between subject (group) designs.

Michelle Dawson said...

Hi again Keith ABA,

I'm repeating myself, but you are claiming that Lovaas-type ABA programs aren't behaviour analytic. In my view, that's the kind of claim that should be accompanied by evidence.

You also stated that "there aren't good studies" of early intensive ABA-based interventions. That is, claims that these interventions are effective are unfounded.

Autism advocates are not demanding funding for non-intensive short term single-behaviour interventions for children whose diagnosis is irrelevant. Indeed, they have claimed that non-intensive ABA-based interventions are ineffective. They are demanding unlimited comprehensive intensive ABA-based interventions for autistics starting at a very early age, and are claiming that these demands are science-based.

You wrote, "I do beleive is, that 100% of kids in a competent ABA program, (with or without therapists) make socially significant changes in behavior."

For some reason, this reminds me of this bit by Ben Goldacre:

"in 1995, only 1% of all articles published in alternative medicine journals gave a negative result. The most recent figure is 5% negative."

And thank you for providing evidence that behaviour analysis is fantastically reinforcing for behaviour analysts. It must really be something, to believe oneself infallible.

But I (again) suggest that such extraordinary claims require evidence, from the kinds of experimental designs that can account for biases. As I noted above, vast quantities of single subject designs, representing (it is claimed) huge leaps forward in behaviour analytic technologies, have not contributed anything to (and may have detracted from) the outcomes of autistic children receiving comprehensive early ABA-based autism interventions.

Also, there's the problem of how "socially significant changes in behavior" are defined, and by whom. And there's the problem that apparent gains (by the standards of whoever decides what is "socially significant") may be accompanied by equal or greater losses or harm. Etc.

See Tryer et al. (2008), for an example of why good experimental design is necessary, and why it is unethical to set aside certain populations, like autistics, as being undeserving of good experimental design.

Re Durand and Carr (1991), apart from the concerns raised above, and the problem of zero standards of diagnosis (etc.), there's those later studies showing that punishment was necessary to make "this procedure" work. I'm sure you can find those published studies.

Then there's the unexplored possibility that a different kind of intervention might be better. E.g., training nonautistics to respond to the communication of autistics (see this, some of the comments are interesting also). This one requires minimal resources, has a strong empirical foundation, is supported by a published RCT, etc.

KeithABA said...

"I'm repeating myself, but you are claiming that Lovaas-type ABA programs aren't behaviour analytic. In my view, that's the kind of claim that should be accompanied by evidence."

An I am repeating myself in saying, if you are going to call the title of your post "Autism and ABA in the UK" then you should explain why you changed the name from EIBI to ABA. The title of this article which you present was not "Autism and ABA...."

"Re Durand and Carr (1991), apart from the concerns raised above, and the problem of zero standards of diagnosis (etc.), there's those later studies showing that punishment was necessary to make "this procedure" work. I'm sure you can find those published studies."

What? Are you confusing extinction and punishment? Could you site what study you are referring to? There are a lot of FCT studies, the majority reqruied a replacement response and extinction. There are very few articles with punishment procedures in JABA.

"Also, there's the problem of how "socially significant changes in behavior" are defined, and by whom."

The parents or guardians are generally the ones that decide what's socially significant.

"And thank you for providing evidence that behaviour analysis is fantastically reinforcing for behaviour analysts. It must really be something, to believe oneself infallible."

And it's the progress the hundreds of children I have worked with over the years that makes behavior analysis reinforcing. Particularly the learning of sign language and actually speaking. Behavior analysis itself is dry and boring. Implementing it in the real world changes your life.

Michelle Dawson said...

Hi Keith ABA,

Either your position is that Lovaas-type ABA programs are not behaviour analytic, or your position is that Magiati et al. (2007) were dishonest in their paper. Those are very serious accusations that should be accompanied by evidence.

Sorry, I assumed that you were familiar with the literature about functional communication training. See Fisher et al. (1993), Hagopian et al. (1998), Hanley et al. (2005), all of them in JABA.

Re the role of parents, and what is "socially significant," see Nordyke et al. (1977; this is in JABA). The word "short-sighted" is used to describe behaviour analysts who become "the parents' agent, rather than the child's or society's" (p. 54).

Re your certainties about the services you provide, see my previous message, including the quote from Ben Goldacre, and the mention of Tryer et al. (2008).

KeithABA said...

Michelle,

My position has always been that the Lovaas program that was used during the RCT is similar, but not characterisitc of a comphrensive ABA program today. I never accused Magiati et al. of being dishonest. I repeat that I am asking why you called it "Autism and ABA." when the actual article that was published was titled, "A two-year prospective follow-up study of community-based early intensive behavioural intervention..."?

So FCT requires punishment and those articles are your support for that statement? I don't get it... You are way smarter than that Michelle!

"FCT with extinction was effective in reducing problem behavior
for the majority of clients and resulted in at least a 90% reduction in problem behavior
in nearly half the applications." (Hagopian 1998).

The articles you sited in no way prove that punishment is neccesary for FCT to work. It shows that sometimes, particularly when delay to reinforcement fading was added, FCT with extinction will not be successful at reducing problem behaviors. FCT works great when the schedule of reinforcement is FR1. Try making the person wait for the reinforcer after an apporpriate response, and they may go back to the previously learned response that has resulted in access to that same reinforcer (usually a challenging behavior).

Punishment can be effective, no one ever denied that. That's an ethical issue anyway, and has nothing to do with the reliability and validity of these procedures. While external validity remains an issue when utilizing single subject designs, the internal validity is very strong. In my opinion, internal validity is stronger in most single subject designs vs. group designs.

We are still debating the same argument overall, which has it's origin in the selection of a dependent variable to measure success of a treatment.

I say, single subject designs are enough to indicate that a procedure successful with one individual may be successful with another. I also accept that changing the frequency of just 1 behavior, such as signing or physical aggression, is a socially significant change.

You say, single subject designs are worthless, because they do not correlate with improved IQ and standardized test scores (outcomes). Change in single isolated behaviors does not equate to socially significant improvements for the individual.

We are probably just going to have to leave it at that....

Michelle Dawson said...

Hi Keith ABA,

I'm not sure what RCT you're referring to. But you you're still claiming that Lovaas-type ABA programs aren't behaviour analytic. And I'd still like some evidence for that.

From the discussion of Hagopian et al. (1998, p. 227, emphasis is mine):

"To summarize ... (a) FCT without extinction was not an effective treatment; (b) FCT with extinction reduced problem behavior in most cases, but produced clinically acceptable outcomes in less than one half of the applications; (c) demand and delay-to reinforcement fading reduced the effectiveness of FCT with extinction in about one half of the applications; and (d) FCT with punishment was effective in every application, independent of whether demand or delay to reinforcement fading was added to the treatment package."

Like one of the other studies I cited, one of the punishments was a basket hold. Basket holds have the possibility of being fatal, see some accounts of this here.

You wrote,

"That's an ethical issue anyway, and has nothing to do with the reliability and validity of these procedures."

Wow. I think that speaks for itself.

As I wrote above, autism advocates are not demanding public policy changes to mandate non-intensive single-behavior interventions. If ever they do, I'll write about those demands and whatever evidence they're based on.

You wrote,

"You say, single subject designs are worthless, because they do not correlate with improved IQ and standardized test scores (outcomes). Change in single isolated behaviors does not equate to socially significant improvements for the individual."

No, this is not what I say, sorry. What I did say can be found in my earlier responses.

KeithABA said...

Michelle wrote:
"As I noted above, vast quantities of single subject designs, representing (it is claimed) huge leaps forward in behaviour analytic technologies, have not contributed anything to (and may have detracted from) the outcomes of autistic children receiving comprehensive early ABA-based autism interventions."

??? That doesn't say that the single subject designs are worthless because they do not result in positive outcomes (standardized testing) ???

" one of the punishments was a basket hold."

I'm not sure what this has to do with your original claim that punishment was neccessary to make FCT successful. You even quoted that in that study it was successful without punishment in just under 50% of the participants.

Why did you call your post "Autism and ABA..." when the Magiata et al article was title, "A two-year prospective follow-up study of community-based early intensive behavioural intervention..."?

Michelle Dawson said...

Hi Keith ABA,

My blog post is (clearly, I hope) about the article authored by Magiati et al. (2007), not about the title of this article. If this isn't clear, I should probably retire.

You had written that my position is that:

"single subject designs are worthless, because they do not correlate with improved IQ and standardized test scores (outcomes). Change in single isolated behaviors does not equate to socially significant improvements for the individual."

That's not my position. My position is stated in the numerous messages above, where you might also find my views re behaviour analytic assumptions about "socially significant improvements."

And again, when autism advocates stop demanding, as public policy, comprehensive, intensive early behaviour interventions for all autsitics, and start demanding non-intensive single-behaviour interventions instead--then I'll be sure to address those new demands and their empirical foundations.

But so far, single subject designs (in their massive quantity) have been invoked by autism advocates as evidence for the effectiveness of intensive early behaviour interventions for all autistics. So that is what I am addressing.

Re basket holds, and functional communication training, my view is that it's important, at both scientific and ethical levels, to report the literature accurately.

KeithABA said...

"My blog post is (clearly, I hope) about the article authored by Magiati et al. (2007), not about the title of this article. If this isn't clear, I should probably retire."

You should not retire, you offer a great critique of ABA and for the most part I agree with a lot of your criticism.

That being said, I feel strongly that you chose the title to associate the negative outcome with ABA, rather than an EIBI group. After reviewing the article I am convinced that the EIBI group was not a competently designed and supervised ABA program.

There are some gleaming problem here with EIBI group in this article.

First and foremost, 1 sentence clearly indicates that the children in the EIBI group were not receiving a comprehensive ABA program. "All families used discrete trial teaching techniques
(Lovaas, 2002), although 2 later introduced ‘Verbal Behaviour’ (Sundberg & Michael, 2001)."

Discrete Trial Teaching Techniques does not equate to ABA! Only 2, I repeat 2 families later included Verbal Behavior training!!!! WOW!!! That's 2 out of 28 participatns that received one of the most important parts of a comprehensive ABA program.

Only 3 of the 28 families in the EIBI group received monthly consultation. The credentials of the consultants from these, "recognised ABA organisations in the UK, Norway or USA;" were not listed. The organizations were not even listed. Who were these consultants and what was their training? What are their certifications? These facts should not be left out of an article.

"Eight families changed their ABA organisation or supervisor consultant at least once." Why weren't those 8 exluded from the results? That is a huge confound. Consistency is so important in an ABA program, and several families changed their consultant "at least," once?

"Most therapists were psychology
or special needs education students or graduates." Again, no list of the credentials or the training they received. There is no way of knowing if any of the therapists or senior therapists were even CABA's.

In conclusion, your critique of this article was very accurate in description of the outcomes and group differences. However, this was a comparison of 2 eclectic therapies. 1 with more hours and some ABA consultation, another with "TEACC Hbased
approaches (Schopler, 1997); PECS (Bondy & Frost, 1994); Makaton (Grove & Walker, 1990) and
SPELL (NAS, 2001); other developmental and behavioural
teaching methods were also used".

This does not excuse the ABA community from doing a well designed group study with a randomized control group. However, this article does absolutely nothing in showing that ABA was not effective, but rather that this EIBI group was no in any way superior to the eclectic preschool group.

Michelle Dawson said...

Hi again Keith ABA,

My title was chosen accurately to reflect the community-based study reported by Magiati et al. (2007).

Your position is (now) that I dishonestly used the term "ABA" in order deliberately to mislead. That's another serious unfounded-in-evidence accusation to add to the pile you've made about other people.

E.g., you are now (or is it again?)saying that various international and UK ABA service providers are not providing ABA services. Where is your evidence for this?

And this is the nth time you've taken the position that Lovaas-type ABA programs (as described in Lovaas, 2002, one of the major ABA manuals) are not behaviour analytic. Additionally, so far as I can tell, you are saying that Lovaas-type ABA programs are to be considered "eclectic" (and therefore ineffective) unless it is explicitly stated that they include VB training.

Also, re the quality of intervention, role of consultants, etc., Sallows and Graupner (2005) found that a much higher level of consultation and supervision (including from at least one BCBA), and other measures taken to ensure higher quality of treatment, did not produce better outcomes in early intensive ABA-based interventions.

KeithABA said...

"Your position is (now) that I dishonestly used the term "ABA" in order deliberately to mislead."

No, I don't see that as dishonest at all. I would never accuse you of dishonesty. I think in your opinion, this was a competently designed and supervised ABA program. I also think it serves your cause to call this ABA because your M.O. has and will be to critique ABA.

"E.g., you are now (or is it again?)saying that various international and UK ABA service providers are not providing ABA services. Where is your evidence for this?"

Providing a service and a competently designed program are entirely different ventures. If someone can afford 2 hours a month of services, then thats what they can afford. If they then grabbed a bunch of students and had them do 40 hours a week of therapy, would I consider that a competent ABA program, absolutely not.

A 40 hour a week program takes more than monthly consult supervision. They didn't even specify how many hours a month. Was it 1 hour, 2 hours, a whole day? Some only got supervision every 4-6 months.

Further, why was there no list of the credentials of the consultants? I have run into more than a handful of occasions in which agencies or individuals claim to be trained in ABA, and they have no clue. In fact, many times they are clinical psychologists who read a book or two about ABA and now consider themselves competent behavior analysts. If the authors can confirm the credentials of the consultants, I would take back this portion of my critique.

Saying that you based your EIBI program on 1 book doesn't mean squat. To later admit that only a few incorporated VB, is just adding insult to injury.

I can go read an introduction to quantum physics, does that make me an expert on the theory of relativity? Am I now ready to induce and explain black holes? Not quite....

Michelle Dawson said...

Hi again Keith ABA,

I reported what Magiati et al. (2007) reported. If I'm not being dishonest, then you're back to you accusing Magiati et al. (2007) of being dishonest.

Also, you're free to accuse recognized international and UK ABA service providers, as well as some independent service providers, of being incompetent and dishonest in the provision of services. But I suggest that you have some evidence handy, when you make those kinds of accusations.

I'm still looking for controlled trials where the time of consultants or supervisors (including BCBAs) and the training of therapists was crucial to the outcomes of autistics in early intensive ABA-based interventions.

To repeat what I pointed out above, the existing evidence is that more supervision/consultation time and more therapist training was not only not associated with better outcomes, it was possibly associated with worse outcomes (Sallows & Graupner, 2005).

Generally, in non-ABA areas, it is well recognized that educational interventions should be manualized. Lovaas (2002) is one of the major existing autism early intensive ABA manuals. But if your point is that there's insufficient evidence to support the effectiveness of any manualized early intensive ABA-based intervention, then I would agree with that.

Anonymous said...

I realise this section of the blog is from 2007 but I want to take my ASD son out of a nursery where he gets ABA nursery. Obviously no one can stop me anyway but I was looking for research to back up my decision. My printer has been busy since finding your site !

Eduardo Carbonell Cruz said...

Estimada Sra. Michelle Dawson:

He leído algunos artículos suyos y comparto con Vd. que lo verdaderamente es estudiar cómo aprenden los niños con diagnóstico de Trastornos Generalizados de Desarrollo. El DSM-IV-TR utiliza unos criterios tan generales que no pueden deducirse de ellos técnicas de enseñanza. En segundo lugar comparto con Vd. que el mundo teórico del autismo está plagado de lugares comunes. Le dejo algunos enlaces a mi blog donde publico artículos precisamente orientado a técnicas de enseñanza y rehabilitación del lenguaje y a tratar de poner de manifiesto muchas de las falacias promovidas por investigadores descuidados. He visto que algunos de ellos los tiene Vd. en un catálogo de pícaros. Le dejo tres muestras: una sobre contacto ocular; otra sobre enseñanza del uso de tú y yo; y otra sobre la ecolalia.
La aproximación que hago al estudio de los niños con diagnóstico de autismo es desde el campo de la neuropsicología (Vigotsky, A.R. Luria, L.S.Tsvetkova...). Soy autodidacta igual que Vd. También comparto con Vd. que no es a través de la modificación de conducta, ni a través de métodos PECS, como puede abordarse la educación de los niños. Hay que comprender las causas profundas del singular perfil de cada niño, entendiendo por perfil, no el resultado de la aplicación de test de distintas funciones cognitivas, sino las formas concretas del desarrollo de sus distintas funciones psicológicas, formas que incluyen los propios medios de compensación que desarrollan estos niños, medios de compensación que pueden aún más alejar a los niños de alcanzar objetivos deseables, pero que, en la medida que la educación reglada y de su entorno no les ofrece ninguna alternativa, es inevitable. El estudio de esos métodos de compensación desarrollados por los propios niños, nos pone sobre pistas importantes de la naturaleza de sus verdaderos problemas.

1http://autismointegral.blogspot.com/2005/12/como-transformar-la-ecolalia-en_06.html
2.
http://autismointegral.blogspot.com/2006/05/autismo-contacto-ocular-y-motricidad.html
Cerebral areas mediating visual redirection of gaze: cooling deactivation of 15 loci in the cat.Lomber SG, Payne BR.
Cerebral Systems Laboratory, School of Behavioral and Brain Sciences, The University of Texas at Dallas, Richardson, Texas 75080, USA. lomber@utdallas.edu
3.
http://autismointegral.blogspot.com/2007/01/como-enseñar-el-uso-de-yo-y-tu-a los-niños.html

Traducción automática.(lamento no hablar inglés)

Dear Mrs. Michelle Dawson:

I have read some articles I share with you and yours. that it truly is studying how children learn to diagnose disorders Generalized Development. The DSM-IV-TR uses such general criteria that can not be deduced from them teaching techniques. Secondly share with you. that the theoretical world of autism is littered with platitudes. He left some links to my blog where public articles precisely oriented teaching techniques and rehabilitation of language and try to bring out many of the fallacies promoted by researchers neglected. I saw that some of them have you. in a catalogue of Rogues. He left three samples: one on eye contact; another on the use of teaching you and I and another on the echolalia.
The approach I am taking to the study of children diagnosed with autism is from the field of neuropsychology (Vigotsky, AR Luria, LSTsvetkova ...). Like you I am self-taught. I also share with you. that is not through changing behavior, or through methods PECS, as can be addressed children's education. You have to understand the root causes of the unique profile of each child, meaning profile, not the result of applying test different cognitive functions, but the specific ways of developing its various functions psychological forms that include their own means of compensation who develop these children, means of compensation that can remove even more children to achieve desirable goals, but that to the extent that informal education and their environment offers them no alternative, it is inevitable. The study of these compensation methods developed by children themselves, puts us important clues about the nature of their real problems.

1http: / / autismointegral.blogspot.com/2005/12/como-transformar-la-ecolalia-en_06.html
2.
http://autismointegral.blogspot.com/2006/05/autismo-contacto-ocular-y-motricidad.html
Brain areas mediating visual redirection of gaze: cooling deactivation of 15 loci in the cat.Lomber SG, Payne BR.
Cerebral Systems Laboratory, School of Behavioral and Brain Sciences, The University of Texas at Dallas, Richardson, Texas 75080, USA. lomber@utdallas.edu
3.
http://autismointegral.blogspot.com/2007/01/como-enseñar-el-uso-de-yo-y-tu-a the-niños.html

Mizuki said...

I have just been turned in by the ABA police (local advocates) for Infraction #1: attempting to "discredit ABA based interventions" #2 by circulating systematic literature and meta reviews as well as critical commentaries; and #3 suggesting to providers that they keep their claims about services to families consistent with the scientific literature (regardless of treatment or method). Over a 3 month period, I have shared relevant documents such as: Howlin et al, 2009; Gernsbacher, 2003; Ospina et al 2008; Roberts, 2006; Seida et al; Spreckley & Boyd, 2009 & Shea, 2004 among others. I did not circulate Rogers and Vismara, 2008 or Reichow & Wolery and the Johnson Meyer "policy" paper or others because they had already been distributed, along with several unreferenced advocacy type documents. I was turned in to Dr. Tristam Smith who then accused me of "intellectual dishonesty" in an e-mail, which they attached to a note sent to my employers. I'm not sure how to respond to this. My experience with ABA psychologists (outside of the DD/ASD specialty) was that they take the science seriously; not like this experience. Do you think I'd have better luck disseminating the information to policy makers directly? (at the State and Natonal level?) I don't think knowledge can be shared unless there are recipients who are open to "receiving" it. It feels like I am being accused of taking someone's religion away.

Michelle Dawson said...

In response to Mizuki, see this, and the comments are relevant also.

I have never known how to respond when, as has often happened, behaviour analysts and those promoting ABA-based autism interventions refuse to take ABA seriously as a field of science.

Mizuki said...

Yeah, it really is a religion. Here is what he said. They did not like that I circulated the "response" to Jacobsen's cost benetit paper. He is so full if It I don't know where to begin in a response.
"The Marcus, Rubin, and Rubin piece (#1 on your list) is a response to a cost-benefit analysis by Jacobson, Mulick, and Green (1998) on EIBI. Jacobson et al. concluded that EIBI should produce substantial cost-savings. This is certainly a reasonable view. As Marcus et al. point out, however, it can be questioned because the authors based their analysis on hypothetical data rather than a direct analysis of children who received EIBI and because some of Jacobson et al.'s assumptions may not hold true (e.g., that "best outcome" children will require no services after EIBI). On balance, I think one could reasonably say that EIBI is likely to be cost-effective, though we don't know exactly what the cost savings would be.



The Spreckley and Boyd article is one of five systematic reviews or meta-analyses that have appeared in the past year and a half or so (Eikeseth, 2008, Research in Autism Spectrum Disorder; Eldevik et al., 2009, JCCAP; Howlin, Magiati, & Charman, AJIDD; Reichow & Wolery, 2009, JADD; Rogers & Vismara, 2008, JCCAP; Spreckley & Boyd, 2009, JPeds). All except Spreckley and Boyd concluded that, despite methodological problems in the studies, the evidence supports the efficacy of EIBI. Thus, it is intellectually dishonest to cite only Spreckley and Boyd and not the other reviews. Also, Spreckley and Boyd made some outright mistakes: They classified a group that received 32 hours per week of EIBI from Sallows and Graupner's (2005) study as not receiving EIBI or ABA, and they compared this group to those whom they did classify as receiving EIBI. If they simply corrected this mistake (or removed the Sallows and Graupner study from the analysis altogether), their data would show a statistically significant advantage for EIBI over treatment as usual. In addition, they excluded some studies on the grounds that they did not analyze children who dropped out of the study; however, these studies did not have dropouts. Correcting this mistake would increase the number of studies in the report and strengthen the evidence for EIBI."

Michelle Dawson said...

(Revising my previous, deleted comment because of an error...)

Just briefly, that isn't a complete list of recent systematic reviews (using the term loosely), never mind the overlap in authors between two of the selected reviews.

Also, there is a general lack of consistency in how the ABA group designs have been classified across the systematic reviews and meta-analyses, and in the ABA literature in general, as well as in narrative reviews (e.g., Myers et al., 2007). This is a feature of the literature as a whole, not just of one paper.

While Spreckley and Boyd (2009) has serious methodological problems (as do all the other papers using meta-analysis with the ABA group designs; for example, they all violate recognized standards), Dr Smith's statement about intention-to-treat analysis is inaccurate.

Setting aside all other problems for the moment, only two additional studies could have been included if they had used an intention to treat analysis. But contrary to what Dr Smith claims, one of them had drop-outs (and it is certainly not the only ABA controlled trial with drop-outs), and the other is a retrospective design.

In non-autism, non-ABA areas, the need for intention-to-treat analyses has been recognized since the early 1980s.

Anonymous said...

I asked Dr. Smith which studies he felt had been mistakedly left out of Spreckley and Boyd (2009) analyses and I'm not sure which studies he is referring to from his response. Do you know which? Dr. Smith: "... My point was that most of the studies in their review didn't have any dropouts, so it doesn't make sense to fault them for failing to include dropouts in the statistical plan. From Table 1, it appears that their decision to do so led them to exclude two studies that otherwise would have scored high enough to be entered into the meta-analyses: Cohen and Sheinkopf. "

PS Sometimes the default account is my daughter's depending on which computer I'm at so sorry if it is confusing; I'm the same person : )

Michelle Dawson said...

The two studies that would have been included if they had received one more point are Cohen et al. (2006) and Sheinkopf and Siegel (1998).

As I wrote above, one of them has drop-outs. The other is a retrospective design--which was also excluded from meta-analysis in Eldevik et al. (2009), as was Magiati et al. (2007).