An anomaly in autism intervention research

Promotion first, science later, if ever. This pattern is near universal when it comes to autism interventions. In the absence of good quality research, autism interventions are loudly claimed to be effective.

For those promoting ABA-based autism interventions, claims of effectiveness unfounded in good quality research were only the first step. The real triumph has been widespread agreement that fair tests of ABA-based interventions are unethical and bad for autistics. As a result, any experimental design carrying the risk of being informative about the benefits and harms of ABA-based interventions has, for a long time now, been considered unethical.

The practice of claiming effectiveness for an autism intervention which has not been fairly tested, then using these claims of effectiveness to deem fair tests unethical, has clear benefits to service providers. And this practice has received wall-to-wall support from autism advocates, who have in turn imposed it on autistics through lobbying and litigation.

Meanwhile, this practice is not admired outside the realm of autism advocacy. Premature claims of effectiveness in themselves make fair tests of interventions more difficult--particularly, as is the case with many autism interventions, when blinding cannot be fully achieved. But those promoting ABA-based autism interventions go further and demand that autistics be entirely denied the benefit and protection of good experimental design.

In my view, this practice--its longstanding and widespread support by autism advocates, its more recent support by ASAN (example here), and its widespread imposition on autistics--continues to greatly harm autistics. Any group so denied such basic standards of science and ethics would be harmed.

Interestingly, not all autism researchers have stooped to prevailing autism advocacy standards.

Some years ago, a pilot RCT of an early autism intervention was published (Aldred et al., 2004). The intervention was manualized, of relatively low intensity, and targeted solely at the parents of preschool autistic children. Through a true experimental design, the intervention was found to have several significant positive effects, most markedly in increasing the expressive language of very young autistic children.

Successful RCTs of early autism interventions are virtually non-existent. But the researchers involved in Aldred et al. (2004) totally failed to go forth and proclaim the effectiveness of their manualized intervention. After all, they only had a small pilot RCT--the same size as the only published RCT, not a notably successful one (Smith et al., 2000, 2001), in the entire 48-year history of research into ABA-based autism interventions.

Instead of issuing premature claims of effectiveness, the authors of Aldred et al. (2004) and many other collaborating UK researchers went on to conduct a large multi-site RCT, the Preschool Autism Communication Trial. They received MRC funding for this 4-year project in 2005.

Some information about the PACT's design has been reported in a recent paper (Aldred & Green, 2009).

The trial began in early 2006 and will finish late this year. The researchers hoped to recruit 144 preschool autistic children across three sites, but exceeded their expectations and recruited 152. These children were randomized into two arms, one receiving the PACT intervention, while children in both arms continued to receive whatever other interventions their parents chose ("treatment as usual").

So far as I know, the PACT is larger than any existing published RCT in the history of autism research. The PACT's design means that it carries a good risk of being informative about the effectiveness of the studied intervention. There is no good reason for the PACT to be unprecedented, but it is.

In one of the PACT's recent newsletters, the PACT's Chief Investigator, Jonathan Green, notes that there has been a lot of interest in the PACT manual and in "training opportunities" with respect to this intervention. Dr Green responds:

Until we have the results of the trial we are not able to disseminate the intervention manual or to undertake any training; but be assured that after the results are in (and depending of course on the outcome in terms of effectiveness!) we do have potential plans in place for such dissemination.

In other words, and keeping in mind there is a successful published pilot RCT, there are no plans to disseminate information about the intervention--not until there is good quality evidence as to its effectiveness. This is the recognized scientific standard in non-autism areas.

Aldred et al. (2004) and the PACT shouldn't in any way be above scrutiny or criticism. For example, I can see problems in how Aldred et al. (2004) was designed (I mention one of them here), which may or may not be carried over to the PACT. But the pilot RCT and the resulting PACT generally demonstrate the only science- and ethics-based sequence when it comes to interventions and treatments: good quality research first, then think about spreading the word.


References:

Aldred, C.R., & Green, J. (2009). Early social communication
interventions for autism British Journal of Hospital Medicine, 70 (3), 143-145

Aldred, C., Green, J., & Adams, C. (2004). A new social communication intervention for children with autism: pilot randomised controlled treatment study suggesting effectiveness Journal of Child Psychology and Psychiatry, 45 (8), 1420-1430 DOI: 10.1111/j.1469-7610.2004.00338.x

Smith, T., Groen, A.D., and Wynn, J.W. (2000, 2001). Randomized trial of intensive early intervention for children with pervasive developmental disorder. American Journal on Mental Retardation, 105, 269-85. Erratum in American Journal on Mental Retardation, 105, 508. Erratum in American Journal on Mental Retardation, 106, 208.


Addendum: This post is included in the 116th Meeting of the Skeptics' Circle hosted by Beyond the Short Coat.

Alan Turing's brilliant essay

In 1950, Alan Turing wrote "Computing Machinery and Intelligence." This one short paper, exploring what came to be called the Turing test, continues to influence research and thinking across multiple fields.

Tyler Cowen and I have co-authored a new paper asking two questions. What does the Turing test really mean? And how many human beings (including Turing) could pass? Our premise is that some aspects of Turing's paper have not received sufficient attention:

Turing’s paper is rich and multi-faceted and we are not seeking to overturn all of the extant interpretations. We do wish to suggest that a potent and indeed subversive perspective in the paper has been underemphasized. Some of the message of Turing’s paper is encouraging us to take a broader perspective on intelligence and some of his points are ethical in nature. Turing’s paper is about the possibility of unusual forms of intelligence, our inability to recognize those intelligences, and the limitations of indistinguishability as a standard for defining intelligence. “Inability to imitate does not rule out intelligence” is an alternative way of reading many parts of his argument. Turing was issuing the warning that we should not dismiss or persecute entities which we cannot easily categorize or understand.

The facts of Turing's life enter into our argument, as does autism in many respects. Here is what we conclude:

It is possible that Turing conceived of his imitation test precisely because he had so much difficulty “passing” and communicating himself. In social settings these facts were seen as disabilities but in the longer term they helped Turing produce this brilliant essay.

Tyler Cowen is a professor of economics at George Mason University. His page is here. He blogs at Marginal Revolution; his post about our paper is here.

Definitely not autism advocacy: Accomplishments, capabilities, and rights

As an advocate for the rights of physically and mentally disabled people, I am acutely aware of how many people in our society consider the disabled to be childlike, helpless, hopeless, nonfunctioning and noncontributing members of society.

This opens a NYT opinion piece written by Evan Kemp. While Mr Kemp raises concerns about a specific fundraising event, his statements have much broader relevance.

For instance, Mr Kemp writes of the great harm that ensues when disabled children are denied the possibility of learning from successful disabled adults. The autism advocacy signature argument that disabled adults with achievements cannot really be disabled is not directly mentioned. But this is just another way to deny that successful disabled adults exist and it carries the same consequences.

Mr Kemp also notes how portraying disability as "overwhelmingly destructive" results in fear of disabled people and our consequent segregation from society. Then there is the issue of research priorities, some of which have the effect of supporting:

the damaging and common prejudice that handicapped people are "sick." As sick people, it follows that we should allow others to take care of all our needs until a cure is found.

Throughout his piece, Mr Kemp emphasizes that regarding disabled people as frightening and pathetic infantilizes us and leads us towards segregation, hopelessness, and dependence, to the great detriment of ourselves and others. Mr Kemp concludes with many recommendations, including:

Problems of economic waste, demoralization and segregation can be solved only when disabled people are depicted in the light of our very real accomplishments, capabilities and rights.

Mr Kemp's opinion piece was published almost three decades ago, in 1981. You can find it here. Autism advocacy continues to run in the opposite direction, rejecting and rolling back what we have learned about disability and human rights, going backwards in time to before Mr Kemp ever said a word.

Evan J. Kemp Jr. died in 1997. You can read his obituary here.

The highest autism prevalence ever?

Baron-Cohen et al. (2009) propose a world where there are 157/10,000 autistics. The authors used what they call a "novel approach" to autism epidemiology and this is worth a closer look.

The study is based on questions sent to schools about the UK SEN (Special Education Needs) register, and a survey, including a screening questionaire, sent to parents of students aged 5 to 9 years. Many schools, 66 out of 162, refused to participate. Of schools that agreed, 79 of 96, representing 8824 out of 11,635 children, provided SEN register information. Of the surveys sent to parents, 29%, representing 3373 out of 11,700 children, were responded to.

Participating schools located 83 children identified as autistic spectrum according to the SEN register. Of these, 10 children were assigned the specific diagnosis of autism (I'll call them "specific-autism"), while the rest were assigned other autistic spectrum diagnoses ("other-spectrum"). None of these diagnoses was verified by Baron-Cohen et al. (2009) in any way. These 83 SEN children constitute 94/10,000 of the proposed 157/10,000.

In the survey, parents were asked whether their child had an existing autistic spectrum diagnosis. The authors "confirmed," to their standards, through information from parents only, diagnoses for 19 children. Four others attended special schools and were assumed to be "confirmed" as autistic without any verification, for a total of 23. No information about how many of these children are specific-autism or other-spectrum is given. The 23 children were expanded into a prevalence of 99/10,000--impressively close to the SEN register figure.

The screening questionaire sent to parents was the recently re-named CAST (Childhood Autism Spectrum Test). All children reported by parents as scoring above one CAST threshold, and a minority of children scoring above a lower CAST threshold, and who were not already reported by the parent survey to be autistic, were invited for assessment. Eventually 77 children underwent assessment by ADOS, ADI-R, and clinical judgment. Eleven children were given autistic spectrum diagnoses--four specific-autism, seven other-spectrum.

So there are three sets of autistic children here: 83 SEN children, 23 diagnosis survey children, and 11 CAST children. The CAST children are considered to represent "unknown" or "undetected" autistics in schools, while the other two sets of children (SEN and diagnosis) represent equivalent samples of "known" autistics who have been located in two different ways from very nearly the same cohort.

Working from the diagnosis and CAST samples, Baron-Cohen et al. (2009) come up with a ratio of total number of cases (known plus unknown) to known cases, which yields a multiplier of 1.67. This in turn is applied to the SEN-based prevalence of 94/10,000, resulting in that grand total of 157/10,000.

And this all looks quite reasonable. Even the Neuroskeptic thinks so. But here are a few things I noticed.

The authors report no means of verifying whether the SEN and diagnosis samples are equivalent, an assumption on which the study depends. There is a dearth of demographic information, including the absence of any reported demographic measure common to both the SEN and diagnosis samples.

And for neither "known" sample (SEN and diagnosis) were any of the diagnoses verified either through direct assessment or direct access to records. Other recently reported UK epidemiology has not arrived at such a tidy agreement between prevalence of existing autism diagnoses and prevalence of autistics on the SEN register, nor at such a high prevalence using information from one or both sources.

Before they started sending out surveys, Baird et al. (2006) found that about 15% of 255 autistics with a "known" autism diagnosis (according to various health and clinical records) were not on the SEN register--under any diagnosis at all. This percentage would likely be higher if SEN autistic spectrum diagnoses only were considered.

Whether the 255 "known," pre-survey diagnoses from Baird et al. (2006) would be considered "confirmed" according to the standards of Baron-Cohen et al. (2009) is unclear to me, but in Baird et al. (2006), they add up to a prevalence of ~45/10,000. This is less than half the prevalence of what should be the comparable "known" diagnosis sample in Baron-Cohen et al. (2009).

Then there is Williams et al. (2008), where the authors had access to both medical and SEN records for an entire 14,062-child cohort. Using both kinds of records, they located a total of 86 "known" autistics for a prevalence of ~62/10,000. Of these, 36% were found in medical records only; 17% were found in SEN records only; and 47% were found in both.

Here again the "known" diagnosis total prevalence, derived from direct examination of complete records rather than selective responding to questions directed at schools or parents, is a much lower figure than found via both the SEN and diagnosis samples in Baron-Cohen et al. (2009). And Williams et al. (2008) show that diagnoses appearing in medical records, and diagnoses recorded via SEN, do not produce the same samples of autistic children--at least, not when there is direct access to these records for a complete cohort.

The original contribution of Baron-Cohen et al. (2009) is, according to the authors, the attempt to identify school-aged autistics who haven't been spotted as atypical. This is claimed to set their study apart from Baird et al. (2006). But 8 of the 77 children chosen for direct assessment through CAST screening in Baron-Cohen et al. (2009) were reported to be from "special schools" which, according to the authors, "require extensive assessment of individual special needs." Yet the authors suggest the CAST-screened sample was, in their words,

...quiet and cooperative at school and not difficult to manage and therefore teachers may not be aware that they have difficulties.

I don't know whether the CAST-screened special school children (who may or may not have been in the final 11) were on the SEN register, but doesn't this seem likely? Assuming for now (because you can only assume) that this was not under an autism diagnosis, in fact these were exactly the children screened for autism by Baird et al. (2006)--all children on the SEN register for any reason.

Further in this direction, Baron-Cohen et al. (2009) surprisingly report no way for the authors to verify whether the CAST-screened children were or were not on the SEN register, under any diagnosis--including autism. But the study hinges on the two samples, the SEN and CAST samples, being independent from each other.

One more thing to note is the actual composition of the 157/10,000. These numbers come from the reported data. Here goes:

11/10,000: "known" specific-autism children (SEN register)
23/10,000: "unknown" specific-autism children (CAST screening)
83/10,000: "known" other-spectrum children (SEN register)
40/10,000: "unknown" other-spectrum children (CAST screening)

If you add up the figures, you get the 157/10,000 total, of which 34/10,000--well within the range of previous findings--represents prevalence for the specific diagnosis of autism. The authors checked ratios of known to unknown autistic males vs females, and found no difference, showing that female autistics were not more likely than male autistics to be undetected. But the authors didn't check ratios of known to unknown autistics for specific-autism vs other-spectrum diagnoses.

If they did, they would find that for every known specific-autism child, there are about two unknown. The opposite is true for other-spectrum children, where about two are known for every unknown child. While only one-third of all specific-autism children are known, two thirds of other-spectrum children are known. Does this mean that children who meet criteria for the specific diagnosis of autism are much harder to spot, or are doing much better in school, than children who are popularly regarded as having "less severe" autistic spectrum diagnoses?

Or maybe the data are just very difficult to interpret. As usual, if I've made any factual errors, please let me know.


References:

Baird, G., Simonoff, E., Pickles, A., Chandler, S., Loucas, T., Meldrum, D., & Charman, T. (2006). Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: the Special Needs and Autism Project (SNAP). Lancet, 368, 210-215.

Baron-Cohen, S., Scott, F., Allison, C., Williams, J., Bolton, P., Matthews, F., & Brayne, C. (2009). Prevalence of autism-spectrum conditions: UK school-based population study The British Journal of Psychiatry, 194 (6), 500-509 DOI: 10.1192/bjp.bp.108.059345

Williams, E., Thomas, K., Sidebotham, H., & Emond, A. (2008). Prevalence and characteristics of autistic spectrum disorders in the ALSPAC cohort Developmental Medicine & Child Neurology, 50 (9), 672-677 DOI: 10.1111/j.1469-8749.2008.03042.x