Monday, January 03, 2011

Are autistic people lost in space?

ResearchBlogging.org
In one short paper, Elizabeth Pellicano and colleagues claim to demolish Simon Baron Cohen's systemizing account of autism. They also conclude that autistics' strong visual search and probabilistic learning abilities fail in large-scale space, ergo in the real world.

The press release starts by declaring that autistic children "lack visual skills required for independence" and does not exaggerate the claims in the paper, which merit a lot of scrutiny. So bear with me, this is not going to be short. First what they did (and didn't do), then what they found, then what it means.

1. What did they do? And what didn't they do?

Pellicano et al.'s purpose was to assess systemizing in autism by testing autistic visual search abilities in true-to-life large-scale space. And this research group has the set-up for it--a room-size visual search task. Except they ignored this possibility, instead choosing what they themselves call a test of "non-visually guided foraging."

In other words, Pellicano et al. do not include visual search tasks at any scale, so you can't draw conclusions from their paper about visual search abilities in autism.

From Figure 1, here is what Pellicano et al.'s foraging task looked like:



It took place in a 4m by 4m space arranged as shown. In two 20-trial blocks, 20 autistic and 20 nonautistic children (age ~11yrs) had to find which of 16 green floor lights, eight on each side of a midline, turned red when pressed. The children were not told that 80% of the concealed red targets were on one side of the room ("rich" side) with only 20% on the other ("sparse" side).

The task instruction was to find the red target as quickly as possible.

But search all you like, in this paper you will find no data about how quickly the participants found the targets. The researchers left out this available measure. They left out number of green lights hit before finding the red target. They left out how many first touches were on the "rich" vs "sparse" sides of the room.

They left out almost every measure they previously deemed important to assess performance on foraging tasks in large-scale space (e.g. here, here, here).

They seem also to overlook their previous systemizing-relevant (you would think) finding of no male-female difference in performance on the same foraging task (experiment 2 here).

Another of their own previous findings is that children's foraging performance is related to their performance on the Corsi block task, a measure of spatial memory. Autistic children don't necessarily show a deficit on this task (e.g. here), but in Pellicano et al., they did. If the goal wasn't simply to assess spatial memory, why not match groups on this ability?

2. What did they find?

Their main measure was sensitivity to probability in the foraging task which, ignoring their previous work, they defined as percent of visits to the rich side of the room. Here the authors report "no overall effect of group," never mind media reports (here is one). There was an effect for the first block of trials, where autistics visited the rich side for 45.50% of their touches vs 61.50% for nonautistics. In the 2nd block, the figures were 62.25% and 63.25%.

Two additional novel foraging measures involved total length of paths to the red target, per floor distance between green lights. These were assessed with highly sophisticated but conspicuously small-scale methods suited to eye-scanning and/or non-foraging pencil-and-paper tasks. So it doesn't matter how much time you take, or how much you have to twist and turn, or how many green lights you have to reach for. All that matters is length of path, and at this level the autistic children were judged significantly (a) less optimal and (b) less repetitive. Because you should ask, according to Pellicano et al., less repetitive means less systematic ergo is now yet another autistic deficit.

The final foraging measure was total revisits to green lights, where the autistics had significantly more in the second but not the first block. One autistic child racked up a big chunk of the autistic revisits (more than 3 SDs above the group mean), which might indicate an atypical understanding of or interest in aspects of the task. There is no information about group differences minus the interesting outlier.

The two non-foraging measures were the Corsi task where autistic children had a significant disadvantage, and an embedded figures task on which they performed significantly better than their controls (yes, interpreted as a global deficit, without evidence for same). Revisits--outlier included--correlated with the non-foraging measures in the autistic group. There is no information about these correlations for the nonautistic children or about any other correlations.

3. What does it mean?

Well first, it's an interesting task, even if it's not a visual search task.

But even if autistics totally failed (they didn't, and search all you want again, but you will find no rationale in this paper for the drop-off-a-cliff thresholds pushed by the authors), this task doesn't map easily onto the authors' sensational claims. These include that autistics can't find "shoes in the bedroom, apples in a supermarket, or a favourite animal at the zoo" ergo can't achieve independence.

Of course I want a whole lot more data, or an excellent rationale (none is provided) for not reporting most of it. And numerous possibilities were overlooked.

There's those other regularities in the distribution of red targets. There were only 40 trials, and each green light position was the target at least once. On the "sparse" side, each position would be the target exactly once (8 targets, 8 positions). The participants were not supposed to notice or be affected by this kind of regularity.

No one knows how autistics would have performed if given accurate task instructions (to take the shortest path, as measured by the authors, to the target). Maybe someone else can bring up motor differences, which plausibly are relevant to this "true-to-life" task. And I wonder how clear, for autistics, the task instructions were with respect to revisits.

Autistics should be notorious by now for noticing aspects of tasks that nonautistics don't (fantastic example at IMFAR last year), and for exploring more possibilities than nonautistics (examples here and here). Writing this off as a bad thing, as autistics being lost in space or some dire equivalent, is shortsighted to say the least.

If you're paying attention, autistics' "less repetitive" and speculatively more exploratory behavior--more aspects of the array are interesting to autistics, more possibilities and probabilities are explored--in "large-scale" space is a predictable finding. There is no reason to trash as useless (or harmful, and Pellicano et al. go there) multiple replications of excellent autistic visual search and probabilistic learning performance. One genuine conclusion is that systemizing, as conceived by Pellicano et al., and also by Baron-Cohen, is too rigid and limited a model to encompass how autistics atypically detect and explore environmental regularities.


Reference:

Pellicano E, Smith AD, Cristino F, Hood BM, Briscoe J, & Gilchrist ID (2010). Children with autism are neither systematic nor optimal foragers. Proceedings of the National Academy of Sciences of the United States of America PMID: 21173235


Postscript: You can find a large-scale space, true-to-life, real-world (and wonderful) embedded figures task performed by an autistic child here.

Post-postscript: This post has been included in the 83rd edition of Encephalon.

19 comments:

daedalus2u said...

What? Are you suggesting that they over systematized the over systematizing idea? But if they did over systematized that, wouldn't that make them ….? ;)

Michelle Dawson said...

It really is a remarkably rigid model, something Pellicano et al. picked up and ran with.

Also, isn't it an attempt to systemize self-propelled types (see the 2006 paper I link to), the sort of error the systemizing model attributes to autistics.

amy said...

They massively oversimplify the idea of systematisation, and then completely miss the point. A major cause of the problem, IMHO, is that most research makes the fundamental error of assuming it knows what it's looking at before it starts looking at it: that is, they start with a pathology model of autism, assume that it is by definition the manifestation of something inherently defective, and then set about trying to document the extent to which it's defective. They're looking to confirm rather than discover.
Not surprising, really, as almost all of the research is conducted by NTs, and that's the hallmark of NT cognition (and both it's greatest strength and it's biggest weakness): they gain efficiency by latching onto very simple patterns quickly. Unfortunately, there are very few genuinely simple patterns in this complex world, so in practice that means that they make a lot of assumptions, tend to see what they expect rather than what's actually there, and then compound the problem by not being very adroit at detecting when these things are happening.

Jon Brock said...

Conflict of interest alert (many of the authors are former colleagues). But I do genuinely think you're being a bit harsh here!

For one, I think it's important to separate what's in the various press releases (this explains why people with autism can't find carrots in the shop) with what the authors actually say in the intro, which is that foraging is more related to real life searching than traditional visual search tasks.

The main point is that, unlike in traditional visual search tasks, there is no visible clue as to the target's location. Actual path length in terms of number of buttons pressed before reaching the target is in large part down to blind luck. What's important is the strategy. Do you take the optimal (shortest) route given the fact that you have no idea where the target is?

I do agree though with some of your concerns. This one outlier who they remove from the correlational analyses but not from the group comparisons. Could he/she be driving the group effects? Also, the task makes pretty heavy demands on planning and spatial working memory and these weren't controlled for.

One other thing I noted. In the first block, the controls show a bias towards the more frequently rewarded side, suggesting that they've learned the contingencies more quickly. But the weird thing is that the controls then don't improve from block one to block two (even though there's massive room for improvement), whereas the autistic kids do improve and actually catch up. I'd be interested to know how consistent the bias was in the control group. My suspicion would be that some controls learned very quickly to always start searching on one side, but others didn't learn at all.

The main problem though is not with the study but with the theory. To me, systemizing is a hand-wavy description of 'male thinking' masquerading as a cognitive theory. So devising a test for it is near impossible. You could argue that in the foraging task there's a systematic regularity that people with autism should be good at picking up. But you could also argue that, because it's a fuzzy rule (ie sometimes it's wrong) and not an "If Then" rule they should have difficulty. Systematizing is so vague a concept that the 'theory' is pretty much unfalsifiable. Although this isn't something that's unique to this theory!

Michelle Dawson said...

Very interesting and much to think about, thanks Jon.

My own full disclosure, I've communicated a lot with one of the authors... who was on the jury of a successful disseration defence... which I attended because I worked with the student. If that makes sense.

Reading the press release before I read the paper, I thought it must be exaggerating. But then I read the paper. In my view, the press release faithfully reflects the paper, in tone, content, conclusions, carrots (or apples, in the press release), and so on.

The participants were not told to take the "optimal" route. What is quickest and what is "optimal" (...if this were the traveling salesman problem, which it isn't) are different things in this task, including or maybe especially because it is large-scale.

So it's inexplicable (my view!) that the authors chose novel small-scale criteria (not involving humans moving bodies through space) to define an "optimal" path in large-scale space. And then told the participants that what mattered was quickness.

In the examples of the not-optimal paths of autistics, one child found a path that took several fewer hits on green lights than the "optimal" path. This was considered wrong, even though we don't know what was quickest.

The autistic revisits outlier was included in the correlation analyses, though with his/her score trimmed down to 3SD above the mean, if I'm reading things right.

I completely agree that Baron-Cohen himself might predict that autistics would do poorly on the foraging task. And excellent point about the nonautistic children not improving on their sensitivity to probability over time (why I gave the figures!). This is very similar to Isabelle Souliere's recent study of category learning, which I linked to somewhere.

Sebastian said...

"Born to Forage - a Model of Autism' is a subtitle of a paper entitled "Mind as a Dynamical System: Implications for Autism".

From the point of view of mind as a dynamical system, it seems conceivable to see autistic people 'lost in space' given no apparent goal or task. Foraging works towards meeting a goal, completing a task, that one finds meaningful.

If a meaningful task is presented, autistics may surpass nonautistics on this skill with 'getting into an attention-tunnel' and hyperfocus at that. They would find a carrot in real life - and maybe forget about a carrot they may already have in their fridge over foraging one.

Nonautistics do that as well, but according to the monotropism hypothesis of autism, just not as much. According to the monotropism hypothesis, autistics and nonautistics differ in degrees only on their predisposition of ability to dynamically go into an attention tunnel - and forage, if they do.

1) III. Born to Forage - a Model of Autism
http://autismandcomputing.org.uk/mind.en.html

2) Attention, Monotropism and the Diagnostic Criteria for Autism
http://aut.sagepub.com/content/9/2/139.abstract

VioletYoshi said...
This comment has been removed by the author.
VioletYoshi said...

This makes no sense to me, as I know many people on the Autism spectrum excel at video games and computer games, which require extensive visual skills. Aside from video games, I'm also rather good at hidden item games like the I-Spy CD-Rom games.

If anything, I think people on the Autism Spectrum are better at visual search abilities, not worse. This could just be from my own personal experience, but there is a lot to be said for the number of Auties who find video games entertaining. You can't play video games, without a visual search ability. Particularly today now that everything is in 3D, for the most part.

Michelle Dawson said...

Not mentioned in my post but-- setting aside all else for a second--Pellicano et al. assume that strong computer-level or small-scale skills are useless in what they call the "real world."

Also, there's the irony (I think...) that interest in strong autistic abilities arose from observations at the genuinely-real-world and large-scale levels.

Sebastian said...

Assuming that "strong computer-level or small-scale skills are useless in .. the "real world" may serve as a reminder that abilities are judged by cultural values - like how useful they are considered to be by society.

For a philosophical-scientific debate, a definition of intelligence would need to be stated.

If the definition of intelligence/ability is "finding an apple or an orange" maybe better definitions could yet be found.

Emily said...

Wow, some really good points here! I'd been meaning to write about how bad this study was...thanks for reminding me!

Apart from the media leaping on any chance to discredit the possibility autistics might be good at something, what struck me most about the study was the quantity of confounding variables involved in the task. It's not just a more "realistic" visual search task.

1) If I understand right, the kids have to move to and touch the appropriate lights (visual-motor coordination as well as visual search). There are studies showing motor coordination is a problem for at least some kids on the spectrum.
2) It adds an implicit learning task, and some studies have found that may be a weakness for autistic people, too.
3) Even leaving aside the added skills involved, the spatial reference frame is different. If I understand the study properly, it's a change from absolute axes to self-referenced axes, which the brain processes somewhat differently in typical individuals.

Any of these factors, or others that I haven't identified, could explain why the autistic kids didn't show the usual "visual search" advantage in this task. So the study doesn't discredit their abilities to do 3D visual search, per se.

And of course, one can't claim that the data translate into an inability to find carrots in the store or socks in the sock drawer without at least a correlational measure of real-world functioning. (BTW, do autistics or their parents actually report difficulties finding things in the real world? If so, I was unaware). However, there are no real-world measures in the study.

And that's leaving aside the utter illogic of claiming that autistic kids aren't systematic enough while promoting the "extreme systematizing" theory.

Looks like another case of overstepping the data in search of a more "exciting" explanation...Jon, I'm surprised you don't disapprove of this study more.

Michelle Dawson said...

Thanks Emily. I'm glad you brought up motor abilities, which may be atypical in autism.

Re implicit learning, overall autistic children have performed well in well-designed tasks (i.e., they actually measure implicit learning) involving complex probabilities.

See the papers I link to in the original post, among others, and also this, which was mostly written in 2006 (published in 2008). In fact I should blog about this fascinating area of research some day.

By the way, there's no information about handedness in Pellicano et al. Also it isn't specified whether children could hit the lights with both hands, or just one.

Chris Richards said...

I'm new to the blog and find that the pressure of making my "first comment" is impairing my ability to spit the words out. In order to avoid deep philosophical wilderness, I'm going to force myself to make a painfully brief comment and then include a link to the philosophy behind it for those interested.

My immediate reaction to this study would be to laugh, if it were not professing to address something that the Advocacy Industry professes to be a grave threat to human civilization. What we're seeing is a total breakdown of good science. Rather than applying rigorous scientific method to their hypothesis, the researchers in question are designing an experiment that has nothing to do with the ability they wish to test and then claiming that this carefully designed simulation proves their hypothesis correct.

The problem is that this "junk science" (which would NOT go unchallenged in other areas) is not generally challenged when it comes to the area of medical, psychological, sociological, or behavioural science regarding child-rearing and education. We are told that child-rearing, education, and the proper development of children are too important to allow the science to be questioned when the reverse is true.

I think it's useless to dissect the precise errors of this study in too much detail because the forest is then lost for the trees. The basic fact is that the scientific methodology underneath it is broken.

So much for painfully brief.

Still shorter than my full thoughts: http://eclecticradical.blogspot.com/2011/03/developmental-disabilities-of.html

Lili Marlene said...

Hello! I'm not sure if you would consider participating in this type of thing, but I thought it was worth a try.
http://incorrectpleasures.blogspot.com/2011/03/bibliographic-infection-to-pass-on.html

nerkul said...

"BTW, do autistics or their parents actually report difficulties finding things in the real world? If so, I was unaware"

I could get lost in my own museum. I have a pet theory that spacial information is meaningless to me until it's sufficiently connected. There must be a term in topology for what I'm thinking of. All paths between A and B that don't cross over themselves are identical when all you have are two points.

Abstract that and generalise. If NTs lay assumptions over any search space they'll usually find the carrots / the other person's intentions quicker but swathes of the general solution including non-local optima will remain unreachable.

Therefore, Jon Brock, systematizing is such a blurry concept because it must be defined as not something else. Perhaps not assuming. These studies of autism are trying to pin down the sound, smell and feel of a lack of blindness. Get it?

Anonymous said...

I was wondering if you had read the report out of UCLA that indicated the autistic mark on the molecular level?
http://www.dailyrx.com/news-article/autistic-brain-mysteries-being-uncovered-13828.html
Brain mass differences which weren't especially new but that there was no lack of defining differentiation tween two major regions?

Anonymous said...

I'm a HFA with ASD (gotta love those acronyms). I don't see, don't hear, don't think, and don't experience the world the same as NTs do. So why in the world should I be judged by the same standards as any NT? "Junk science" tests and research like this are flawed from the very beginning because Autistic people are measured by the same standards as NTs. The same can be said for SBC and (junk) Theory Of Mind.

How about we turn things around and pit NTs against Autistics, only judging them by Autistic standards? I do think they would protest.

I am my own person and refuse to be judged by and against any NT standards! If people really want to know what's going on, stop ignoring us and talk to us!

ictus75

YANAGITAtokinori said...

If you are intereseted in me, You can study about me.
I want to become testee.
I have also lighter Autism and I graduated just from Junior high But I can grow up me as my teacher and I can teach me by myself.
http://woodwind.webseason.net/
http://www.youtube.com/user/YANAGITAtokinori?feature=mhee

Alejandro said...

Actually, the truth is that the task is on too large a scale. The Empathizing-Systemizing Theory states that persons with autism need to build up with the details and repeated observations to Systemize. They plunged participants into the deep end from the start and asked them to figure out a huge system of lights. And within a short testing time. There's no time for details or to learn the system for the test sample here. It is apparent from the abstract alone that this is not a valid test of SBC's theory. It is far too large a scale. Reading even the abstract, I had many doubts as to whether Pellicano et al. ever read SBC's theory themselves.
To trust this ONE paper... You'd be throwing away SBC's theory on a very poorly designed study. This paper is just a testament to why researchers need to do their background reading before calling for grants and participants.