On not noticing major problems with a study – and what that may say about the current incentive structure in Science.

I had an …interesting…experience the other day. I regularly go to the neuro journal club. Not because I do brains, but because it has lots of interesting members, and is quite active. (So, I felt lonely and under stimulated. Can even happen to introverts).

I had decided to present a paper I didn’t get to in the spring, so I had it all prepared, although as I read it I wasn’t sure what possessed me to present an fMRI study, as I’m not at all up on the methodological intricacies. It is this paper, by Snow et al. . Probably behind that dang paywall.

But, it was an interesting topic. They were comparing responses to real objects with responses to photographs of the same real objects. We spend a lot of time in psychology – from the neuro level up to the social level, and certainly across the developmental span – exposing people to images. For lots of good reasons, of course, like control, or ease, or well we don’t know shit about this, so let’s start simply. Pictures are alsopsychologically potent. Just look at the popularity of films, photographs, pornography. So, it is certainly not wrong to use pictures to probe minds.

But, pictures have some odd features also. We readily read them as 3-D and get fooled in fun and interesting ways that would not quite be possible (at least not for very long) in a 3-D world.

I began the talk with a picture of the Penrose triangle, and what Nicholas Humphrey call the Gregunddrum (after Richard Gregory who constructed it). (Go here to the Wikipedia page, and you’ll see both)

Much papers have been published discussing visual illusions (they do tell us something about the visual system), but, perhaps they are illusions because our visual systems have developed to deal with a 3-dimensional world. (My buddy Andrew has lots to say about that)

The researchers made use of new fMRI technology developed for research on haptic perception – a way in which you can tilt the head so you can actually interact with real objects. What they tesedt is called fMRA – an evident adaptation of the BOLD signal when you repeat a stimulus. You repeat a stimulus, the BOLD response gets attenuated. This has been replicated in a lot of ways, with all sorts of stimuli – all 2D, but with shadings, and implied textures, etc, and has been used to probe the visual system. In their experiment, they present participants with the real object, or a picture of that exact same object. Sometimes repeated (once) and sometimes not.

What they found, from my amateur understanding of the whole thing, was that the 2D pictures generated the expected fMRA, but the 3D items did not. And, then they proceeded to discuss why this may be so.

My friends, who actually do brains, and actually do studies with fMRI ripped it to pieces. They had collected their 13 subjects using different coils – but aggregated across (there were only 3 in the other coil, but still). The objects were of metal – most likely fMRI safe, but still metal, which is known to influence the recording. Why did they only do one repeat, and not multiple repeats? How come they did not control better where the participants kept their eyes (they claimed they could not use an eye tracker in that coil – fair enough – but where they kept their eyes may be very important). Was it really a good idea to only analyze the visual streams when maybe what is an interesting difference between pictures and objects may be in some motor area, indicating some kind of affordance (in the Gibsoninan way – the drinkability, the grabability, the interactability etc), and couldn’t they have come up with more interesting objects that would be more interesting to look at or perhaps grab. And, their conclusions, their discussion about what it could be was just implausible.

Lots of this I could not have noticed. I don’t do brains, as I said. And, the critique was of methodology; a background knowledge that you have if you do this research, but is invisible to you if you don’t. I’m highly aware of how important knowledge of methodology is, of course, and have insisted in my teaching that students actually do hands on work, because the problems – oh those devils – are in the details, not in the narratives. You can claim you have evidence for something, but you have to scrutinize the measures.

And, yes, it was a bit uncomfortable. I had – unwittingly – picked a paper that had an interesting topic, but was not very well executed and reasoned. But it was also very very very educational. For me and for the other participants in the journal club who may not yet be as sophisticated when it comes to this type of methodology.

Another interesting observation came up – which is in part why I put this story on this blog. I had originally prepared this paper together with another paper that discussed ideas about how to fix science (focus there was on how to deal with replicability and the file-drawer) – which resulted in a very interesting discussion on the issues facing science today.

As one of the researchers suggested, this paper may very well be a good illustration of the problems with the current incentive system. The speculation (and, of course, this is speculative, because we did not have the scientists present) is that this was a study that didn’t quite work out as expected – we all have tons of those: A pilot, that raised interesting questions, but where you really needed to fix the problems, and proceed with a new version probing the phenomenon further. Instead, a story is spun that could possibly be consistent with their weak results – that may not even have been the initial story, who knows. But, one that follows the current science narrative – looky here at our surprising result – published in a channel that Nature has open for kind of…lesser research. Because you must publish, and you must follow the narrative.

There would be a couple of paths that, for the purpose of science, would have been much more interesting. Either going back to the drawing board, learn from the failures (which we do all the time anyway), and come up with improved research. Or, possibly, if it must be published (and why not), publish it as a work in progress, with an enumeration of the problems, to be discussed and improved upon.

As it was, an unwitting person like me (and, of course, I’m a highly trained unwitting person, but just in a sister area – but with an interest in incorporating results from the brain sciences) takes the story to mean something that, well, it didn’t quite do. I needed the input from 4 of my more knowledgeable colleagues to understand the shortcomings. This is a problem, because it isn’t always that you have access to knowledgeable colleagues to help you go through and point out the issues with published results.


About asehelene

... because if I'm in a room with a second person, I want to be reasonably sure I'm the crazier one.
This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

One Response to On not noticing major problems with a study – and what that may say about the current incentive structure in Science.

  1. As authors of the paper in question, we are delighted that you decided that our paper was an interesting topic for discussion in your neuro journal club; however, would like to clarify some misunderstandings that arose in the group’s discussion.

    1) Regarding the criticism that the subjects were scanned with two different coil configurations, this would be a substantial concern if we were comparing between groups scanned in different coils. However, all of our statistical comparisons were within-subjects repeated-measures. Moreover, while all participants had the same coil (6 channels) used beneath the head, three had an additional coil (4 channels) suspended over the forehead. The additional coil provided additional signal:noise, especially over the frontal cortex. While we could have scanned the last three participants using the more limited configuration, it seems a contorted argument to suggest that the experiment would have been better had we limited signal:noise further.

    2) Regarding the criticism that the objects were made of metal: in fact, all of our stimuli were non-ferrous — they were made of plastic or paper (although this was not explicitly stated). As we explained in the methods section, we ran careful tests to ensure that none of the stimuli induced artifacts when a phantom was scanned.

    3) Regarding the criticism that we only had one repetition, fMRA studies using repeated vs. unrepeated stimulus pairs are very common (e.g., see Grill-Spector, Henson & Martin, Trends in Cognitive Sciences, 2006 for a review). We argue that ‘event-related’ approaches have merit over long blocks of repeated or unrepeated stimuli because such designs suffer less from concerns relating to differences in attention, boredom, or stimulus predictability. Moreover, there are many methodological constraints that arise from working with real objects and at the time of that experiment, we could only change real objects manually within a very tight space (bore diameter = 60 cm). Thus we needed the long inter-trial intervals to enable the experimenter to change the stimulus pairs from trial to trial. We agree some experiments on real-world objects would benefit from long stimulus sequences and we have built a very elaborate piece of apparatus called the Delivery of Real Objects for Imaging Device (DROID) to enable this. One motivation for building this device was our intriguing findings from the paper in question.

    4) Regarding the criticism that we didn’t control fixation, as discussed in the paper, participants were instructed to maintain gaze upon a fixation point throughout the experiment but in-scanner eye tracking is simply not possible in the configuration used. We covered this issue at length in our discussion.

    5) Regarding the criticism that only the visual streams were analyzed, this is patently false. We performed a voxelwise analysis of group data that would have revealed activation in any region of the brain that showed an effect for any of our contrasts, including regions such as motor and somatosensory cortices. There simply were no effects for real objects in these regions; the problem was not that we didn’t look for them.

    6) Regarding the criticism that we should have used “more interesting objects”, the objects we used are typical of past studies that have found repetition effects for pictures of objects (Grill-Spector et al., 2006). We are now following up on the hypotheses generated from the study in question using tools (which have strong action associations) vs. non-tools (which have weak action associations).

    7) Regarding the criticism that our conclusions and discussion were “implausible”, we would be perfectly happy to discuss this further if you could provide more tangible critiques.

    We wanted to make this result – which we ourselves found both surprising and intriguing — available to the scientific community without too much delay. We are now running a series of experiments to test the working hypotheses we put forward. Pilot data are corroborating the main idea that the brain processes real objects quite differently than pictures of objects.

    Working with objects is extremely challenging and our first study in particular had technical limitations. However, if one takes the “long view” of science, what is important is not the minutiae of one study but the accumulation and corroboration or refutation of interesting ideas. Our hope in publishing this paper was to put forth a provocative suggestion that in our reliance on maintaining extreme control over stimuli, the field may have sacrificed ecological validity and missed out on the sorts of interesting questions that sparked your interest in our paper.

    Jody Culham and Jacqueline Snow

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s