Our paper is out! Go check. It is on eye-witness memory (how did I get here?)
Here it is. Elsevier, but open access.
Farhan is actually my one and only complete Doctoral Student(I was co-advisor). The work is kind of a second dip into a rather rich data-set with a follow-up study.
The main work (published here, no idea how open) asked what happens to your memory of a crime you witnessed as you keep talking about it over and over again, which is, of course, what happens if you are a witness. This is the study Farhan, and Carl Martin (and to some extent me) came up with:
A bunch of participants come in to watch a movie of a kidnapping. It is not a scintillating, well-cut, engaging movie. No. It is filmed from a single point of view, with a tad of zoom, and panorama, trying to mimic what things would look like if you were actually there, looking at the surroundings. The same movie has been used several times previous (the work is part of a larger program that Carl-Martin Allwood, the main advisor. )
They are then randomized into four conditions. In one, they come back 5 times, over several weeks. Each time, they meet a new person, and they discuss what happened in the movie. The new person (a confederate) has the job to ask questions about the event so they can understand everything that happened. In another, they also come back 5 times, but this group just retells the content of the movie to Farhan, who doesn’t discuss it at all with them. In a third, thrown in for ecological validity, they are given a schedule for when they are to talk about the movie with a friend or family member – different one each time. And, then it is the silent control: Do not talk about the kidnap movie! (The one and only rule).
Now all but the control group have talked about the movie, and all in slightly different ways. Time for the final two events. First, they come back and write down everything they remember from the movie. A day or two later, they come back and confidence rate each remembrance. This means that between the recall, and the next session their response has been segmented up into single statements, and a confidence scale put beneath. I think Farhan had some help here. They are all done in Swedish, and he is from Pakistan. (Yes, he speaks Swedish by now, but not then). Massive job.
We had also thrown what we called a “focused questions” task at them, which is where we asked more pointed yes-no questions about the film, and asked them to judge their confidence in their answers also.
You can read all about it in the paper, if you are interested.
We weren’t just interested in memory, but also in what is called meta-memory and calibration. Let me go through this for a bit, without doing the math. Meta-memory has to do with how well your confidence in your memory aligns with how correct your memory actually is.
For example, I run a lot of seminars that are obligatory, and students have to sign a roster as evidence they have been there. Occasionally a student gets back to me asking why they had not gotten credit (your name is not on the list). But, I was there, don’t you remember? I don’t. I may very well recognize the student, but the memory trace of who participated in which seminar disappears rather rapidly. I don’t remember whether or not they were present, and my confidence is low. But, some students stick out – usually because I know them from before. So, I do remember that both Rob and Drew participated in my two seminars on theory of science, even now, several years later, and my confidence in that my memory is correct is very high. These are examples of being well calibrated. When memory trace is iffy, my confidence is low, and when memory trace is high, my confidence is high.
Then we have the case of Em, who I was also completely sure had taken that theory of science course with me some years back, until she informed me that she had never been a clinical student. I actually have an episodic memory of her (or, most likely, someone that looks similar to her), and it is false. Here my meta memory was poorly calibrated, because I was quite confident about something that just could not have occurred. (I think that is the more reasonable explanation than the alternative that Em somehow is mistaken about her educational path).
Why is this interesting? It is because we tend to use our own and others confidence as an indicator of truth or correctness or certainty – depending on what you are looking for. In court, a confident witness is believed more than someone that is not confident. It is easier to trust someone who sounds like they know what they are doing. As an aside, that is a short-cut that can be taken advantage of, like so many Mamet movies shows.
But, confidence and accuracy does not always track that well. Apart from con-people. (The Dunning-Kruger effect is one of those).
One way that we try to investigate calibration between confidence and performance is to ask participants to perform (recall an event, tell a joke, solve a small problem, predict the weather), and then to ask them how confident they are they got it right. Now, most of us are going to feel confident about some of the performances and less confident about others, and this will vary along some kind of scale that is ordered, perhaps like: would bet my life on it, pretty sure, maybe, dunno, totally guessed. We usually ask them to do it in percentages or likert scales. Now, when you take all the performances together and bin them into those that were gotten right most of the time, those that were right 90% of the time, 80% of the time and so on this will be reflected in the confidence. When the performance is at guessing level, confidence should be at guess. When performance is near perfect, confidence should also be high. When in between, confidence should be in between. If you map it on a plane with accuracy as one axis and confidence as the other, perfect calibration is illustrated by a perfect diagonal.
If they don’t track, and confidence is used as a proxy for reliability it is clear why this becomes forensically interesting.
It isn’t like people are completely clueless, or that the confidence tracking is always off. In fact, in the first paper we published, participants were relatively well calibrated, although not perfectly so. At least for some of the free-recall questions. But, on the yes-no questions they were really lousy. I think the clearest interpretation of the results were that the participants were guessing, and that their confidence ratings suggest they knew they didn’t know.
At this point Farhan was diving into the research on central and peripheral pieces of information in recall. Loftus and Christianson has looked at this, as well as many others (you can read all about it in our paper). One thing he noticed with the responses was that participants seemed to be fairly good at the gist in the free recall, but were not terribly good at detail information, such as color of t-shirts and the like – all questions that were part of the fixed questionnaire. Perhaps this was the key to why meta-cognitive performance varied so much.
What he did, first, was to subdivide all of the statements in each recall into those that were what we first called forensically central, but changed to action information (throughout the lengthy review process) – what we then called forensically peripheral, but changed to detail information (t-shirt color, hair color etc), and then non-forensic information. The action and detail information is the kind of information that it is important to get from a witness, because you cannot get it any other way (what happened, how did things look), whereas the non-forensic is stuff that you can come back and find again (what the bus-stop looks like, how many buildings, etc.).
First, people recalled way more of the action information, and were better at it and better calibrated. But, of course, this was going back and sorting through data we had already collected for other purposes.
So, we ran a second, much smaller study, where participants again saw the movie, and then answered a new set of focused questions as well as confidence judged their responses. This time the questionnaire contained both detailed questions and action questions. And, again, we saw the same pattern. They were better at remembering actions correctly (and were more confident) than details.
Which is a small finding, and perhaps not entirely novel, but nevertheless neat.