What a difference a century makes – On March 8. For those who get gloomy.

My grandmothers were born 1898 and 1902, when women were yet allowed to vote in either Sweden or Norway. When my grandfather talked to one of the head political honchos in the small village where he was the post-master, for a loan (or a signature for a loan) so he could pay for higher education of his eldest daughter, my mother, this head honcho thought this was just a waste of money, spending it on educating daughters who could, you know, become hairdressers or something. Happily my grandfather didn’t agree, and my mother got her teaching degree.

One day, late 50’s, early 60’s, the headmaster of the school where my mother and father were teaching (and, good friends of theirs) saw my mother wearing pants to work, and he asked if she was having an “idrotts dag” – days we have in Swedish schools where all students do some sports, or at the very minimum spend some time outdoors. Women just did not wear pants to work.

A couple of years ago, when I was running a seminar where we were discussing a paper where they investigated how dress-style influence judgments of competence, warmth, and other measures, a few of the female students reacted strongly to the fact that business-attire for American Women in the late 90’s meant skirt and pumps. As one of them said, some of us just feel awkward in a skirt!

The times they are a’changing.

Jill Ker Conway, contemporary with my parents, grew up in the Australian Bush, and had a stellar academic career in the US and Canada. In her books, Road from Coorain and True North, especially true north, she chronicles the blatant and hostile sexism she encountered during her academic career. I have encountered none of that.

Too bad that that cigarette company took that slogan, because we have come a long way.

Doesn’t mean it is all roses. Doesn’t mean there aren’t things to do. Doesn’t mean we are now safe from sliding back to where women are routinely seen as possessions, chattel or high-prized pets. But we are, now, in a place in the west at least where we clearly can say that if no women are present it is because the dudes in charge just didn’t look carefully enough, not because there are no competent women.

And, yes, sometimes you do have to remind them that, along with the dudes, there were also some sharp women present*, and some men have not yet to get the memo that being a letch is just so passe.

But, whenever I’m tempted to be depressed, I look back just a tiny amount of time, on the circumstances for the women who came before me – some of whom are still around (like my mom, and Jill).

Not bad. Not bad in slightly more than a century.

* Scroll down to Bobbie Spellmans comment, and David lists all of them. Impressive bunch. Or just read the whole post with the comments. Definitely worth your time.

Posted in Uncategorized | Leave a comment

Reflections on teaching and science.

I’m reflecting on teaching. Which in itself can be good, as long as it isn’t so much reflection that you end up in a mirror maze with copies of your thoughts to infinity and beyond.

But, in this case, the reflection was prompted by a post from John Horgan, where he speculated that perhaps the current reproducibility crisis could be traced to practices in teaching the labs. Supposedly students would fudge data obtained to get it correct on reports, in order to get good grades.
Fudging data-points is not something that would get you better grades in psychology. It really isn’t a point in doing so for grade purposes (at least where I have been), because we know damned well how hard it is to get significant results so grading is never based on whether your experiment worked out or not. I actually had one of my profs telling me I could get my PhD based on a non-significant dissertation, as long as the reasoning and method was reasonable. (With my 12 x 30000 data points, that was not a problem!). I usually tell my own students something similar.

I have engaged in some sinful things like fishing for p-values (usually with the embarrassed caveat that this is not really how you should do it) so that my undergraduate students can practice writing up results section with something that we can pretend is meaningful. And, yes, those beautiful euphemisms for p > .05 (which surely God loves just as much) that Matthew Hankins so nicely collate for us – I’ve used them.

I have wondered about the role of teaching in the problems abounding, contemplating, as I am right now, on its low status. In coming up, though stats was usually taught by people who had done it for a long time, methods were relegated to the grad-students and the recent docs who needed something to do. In fact, at my grad institution, the methods and APA style course was handed to all the graduate students in year three, as part of their education. Actually, education wise, that was nice for us. The rest of methods were very much learned on the job as a lab assistant. Informally, it was suggested this was done because faculty did not want to be stuck with methods, and why not make it a learning expericens for the new grad students? (True? Not sure). Doesn’t say much about the value of teaching, especially about the basic mechanisms of doing research.

The other day, “Simply statistics” (in this post) stated that the squabbling agains p is misguided. The problem isn’t really the p-value, and that one should use effect sizes or other statistical procedures. The problem is a lack of analytical skill, and a lack of teaching analysis. Statistics is, after all, just a tool. Something that should help us keep track of our data, and sort out our results. It cannot provide a magical line that demarcates results from non-results. But, this is how it has been used.

What he suggests is that, beyond statistics and probability, there need to be teaching on how to analyze – the part of the theory and ideas that are not just in the statistical package. This I agree with. I’d like to add, it would be good if this was maintained throughout the schooling. Statistics and analysis of data is a skill that must be trained and repeated, not something to suffer through, get a good enough grade, and then forget. To do so, of course, you need to plan the teaching in a more overarching way. Did students learn statistics and methods last semester? Great, now that we have an area course, make sure that students keep using their fledgling skills in reading papers, planning research and analyzing results in any possible project. Do that in the next course also. This is not something you acquire overnight and once and for all.

You start, like pedagogic texts will tell you, learning some rule about “how it is”, because, really, that is probably the best way to start. A rule – we look for rules (if I believe Tomasello and Bloom, and I do), and tend to treat them as laws. Until, in the next stage, you confuse the students with exceptions. Slowly, you can build the skill necessary to understand and analyze what acquired data may actually mean, which is more than pushing buttons on SPSS. This learning does not end. I’m still at it, and my PhD is over a decade old.

Teaching Takes Time. And Skill.

In the chapter on culture and cognition in Viren Swami’s Evolutionary Psychology, which I used for my course in Evolutionary Psychology, they spend a lot of time trying to understand the evolution of the learning mechanisms that have allowed us to accumulate and transfer our technical knowledge across time, distributed across people. There’s imitation, there is emulation, and there is direct teaching. This teaching has a cost, to the teacher. That is, the teacher must take the time to create learning situations that fit with the stage of the learner. Having gone through the process is not enough. We forget how we learned, and how it was to not know – the Curse of Knowledge. This teaching exist in other species also. I was charmed by the sequence meerkats teach their young to hunt scorpions: from dead, to disabled, to fully functional prey.

Also, you cannot transfer your current level of knowledge directly to the students, even at college level (until Gibson’s Sim Stims become a reality, which is never). It has to be adjusted to allow learning to occur. This adjustment is in itself a learning process. So loopy.

The time you teach, and prepare to teach to those new in the field is time you cannot spend working on other projects, or keep abreast of the literature, or write grants. There is a possibility for synergy. The act of teaching helps with consolidating and deepening knowledge. It is well known, of course, that in order to really learn something, the best way is to teach it. (One of the reasons I keep designing new courses).

But, it doesn’t bring glory, it doesn’t bring grants, it doesn’t bring status, as it is set up right now.

But, perhaps John Horgan is right, that this is what has undermined the quality of the science that is done, because the new scientists do not get sufficient training to be able to do really good work (which is such a waste, considering the passion and cleverness of this group), and the glory comes with publishing papers. (I just read through this paper by Charles Lambdin which bring this up and more. Either Fred Hasselman or Hans Ijserman linked that in on Twitter. Paywall, alas).

Another lesson from evolutionary psychology (and also Richerson & Boyds work on cultural transfer) is that knowledge can become watered down and disappear if it is not carried forward. In their example, from Tasmania, it was because the population shrank, and key knowledge-bearers died without having transferred their knowledge sufficiently. The population in science is not under threat of shrinking, but the lowering of status of those involved in the knowledge transfer (adjuncts anyone?) may have a similar effect. (Here I’m wildly speculating – I’d like to think up a way of empirically test that conjecture).

I teach too much, at least from the perspective of current incentives, and research and publish way too little, and I sometimes feel my effort is invisible. In fact, I have no clear idea whether I’m any good at it. I like thinking about what students of psychology need to know and need to learn in order to be astute consumers and possibly producers of knowledge, and then to stick that in my courses. There is an element of self-interest here, because I’d like to feel like I’m valued, and teaching constantly gets dissed.

But, if not enough time is set aside to properly train the new generations – and this can be done in an apprentice setting – we will produce what?

Posted in Uncategorized | Leave a comment

Some continuous musings on emotion.

I’m trying to think about emotion. Or think about how to think about them. What is a good, fruitful conceptualization.

Perhaps, first, in line with Joseph LeDoux one should really jettison the term “Emotion” as it is more of a folk-psychological term, which muddies how you think about it. (Emotion Review have had issues recently discussing this). Of course, the folk psychology of emotion is itself interesting – lots of cultural variation. Swedish doesn’t have a word for emotions. We use feelings. Covers roughly the same thing. There’s also variation across history.

The first Emotion Review of 2014 has a section looking at James legacy. If you have read any intro book on psychology the past 15 or so years (when they even had anything about emotion), you would have heard about the bear, and running, and that was the emotion, whereas Cannon said no, it is all in the brain – all this is covered in the issue, in its proper historical view. It always irritated me that these cartoon versions of their theories were something that we taught our undergrads in what would, perhaps, be their only encounter with emotion research, considering that the state of research is very different, and has been for a very long time. Not wrong to look at the historical, of course, but it was not presented as historical. In our book it was presented as “theories of emotion”. I would have liked them to, instead, look at categorical vs. dimensional accounts, and expand on the appraisal theories.

But, an account by Phoebe Ellsworth  in the Emotion Review issue stuck with me for a couple of reasons. (Title is “Basic Emotions and the Rocks of New Hampshire”)

She worked with Ekman and Friesen on their basic emotion program. She had come into this, viscerally convinced that there were similarities in emotional experience and expression across cultures from some films she had seen.

At the time, in text books she said, textbooks used to display picture of faces that were distorted in a grimace, asking if one could tell what the person was feeling. Then they would feel the entire picture in context and reveal that it was some winning moment and go ha-HA you thought you could read emotions from faces, but neener neener you are soooo wrong. (OK, I’m exaggerating). But, this was mainly the result from the Landis work that no IRB would approve these days.

So nothing is new under the sun, because I have seen things like that published in the 2000 – showing how the body informs, and the face is not necessarily the man informant, as a refutation to the basic emotion in the face thing. Sure, context, body, face, all matters in how we interpret what someone is feeling, and that may not entirely be signaled by the face (I’m thinking studies by deGelder and Todorov, and similar). And, yes, that it is important to keep in mind. But, I just found out that is not new…. Why is research memory so damned short?

She also gave a bit of insider info on how they were preparing for the cross-cultural work on expression recognition, and how they selected the expressions. The six is, in some ways, an artifact of the time and resources they had. They had enough pictures of these expressions that were viable, but not enough of other theoretical expressions. (Much of them were derived from Silvan Tomkins ideas).

And, yes, even I know that what is supposed to be the basic emotions vary, and vary across the researchers. You can find that in tables in emotion textbooks. I read that in papers during the late nineties. You can find it on the web. There are some overlaps always, and some odd ones.

Also, historically, what are the emotions and what are not seems to have shifted. Is love? Shame? Awe? Fear?  emotions? Why? Why not?

I’m not happy with either the categorical or the dimensional, or the various appraisal theories. But, at least they are theories. I think they are frequently compatible, neither of them seem to win.

Really, what I want is a more dynamical system account of emotions. But, I may muse on that one later.


Posted in Uncategorized | Tagged | Leave a comment

Through the looking glass into an oddly analyzed clinical paper

My curiosity turned me down a dark alley of oddly reported and interpreted statistics. It has fancy things in it, like effect sizes, and even confidence intervals, and “Wilcoxon sign tests” in it, and claims of large effects. Perhaps I’m not sophisticated enough to understand its meaning, but to me it seems more like a fun-house out of the twilight zone, or the research mirror world of that old post-modern bs academic writing, with statistical concepts as the lily-pads rather than obscure hipster-words.*

What lured me was a second posting of a data-table from Keith Laws, where he lamented that he still could not make sense out of it. So I re-tweeted. Why not. A click is easy. Daniel Lakens responded, and I was now the witness to a conversation suggesting that this was some crappy analysis. Yes, I had looked at the data-table at one point before, but nothing of value had turned up in my head, so I had nothing to say without looking at the paper. And as schizophrenia and CBT and clinical trials are way off even my meandering paths, I had simply refrained from that. But after the back and forth for a bit (and being copied in on a slur accusation, and, as usual giggling about some remark from DrNeil Martin) I just had to go look. Keith had kindly linked in drop-box copy for anybody’s perusal.

For you, I give you the title and a link to the abstract – it is paywalled. (I just don’t want to use the drop.box copy for a blog). High-Yeld Cognitive behavioarl techniques for psychosis delivered by case managers to their clients with persistent psychotic symptoms. An exloratory trial.

It is a strange world. I wasn’t sure if I should giggle, or possibly wonder if I had misunderstood something about the statistics they were doing, or getting deeply depressed that this passed peer-reviewers, considering that clinical psychology is the one area where we have the most realistic opportunity to both do good, and to do great harm.

From my understanding, having looked at it now in my rabbit-grazing way, the question the group was interested in is whether it is feasible to train Case Managers to deliver a particular kind of Cognitive Behavioral Therapy to their psychotic patients.**

They assessed 69 patients for eligibility for the treatment, and ended up with a total of 38. They also trained 13 case managers to deliver the treatment. Training took five days, and all case-managers received weekly supervision during treatment. I can’t find how long the treatment lasted when I look through it this time, but I assume it took place over several weeks. Patients were assessed on a number of functions at base-line and at the follow-up. The scales or whatever protocols they use for assessment are unknown to me, so I have no means of knowing if they are good, but what they assess seem to be things that it would be reasonable to assess. This is standard, of course. I wouldn’t bother to explain how the IAT works if I was submitting something to a Social Cognition Journal.

So far so good. Nothing is remarkable or out of the ordinary from this non-experts point of view. Doing clinical research seems like a massive job, and I appreciate that there is a great deal of difficulty doing it well.
But, now comes the analysis. First there are four figures of histograms with error bars, showing before-and after scores on the different measures. I can’t find any explicit inferentials about these results in the text, although they claim to have done both t-tests and Wilcoxon sign tests (I looked the latter up on Wikipedia. Both types seem reasonable for before and after assessment). But, just looking at the graphs it is clear that inferentials aren’t really needed because it doesn’t look like anything happened. The means tend to be somewhat lower in the “after” but the standard-errors are rather large and overlapping (I even double checked my Cumming New Statistic book to make sure I understood this). It really looks like the intervention has had no effect whatsoever, at least when you aggregate across all 38 participants, which I assume they did. They claim that no data was missing.

They also do an “effect size analysis” using “Cohen’s d-methodology” referring to Cohens entire 1988 book. Well, fair enough, but I wanted to know if they meant something different by this than we do when we do t-tests and calculate effect size. I gather that this is what they are listing in that table that Keith tweeted in that he could not make heads or tails out of, and that Daniel think is just horrible, and I think resembles a sinister hall of mirrors, or possibly a run-way made of bamboo in the south eastern war theatre in the late 40’s.

Now effect sizes are nice, of course. In this case they run from the middling to large, and also include a few negative ones (suggesting that things got worse). But, one must remember that with only 38 participants, effect-sizes tend to be inflated, as the handy chart in Dan Simon’s blog shows (simulations of effect size estimates where the true effect size is zero – you can do that when you simulate).

The table also shows confidence intervals. I take it that it is for the effect sizes. I looked up how you calculate confidence intervals for effect sizes to try to make sense of this, and you can do it of course. It is a bit trickier than just calculating confidence intervals for estimated means – involving non-central non-symmetric t-distributions, but it can be done, and evidently there are nice R-algorithms for it.

The confidence intervals are large, and all go from a number less than zero to above. That is, for every single effect size, the “no effect whatsoever” is still within the possible estimate. There are likely a couple of typos there also – two confidence intervals are identical. One starts at the same number as the estimated effect size. (It is not the only place where the copy editors and proof readers missed. Figure 5 lacks labels on the axes.)

None of this is backed up in the analysis section, which is all of 5 lines long, naming a number of tests they claim they performed. Of course, looking at the graphs and the tables it really looks like there isn’t much to write about anyway, because I doubt anything would have acquired that magical p<.05 level, but it would have been nice to actually see the values, and the df’s and all that in numbers, because my feeling right now is that I’m not sure the researchers know what they are doing, at least not statistically.

I wouldn’t have let this kind of reporting pass in my undergraduates (who have a good excuse feeling wobbly about stats). It should be a fairly straightforward analysis with a before and after group.

Sure, perhaps it isn’t appropriate or feasible to write down all those numbers in all cases. Right now I have a paper out where I don’t report the inferentials, and only show means with standard errors, but the journal is one focusing on film, and is mostly using their kind of qualitative analyses. I wanted to illustrate that we can induce emotions with films, but showing the data was more supplementary than all the other things I wrote about.
I’m not sure this paper can get away with the excuse, especially as it starts its discussion claiming that the results showed large effect sizes (never mind those confidence intervals), and that the intervention showed good, significant results never mind that the table all suggests that nothing happened, or at least that if something happened it is so overwhelmed with that pesky crud-factor the signal doesn’t make it outside the noise.

They don’t look at the training of the case managers, which I thought was part of the question. There are a lot of claims, but they don’t seem anchored in the data they show, and none of that should be particularly difficult to show.

And, yes, sure, they are aware that the sample is small, and there is nothing that seemed control-like, but they are confident they have shown some kind of feasibility for training case managers to deliver this type of therapy. It seems akin to reading palms.

Now, why, oh, why, did I dive into this sinister mirror world, when I don’t do clinical? I should have stayed in the fun-house of small n counter-intuitive findings in social psychology. We can snark and replicate one another, and nobody’s mental health is in danger.

Still I wonder, did I miss something? Is it some analysis method I don’t understand (yes, there are, plenty of those of course), but that is pertinent to this one?

Am I, in the twilight zone?

* A quote from Katha Politt that I read in a Socal book/article late last milennium keep sticking with me as a perfect illustration of mindless attempts at influence by using particular keywords: A frog jumping from lily pad to lily pad. I finally found where it is from with a little bit of google-fu. The article is called Pomolotov cocktail.. The Frog quote is at the very bottom.

**Never mind that CBT for schizophrenia seems to do very little based on this recent meta-analysis. There is a place to play with the data for the meta-analysis, but for the moment I have lost that link in my twitter flow.

On edit: I had hoped Keith would see that one, and provide me with the link, and he kindly did. Here. Go play with meta-analysis data.

Posted in Uncategorized | Leave a comment

Confidence is in the Action, not the details. (On our recent paper).

Our paper is out! Go check. It is on eye-witness memory (how did I get here?)

Here it is. Elsevier, but open access.

Farhan is actually my one and only complete Doctoral Student(I was co-advisor). The work is kind of a second dip into a rather rich data-set with a follow-up study.

The main work (published here, no idea how open) asked what happens to your memory of a crime you witnessed as you keep talking about it over and over again, which is, of course, what happens if you are a witness. This is the study Farhan, and Carl Martin (and to some extent me) came up with:

A bunch of participants come in to watch a movie of a kidnapping. It is not a scintillating, well-cut, engaging movie. No. It is filmed from a single point of view, with a tad of zoom, and panorama, trying to mimic what things would look like if you were actually there, looking at the surroundings. The same movie has been used several times previous (the work is part of a larger program that Carl-Martin Allwood, the main advisor. )

They are then randomized into four conditions. In one, they come back 5 times, over several weeks. Each time, they meet a new person, and they discuss what happened in the movie. The new person (a confederate) has the job to ask questions about the event so they can understand everything that happened. In another, they also come back 5 times, but this group just retells the content of the movie to Farhan, who doesn’t discuss it at all with them. In a third, thrown in for ecological validity, they are given a schedule for when they are to talk about the movie with a friend or family member – different one each time. And, then it is the silent control: Do not talk about the kidnap movie! (The one and only rule).

Now all but the control group have talked about the movie, and all in slightly different ways. Time for the final two events. First, they come back and write down everything they remember from the movie. A day or two later, they come back and confidence rate each remembrance. This means that between the recall, and the next session their response has been segmented up into single statements, and a confidence scale put beneath. I think Farhan had some help here. They are all done in Swedish, and he is from Pakistan. (Yes, he speaks Swedish by now, but not then). Massive job.

We had also thrown what we called a “focused questions” task at them, which is where we asked more pointed yes-no questions about the film, and asked them to judge their confidence in their answers also.

You can read all about it in the paper, if you are interested.

We weren’t just interested in memory, but also in what is called meta-memory and calibration. Let me go through this for a bit, without doing the math. Meta-memory has to do with how well your confidence in your memory aligns with how correct your memory actually is.

For example, I run a lot of seminars that are obligatory, and students have to sign a roster as evidence they have been there. Occasionally a student gets back to me asking why they had not gotten credit (your name is not on the list). But, I was there, don’t you remember? I don’t. I may very well recognize the student, but the memory trace of who participated in which seminar disappears rather rapidly. I don’t remember whether or not they were present, and my confidence is low. But, some students stick out – usually because I know them from before. So, I do remember that both Rob and Drew participated in my two seminars on theory of science, even now, several years later, and my confidence in that my memory is correct is very high. These are examples of being well calibrated. When memory trace is iffy, my confidence is low, and when memory trace is high, my confidence is high.

Then we have the case of Em, who I was also completely sure had taken that theory of science course with me some years back, until she informed me that she had never been a clinical student. I actually have an episodic memory of her (or, most likely, someone that looks similar to her), and it is false. Here my meta memory was poorly calibrated, because I was quite confident about something that just could not have occurred. (I think that is the more reasonable explanation than the alternative that Em somehow is mistaken about her educational path).

Why is this interesting? It is because we tend to use our own and others confidence as an indicator of truth or correctness or certainty – depending on what you are looking for. In court, a confident witness is believed more than someone that is not confident. It is easier to trust someone who sounds like they know what they are doing. As an aside, that is a short-cut that can be taken advantage of, like so many Mamet movies shows.

But, confidence and accuracy does not always track that well. Apart from con-people. (The Dunning-Kruger effect is one of those).

One way that we try to investigate calibration between confidence and performance is to ask participants to perform (recall an event, tell a joke, solve a small problem, predict the weather), and then to ask them how confident they are they got it right. Now, most of us are going to feel confident about some of the performances and less confident about others, and this will vary along some kind of scale that is ordered, perhaps like: would bet my life on it, pretty sure, maybe, dunno, totally guessed. We usually ask them to do it in percentages or likert scales. Now, when you take all the performances together and bin them into those that were gotten right most of the time, those that were right 90% of the time, 80% of the time and so on this will be reflected in the confidence. When the performance is at guessing level, confidence should be at guess. When performance is near perfect, confidence should also be high. When in between, confidence should be in between. If you map it on a plane with accuracy as one axis and confidence as the other, perfect calibration is illustrated by a perfect diagonal.

If they don’t track, and confidence is used as a proxy for reliability it is clear why this becomes forensically interesting.

It isn’t like people are completely clueless, or that the confidence tracking is always off. In fact, in the first paper we published, participants were relatively well calibrated, although not perfectly so. At least for some of the free-recall questions. But, on the yes-no questions they were really lousy. I think the clearest interpretation of the results were that the participants were guessing, and that their confidence ratings suggest they knew they didn’t know.

At this point Farhan was diving into the research on central and peripheral pieces of information in recall. Loftus and Christianson has looked at this, as well as many others (you can read all about it in our paper). One thing he noticed with the responses was that participants seemed to be fairly good at the gist in the free recall, but were not terribly good at detail information, such as color of t-shirts and the like – all questions that were part of the fixed questionnaire. Perhaps this was the key to why meta-cognitive performance varied so much.

What he did, first, was to subdivide all of the statements in each recall into those that were what we first called forensically central, but changed to action information (throughout the lengthy review process) – what we then called forensically peripheral, but changed to detail information (t-shirt color, hair color etc), and then non-forensic information. The action and detail information is the kind of information that it is important to get from a witness, because you cannot get it any other way (what happened, how did things look), whereas the non-forensic is stuff that you can come back and find again (what the bus-stop looks like, how many buildings, etc.).

First, people recalled way more of the action information, and were better at it and better calibrated. But, of course, this was going back and sorting through data we had already collected for other purposes.

So, we ran a second, much smaller study, where participants again saw the movie, and then answered a new set of focused questions as well as confidence judged their responses. This time the questionnaire contained both detailed questions and action questions. And, again, we saw the same pattern. They were better at remembering actions correctly (and were more confident) than details.

Which is a small finding, and perhaps not entirely novel, but nevertheless neat.

Posted in Uncategorized | Leave a comment

2013 in review via WordPress this time (so just the blog).

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

A San Francisco cable car holds 60 people. This blog was viewed about 3,400 times in 2013. If it were a cable car, it would take about 57 trips to carry that many people.

Click here to see the complete report.

Posted in Uncategorized | Leave a comment

Things I actually did 2013.

I went back and looked at my “things I may do” post from just about one year ago.

What did I do? Besides teaching. (Which I did a lot).

Well, I got more involved in Fixing Science. In February I went to Brussels for the “beyond questionable practices” symposium. Got to hang out with Daniel Lakens, and said hi to Brian Nosek, and had generally a good time.

Then in June I went to Nijmegen for their Robust Sciences symposium. Spent even more time with Daniel this time, and got to meet Rolf Zwaan. (Plus lots of others).

As a result, I’m adjusting how I’m teaching my masters students so they will start out beyond Questionable Practices. We used the Open Science Framework. Ran into some snags, but I will definitely continue that.

I became part of the Open Science Bloggers. So far I have only contributed one post but I have done a few things like peer reviewing and definitely promoting as best I can.

I was asked to be on a special editorial team for Perspectives of Psychological Science, which made me go both Wayne and Garth and Lawrence of Arabia*. I will say no more, as it is on-going.

I blogged a lot in the spring, and not so much during fall.

This has also been the year of twitter for me. I’m having a lot of good conversations and good contacts through it. I am fond of on-line sociality. I have lots of “invisible friends” since close to a decade and a half. Funny thing, I’m chatting more with academics in other countries than with my own colleagues. We are too busy in our offices. (And, I adore my colleagues).

It is all very inspiring.

Research wise? My film group collaboration resulted in an invitation to a weekend symposium on film and quantitative measures. It was held at the Humanities Lab, which is a much nicer lab than we have anywhere in Psychology. They have an eye-tracking room! 20 eye trackers with computer screens. Among all sorts of other nifty things. So, the Film people invited a whole bunch of their colleagues that would be interested, including Anabel Cohen from Canada (who trained under Hebb I found out) who does movies and music, and I got invited into it on the basis of having used both movies and music to induce emotions.

I was mildly non-plussed first, because that was just our manipulation. Then I realized that this is actually very interesting. So, I presented what we did when I was a grad student, and put together a whole bunch of info on movies as emotion inducers. It is going to turn into a paper, as all the presenters at that symposium are going to contribute to a special issue of Music and the Moving Image. I did my updated version right before going on vacation.

I’ve also, from this, become part of a group of Danish and Swedish researchers interested in using quantitative methods for film research. It is very exciting, and I hope I have more time this spring thinking about it, and doing something about it.

The Kuleshov paper we are working on in the film group is still under construction. They eye-tracking data are… well… didn’t turn out as simple as we had predicted, and a bit more will be explored. But, there were some effects. Alas, our application did not get funded this time either. We will try one more time.

The Film Connection also resulted in me being on a “betygskommitte” (grading committee) for a PhD for the first time. A really wonderful dissertation by Roger Johansson, who has looked at minds eye and eye-movements. It took place as part of the 25 year celebration for the department of Cognitive Science. This department, at Lund, is not connected to psychology, for some mysterious reason having to do with a past I’m only vaguely aware of. Great speakers, both from inside and outside – like Frans de Waal, and Nicholas Humphreys (though he is not entirely up on the wobbliness of social priming, I think).

I got a paper published, with my former PhD student and the other advisor. From his dissertation work. This is a paper that have languished at various journals for up to a year, so we are pleased that it is coming out. One of the last things I did before going on break is proof-reading the manuscript. (All the eta squared had been printed as h squared.)

We submitted another, from that same dissertation. We will see where that goes.

Slowly, my publications are increasing.

The fall nearly did me in. I taught Theory of Science for the second time in this format. I love it, but as it was just the second time, it still required quite a bit of prep work. I also started up our new Evolutionary Psychology theme course. I didn’t want to delay it (it is given once a year only), but this meant I had a completely new course on top of my just once before course. Then I agreed to be a co-teacher for a doctoral course on computational methods. I had to agree, first, because I instigated it (although did not do the planning), and second, I was the only female. Cannot back out of that.

All I did was prepare courses, until I started having stress symptoms with tingling tongue and rapid heart beats, and towards the end feeling a failure of emotion regulation imminent. My clinical, and my stress researching friends told me to take it easy, and I believe them.

This Christmas I have not looked at my work e-mail. I have read research, but the stuff I want to read, in the pace I want to do it, which is nice. The students are wonderful, but sometimes the demands are just too much.

I did get some of the students to do a preliminary snake study, but I don’t quite know where that will go.

Another group did an eye color study that I was interested in. Results were – weird – but interesting. I still have to consider where to go with this (we are collaborating with the original authors here).

The wobbly data is still wobbly. I presented it in Stockholm, but I think I will need to embed it into a more meta-analytic study about this, because it is wobbly. Also, not written up yet. Also, still on the figuring out meta-analysis stuff thing (which I have been reading about this vacation).

Still want to do Bayesian stats, but have not found the time.

Emotion group – working on, but this fall was a bust. It is still going, but overlaps quite a bit with the film group.

Well, I still feel OK. Got something accomplished. Most fun are the people I talk to on twitter, which makes me feel connected and interested, and motivated.

And, 2014? No idea. I hope to be able to manage the spring so I can do research. More blogging. More fixing science.

First up, though, almost, I get to meet another one of my new twitter buds, JP deRuiter. I invited him to give a talk, and he is coming in January. Turns out he is also friends with someone in the Humanities Lab. This will be fun. (Now, who to invite next….)


Posted in Uncategorized | Leave a comment

Marketing Psychology?

In 2007 I started my Marketing Psychology course. I created it – the Bologna initiative suggested that we should have courses that were Useful. Applicable to the working world. And, I knew this one would be. You see, I’m basing it on Robert Cialdini’s influence, which is a wonderful book, and you should get it and read it. The beauty of his work is that he took what good sales-persons and marketers know already, and made it explicit in social psychological terms.

It is a classic, both in academia, and in business.

I come from marketing. In my Past Life (where I was not Cleopatra, unlike so many others), I ended up working in an In House agency, for the company that (in part) ran United Airlines frequent flyer program. I was in Production – that is, the buyer of materials and supervisor that everything gets mailed correctly and on time.

The people working there were, well, superstitious (believed in subliminal advertising for example, being worried about apparent x’s on pepsi bottles, implying sex, and just being non-concerned with more supraliminal advertising like that bikiniclad woman splayed on a billboard next to the freeway hawking piss-beer). Also, not necessarily that great at the simplest pieces of statistics. Nice people otherwise.

But, I knew, from the inside, that marketers weren’t attempting to manipulate, they were attempting to pander, because they know, like anybody working in that business, that most things never ever pan out. Nothing works. Most products disappear. It is for naught. Like the lads and ladies of the night, we put on the colors, and display our wares, hoping for takers.

No wonder they are superstitious.

In my attitudes course, in grad school, we were invaded by the business students who wanted to know what us psychologists knew about influence and persuasion and attitude change.

At some other point, my marketer husband and me both realized that we used Cialdini. For him it was for the direct marketing community. For me, it was the social psychology.

If anything would be truly applied it would be this.

I didn’t need to market the course. We announced it, and they came. In droves. From all over the university. Half of them from Economics. The next largest segment from psychology, but then also from media, rhetoric, political science, philosophy, etc.

It is a great course. The students are wonderful and dedicated and inventive. It is part theoretical, and part applied, and from the beginning, they have been out finding clients for whom they can create marketing campaigns, based on Cialdini’s principles.

And the principles work! I had this one guy who came back to me all excited after a tour with his band. Instead of being stingy with the swag, they started giving out free buttons and things, and then outsold all the other bands in Swag terms (principle of reciprocity). Everybody can tell the tale of where social proof worked – like everybody lining in the most populous line, believing the empty line is empty for a reason other than, well, nobody else has gone there yet. And, scarcity, I think we can all relate to wanting that thing we can’t have more than everything else.

But, like I said, I have done nothing, zip, zilch, nada to market it. Other than whatever is the standard announcement of courses. It has great word of mouth.

I’m astounded. And gratified. It seems like I found a need, and filled it, and selected some great literature, and were given a nice team of co-teachers (both grad students at the time).

In our department we have both theme courses (which my marketing course is), and basic courses. The theme courses are more applied. There were some concerns when we created these that we were more pandering to some business need than adhering to the need of our subject, and that we actually needed to focus on the core of our business, which is basic research. And, I agree. I think the applied stuff is a great selling point (lure them whith the use, retain them with the fascination), and my course has actually induced some students to minor in psychology.

But, I keep thinking that psychology has an image problem, especially the kind of psychology that I am involved in (basic research, not the therapy kind), and have wondered how to market us better. I’ve been thinking of using the principles for my course, but have realized that, even if it is about marketing, it marketed itself. What does it say? Why do people continuously conflate psychology with the kind where you lay on a couch and talk about your mother? Why don’t people know about my kind of psychology, and could I use the principles from Cialdini to change things? So that courses in the core topics don’t get cut.

I’ve mentioned that I don’t really get to call myself a psychologist in Sweden, although my PhD is in Cog Sci and Social Psychology, and I teach at the institute for Psychology, on the non-clinical side.

I’ve written about the Thomas Quick affair, which feels like an open wound, and something that psychology needs to deal with (and perhaps defend against). But, the other night I also watched this half-assed Swedish Murder Mystery, mainly because the setting was in Ludvika and Grängesberg (I grew up in Ludvika, so I got to see the Train station, and some of the mining areas around). The main character is a forensic psychologist! Who is a profiler, of course. And, states that the serial killer (who, btw, was found killed) murdered young boys, because his father had made him feel useless and unloved at exactly that time, because, well, we all know that people behave badly because their parents did shit to them at some time (or do we? I’m not sure at all, but the tv programs say that). Otherwise, he doesn’t seem terribly psychological in his bad decisions and bad behavior and penchant for fucking around and behaving badly, but I’m not a clinical psychologist, so what the fuck do I know.
I got tired of it. I mean, Cracker was fun ‘n all in his pre-Hagrid days, but even then he didn’t seem to really have anything to do with real psychology as opposed to psychology in the minds of TV producers. I had hoped that a made-for TV murder mystery in Sweden a couple of decades later would have moved beyond Cracker, but no.

Not even our union knows what someone like me does. It is all clinical and counceling.

The not-that-kind of psychology, the basic, the scientific and very far away from Freud Psychology needs an image do-over and a marketing campaign. And I have no idea how to do it. Some teacher in marketing psychology I am.

I don’t know how to do it. People are fascinated by psychology, but don’t seem to want to deal with the real stuff (like the report that said the students didn’t like the quantitative stuff – and which I can’t find right now. Well, Fairytales and folk psychology it is then).

I want to wrest the topic from crappy TV shows suggesting we can profile people, and belief that psychology is about reading thoughts and minds. But how?

Perhaps I will put my marketing students on it.

Posted in Uncategorized | Leave a comment

The serial killer that wasn’t

In mid to late 90’s I read about the case of the Swedish serial killer Thomas Quick. The source memory for it is forever dissolved. Most likely I had surfed in on a Swedish newspaper site in some fit of either nostalgia or procrastination. There I read how he had been brought to some place of a murder that he had confessed to during therapy in order to maybe locate remains. What I do recall is that I immediately thought that he had murdered exactly nobody, and that this was a case of therapist-induced false memories. I wondered, how could Sweden be so behind the curve? This was late ‘90’s, and the recovered memory craze had swept over the US during the 80’s and then deflated under the research of Elizabeth Loftus.

Thomas Quick was eventually convicted of 8 murders – real, disappeared people. He was already in a mental hospital when he started to recall all these, and he kept remaining there, as far as I know. (I think this site can give you the gist of all this).

I then move back to Sweden, and some years back, the case started to unravel. Lawyers for Thomas Quick – who has now changed his name to Sture Bergwall – asked to have the cases retried to exonerate him. As of this year, he has been exonerated of all eight murders. I was right.

It is considered one of the largest legal travesties in Sweden. The prosecutor has maintained that there were good evidence to prosecute him throughout the trials that eventually exonerated Bergwall/Quick. Today, in the newspaper, two days after the program I’ll summarize was sent, the Judge for the original trials still maintain that the evidence was good, and he did nothing wrong. He claims he “corpse-dog” marked the spot – in rough translation. (It is a dog trained to find corpses). No body was found there, though.

Yesterday, that is November 27 2013, Swedish Television sent a program examining the psychological travesty. I watched it today, the 28th, and it makes me both depressed and shocked and sometimes angry. But, it is hard to be blindingly angry. People are so fallible.

It is mostly in Swedish (apart from the interviews with Elizabeth Loftus), I link it for completeness, but don’t bother clicking if you are not Swedish in Sweden. I think they also limit the zones in which you can watch it – which doesn’t make sense considering that the language itself limits who can watch it.

It is called “The woman behind Thomas Quick” by Swedish investigative journalist Dan Josefsson. (And, I am summarizing here what I learned from this program, along with my own observations).

Quick/Bergwall grew up in a small place in Dalarna (in fact, as I found out, the same small place my grandparents lived in until I was nine). At some points he got into drugs, and was misbehaving in the way that addicts are. There were probably other problems (looking at Wikipedia). He had spent time in a mental hospital in the 70’s. Then he got clean. Life seemed fine. He was involved in a small business with his siblings. He got into drugs again, and robbed a bank. His siblings turned their back on him. He was convicted and sentenced to a mental institution – Säter. So, they must have deemed that he needed treatment rather than imprisonment.

Säter has had a mental hospital for a long time. It is a small town in the same province where I grew up – smaller than my own hometown Ludvika, but it was well known for its mental hospital. In fact, you would say someone was “på säter” to mean that they are kind of crazy. Way for a city to get known. I actually went on a school visit there, my next-to-last year of high-school. Went through, looking at the patients. (From my perspective now, it seems absurd that a school would arrange an educational visit to a Mental Hospital, and to actually have them move through some of the patient areas).

For a long time they had a special wing for the “criminally insane”. Or, as my husband suggested (when I was at a loss for words) “psycho-killers”. (Ques-que-c’est). It was closed down, and a new facility was opened – one where they would not rely on restraints and medication and keeping their wards away from the rest of the population, but to cure them with talk therapy!

Head of this effort was a woman, Margit Norell, a trained psychoanalyst with very clear theories, it appears (she died 2005, in her 90’s). The other therapists and doctors were also supervised by her in their therapy work, and were also her clients! The few people who actually are willing to talk on this program talks of her as charismatic, as having the answers, as being the best advisor and therapist that there is. They felt especially selected. One of the men seems on the verge of tears telling this in a way suggesting that he realizes how problematic this is now. Another seems more incredulous, although honest about what it was like.

The theories Margit Norell were advancing rested on the notion that the problems that people had were rooted in trauma during childhood, especially sexual trauma. These had then been repressed. The cure would be to recall these repressed memories.

I can almost yawn at these ideas. I heard and read of them in the 80’s – the repressed memory stories. How recalling them would heal you. It was LA, of course, but I think there was a zeit-geist. The documentary talks about this as a cult around her, but, of course, she was not alone in believeing this. I have met too many other people – therapist, therapees, articles in LA times. Elizabeth Loftus got her ideas from somewhere.

Sture Bergwall is doing therapy with this group. In 1992, the hospital figures he is ready to get out into the real world again, getting into his own apartment and reintegrate in society. Bank-robbery and drug addiction only get you so long in the hospital. But, it seems in the interview with him that this is scaring him. He is lonely, he is afraid of relapse, he wants to stay. And, possibly from his addict days, he is used to lying to get what he needs. So he does. He starts hinting that perhaps he has some memories of having killed someone emerging.

As one of the interviewed therapists said, they were starved for facts. They had this beautiful theory of repressed memory – here they seemed to have some evidence that they were right! Finally. A theory starved for facts. That was such a beautiful, and frightening notion I wrote it down.

And, some of the writings to Quick/Bergwall, shown in the program, from the doctors and/or therapists suggested that they are so appreciative of this. It is shocking to see. In rough translation (from memory) one of the doctors wrote that, although he thought that the acts of Bergwall were grotesque he was deeply admiring him for bringing them up and being willing to face them. (Just a doctor writing that to his patient seems so boundary crossing to me it is shocking. )

No wonder he was willing to recall more and more murders. I think he confessed to 20 or 30. He got so much attention. Was so much of this group.

The journalist pulls a stunt that has psychologist me cringing (because IRB) – he arranges with people to interview them under the guise that he is writing a book about Margit Norell, and then secretly filming them. He first interviews Birgitta Ståhle that was the main therapist of Bergwall, but who also have been trained by and in therapy with Norell. She is rather gushing about her. Later, he inteviews another woman, who was also a therapist of Bergwall. She was, in a sense Norell’s granddaughter, as she has been in therapy under Ståhle. I think. (God, I don’t want to go back and view it again. Maybe I should just call it woman x woman y). This second woman is disguised to protect others who are named. In the program she has been in therapy for 30 years, as well as being a therapist. She sounds convinced that she is recovering memories of molestation from when she is one. One of them – I don’t recall which – claims that they are helping people recall things, and says “if it feels true, it probably is true”. I had to note that one down too. There’s too much actual research out there suggesting that this is a really crappy heuristic. But, I remember this one too from the 80’s. An uncritical thinking that if it feels… something it must be.

Göran Fransson is one of the people willing to be interviewed in this. In deep hindsight, of course, and he is aware of this. At the start, when they are going to cure the patients with talk therapy, the feeling was heady. A new start. They were going to fix this! (Oh, how this reminds me about that paradigm shift evidently suggested in the British system). But, he also is incredulous. Like he knows that once he believed all this, but he no longer does, and how do you get into that frame? In hind sight?

At one point, the journalist is asking him about what happened after a fall-out where he left the hospital. He had now read the other research and realized that the theories of Margit were wrong. The evidence pointed in the complete opposite direction. Specifically he asks, why did Fransson not do anything? Why didn’t he step forward. (By now the police were involved, taking Bergwall to places where the alleged murders took place, looking for remains that never turned up). And Fransson replies, yes, sure, of course I in some way had a responsibility to do that, but I didn’t do it and why should I? There was someone else now involved, someone who really should know the research.

This someone else is Sven Åke Christianson. I came across him in the early 90’s. I was interested in emotion, which made me weird at that time, but I had a partner in crime, Charles, and there was other people at UCLA who also were interested in emotions, and we met every once in a while at the Neuro Psychiatric Institute. I was by far the junior person being an undergraduate. The other people were therapists and psychiatrists. At one point, Daniel, the psychiatrist that started the group, kind of casually mentioned that there really was no independent corrobating evidence that people could repress and later recall memories of early sexual abuse. Specifically, he said that one potentially could look at hospital admissions for molestation, and then check if that fit with the recalled memory. None of that was there. It was all done in the therapists room.

This was rather stunning to me, because I had had the impression (from numerous articles in the LA times, and other sources) that this was well supported with research. I’m still stunned!

There wasn’t much literature on emotion then, so I came across Loftus and Christianson’s work where they showed slides of events to participants, where a critical slide was either awful (bike accident), neutral (nothing happens) or weird (protagonist carrying the bike), and showing increased recall for the emotionally upsetting picture. There were a few other studies they did in the same vein – one with erotic contents. All showing similar effects. I have taught that paper in my emotion class. We also reference that work in the paper that just now went in press – work on recall in forensic situations.

Christianson got involved with the case, but instead of pointing out what the research said about how easy it is to create false memories of even really horrid events, he becomes expert witness for the prosecution. He says that, yes, people can repress memories of horrid crimes they have committed. He even goes with the team when Bergwall is brought to crime scenes in order to dredge up memories – there are filmclips of this. In one Christianson is suggesting that Bergwall do something with his posture, because body movement can help with the recall! (Oh, embodied cognition – but I knew that the therapy peeps had thought of the body containing memories from waaaaay before, like the 50’s. I think that guy who started a school that eventuall became just nakedness and sex. I have completely forgotten the name, because, well, prudish me, shudder).

When Loftus is interviewed about this she says I have no idea what happened. The journalist, on a hunch, e-mails Christianson with the same mild lure – that he is interested in writing a book about Margit Norell. And, he gets a response and invitation. He brings his secret filming device. On this hidden camera, we see Christianson stating that this woman was, aside from his children, the most important person in his life. She became his therapist (up until her death). He feels like she was like the mother he didn’t have, and how wonderfully warm and affirming she was.

Note that he has the title Professor these days, in psychology, which is not something you automatically get as soon as you start your tenure track in Sweden (I’m a lowly lecturer, even though my employment is secure. I may never make professor, the way the structure is set up, no matter how much I end up publishing eventually). He has also been expert witness for the prosecution in most of the cases where Bergwall was convicted. Saying that you can repress memories. Despite his own research. In fact, in other instances of murder, newspapers have called on him as an expert about the motivation behind murders – and he has written a book about what supposedly is in the mind of a serial killer.

Time and again, the notion of “sect” is brought up. That it was a sect. This warm, strong woman who had the answers, who was the Best Therapist, and if you worked with her you were Chosen, and why would you then question it? Göran Fransson brings it up. That it all seems so weird and unbelieveable from this side of history. And, possibly, that anyone could fall into this.

In the story, they also talk about this as isolated. A sect with a sect leader. But, the ideas of repressed memories was not something that this woman was alone in believing. Like I said, it was ubiquitous in the 80′s, at least in the mildly depressed/distressed circles I was running in in LA.

I kept seeing the allure of the all knowing, warm, accepting and acknowledging person that seems to have swayed Christianson. I mildly thank my paranoid, mistrusting personality, because, of course, I can so see how wonderful this would be, but I always expect the other shoe to drop. It is a crappy shelter against it, because I rarely feel safe and warm-fuzzy.

I also think of prospective and retrospective thinking. All of this is in retrospect. But, it is different when you don’t have the outcome in hand. When you don’t know. Should you really have known better? Really, honestly, up until that meeting at the Neuro Psychiatric Institute, I thought that repressed and then recalled memories were possible, because so much of what I read, in books on psychology, and in newspapers, stated that this was uncontroversial. (I always stopped, though, when they brought up the satanic sacrifices of babies, because I thought surely that should have some independent evidence suggesting it had happened, and there were none).

As the journalist points out, so far what the score is from this is that 8 murderers are still out there, and Sture Bergwall was kept behind bars for more than 20 years longer than originally planned. Did any of the doctors and therapists pay a price? Yet? How many other clients are also victims of these warm convictions?

Just yesterday and today I have seen tweeted an account of a couple who are finally freed from prison, 20 years later, because the accusation of molestation was not believeable. This in Arizona (if I remember right).
Who else?

When you are wrong, but dedicatedly so, in therapy, there may be a lot of suffering.

And how? How? Mere exposure was one I brought up, and which my colleague Michael concurs with (he has done research on it). The more you see something stated, the more true it seems. I know that was the case with Bargh’s elderly research. I have written about my attempt at priming earlier. But, I know that once I saw it cited by two people who had been skeptical, but who I trusted (my advisor, Danny Kahneman), I figured it meant it really had been well established enough that it could be cited. This is more an embarrassment for the field. Nobody went to jail and no murderers went free.

But, eternal vigilance is necessary. It is so easy to go completely wrong. And, I worry that psychology – especially clinical right now – is falling back into that trap again.

I don’t know what the repercussions will be from this Documentary. I suspect that the meeting where I provided background materials (which ended up being my “too many” post) was really prompted by this. How can we protect the reputation of psychology, when it is so intensely identified with the clinical aspects (even today I fielded questions about being a not that kind of psychologist) when things go this horrifically, scandalously wrong. My discussion was in another direction, of course, but still.

Posted in Uncategorized | 1 Comment

Too Many

My colleague asked me to do a summary about what the current issues are with psychology, for a national committee he is a member of.

I produced a 3 page document (in Swedish), with some of the main points. Headed with this

• Lots of published research is false.
• Incentives are not aligned with good research practices

o Publication bias
o Questionable research practices to increase chance of publication
o Price is not borne by QRP researchers/Journals

• Initiatives to handle the problem

o Several initiatives centered on replication (reproducibility project)
o Open science framework (ease of sharing materials and data)
o Pre-registration of research
o Published peer-review
o Post-publication peer-review

• Support

o Support for participating in replication/archive
o Support for participating in post-pub peer-review
o Developing better metrics to adjust problematic incentives

But, prior to that, I wrote pages and pages (and pages) to collect my thoughts. And, this is some of what I wrote.

What we have here are much too many scientists for too few slots both at universities and in journals, and for funding. And, as is the case whenever there is too many for too little, the Darwinian struggle will take place with selection for the fittest – whatever the fittest may be – not necessarily good science.

How did we get here? I’m no historian, but the system with publishing and peer-review and appointments was devised when the scientists were fewer, as were the outlets for findings. Even then, the scientific process was likely Darwinian (borrowing from Hull, science as a process), with tribal like groups pursuing one version of truth, with other groups opposing. The self-correction that science is famed for, was not in the individual scientists or even scientific groups, where one stubbornly could cling to phlogiston, or cladistics, or whatever theory that is even still going or dead, because rival groups would obligingly set out to test and rip it apart, if at all possible.

At some point, it started to grow. Historians will know better – my account is a hearsay account puzzled together from Meehl, the Economist, Older academics, and snippets I can’t for my life place, so it can certainly be critiqued. There was money to be had for research that the universities could skim off to pay for other activities (per both Meehl and Stephan, and an random commenter on Stephen Hsu’s blog). There was research to be done in the hard, and semi-hard, and medical and soft sciences, and there needed to be people to people those labs and write up accounts and educate the new students. Who to select and promote? When I read accounts of American universities from the mid last century, it seemed like they were mostly home-grown. The home-grown academic still exist in Sweden. There were places to fill, and here was a good person who had just gotten a doctorate.

But, as more and more doctors were graduated, the need for selection set in (I now am recalling Malthus). Who to retain? Who to promote? Who to grant funds?

Those who are either already doing interesting science, or promise to do interesting science.

But, this is notoriously difficult to assess. Sure, you can probably assess skill or lack thereof, but is someone following a viable path to discovery, engaging in what will turn out to be a blind path, or engaging in some kind of pseudoscience? Anybody with a bit of familiarity with the science of forecasting, or modern theory of science knows that this is just about impossible, except in hind-sight, where it seems obvious. Some of the smartest minds tried to find demarcation criteria, and instead discovered that they could not be found.

Successful science is determined historically, but prior to that it is a matter of placing bets.

An obvious way is to actually look at what a scientist is doing. Evaluate the soundness of the research, and how interesting it seems on this side of the future. This can be done, especially by close peers – this is, after all, what peer review is based on. Of course, we know that it is far from perfect. And it does not help us get around the problem with the unpredictable future.

But, hiring and promotion committees, and even some granting committees are not experts in that particular mile-deep pinhole of research, and have to find other ways of evaluating what is good, and deserving.

And, what do you do? Rely on a heuristic so common that marketers have stumbled on it unwittingly, social psychologists have thoroughly investigated its effects and side-effects, and evolutionary psychologists have dubbed it as an evolved heuristic for the use in social learning – looking at the decisions of Authorities. We may not be able to properly assess the work, but the researchers Peers can, let’s see what they say.

You use peer-review, or Peer-review by proxy – Journal articles.

This is not a bad index, at first glance. We use it all the time, because none of us have the time to dig through and learn to understand everything from scratch. If you are interested in focusing on research at your institution, and want to know who might be successful in the future, look at their past record, especially the record with their peers. The already published and cited are more likely to be published and cited in the future. Hence, publish or perish emerged.

But, perhaps, as I just lectured about in my evolutionary psychology group about this heuristic – the prestige index – it can give rise to other derivative indices, resulting in a run-away prestige selection. Very much like that one supposedly responsible for the peacocks tail (which Behavioral Ecologists looking at sexual selection will groan at for being so stale): A Fisherian run-away selection for indices of prestige.

Somewhere in here entered the ranking of journals, which also began in a history that I’m only foggily aware of. The roots of the impact factor can be found in another case of scarcity – space. Library space, in this case. Librarians needed to make reasonable bets on which journals to purchase and retain, as the output steadily increased (Arbeman’s “the half life of facts” illustrate this nicely). What to do? Well, look at which journals scientists mostly read, and mostly cite. Another bit of prestige selection, perhaps, or at least a bit of frequency selection (what do they all like? Let’s keep that).

Now we have an index. But, what do we do with the index? What humans, at least those interested in rank and prestige and competition, do: we use that to measure worth. It is thebeginning of the Fisherian prestige run-away selection. Instead of just looking at whether someone was published in a peer-reviewed journal, and how often, and how frequently cited, look at the rank of the journal. And, from the competing scientists view – if I get it into one of the journals that are more highly ranked (because more scientists read them), the more eyeballs my research gets, with potentially more citations, which raises my prestige.

Meanwhile, the universities kept churning out PhD’s, and although I surmise there were more university positions for teaching the ever growing number of undergraduates, the number of faculty slots were not growing as fast (by a long shot) as the number of researchers, all who at some point were best in class, with top marks, and willing to work out of love or obsession with their topic, or desire for – well – prestige.

How do you distinguish yourself among peers who are all just as excellent and inventive and passionate, and highly trained as you? Well, you look for competitive advantage, the economists will say. This can start as plain competing. You publish more – your CV is simply longer. Or, maybe it is shorter, but it is in those highly ranked journals. It starts becoming important to be strategic – am I in the right niche? What are they paying for this year? Where should I place my article? And, should it really, um, be just one? How about splitting it up in two? Can we collect data faster? We’ll label that data-set later. Could we add this co-variate and perhaps get the p-value into one that ups the chances of publishing? Remove a few outliers, perhaps a few more. Check on the data, do we need to collect a few more? No?

Then you have the journals. One of the limiting sources, by virtue of, you know, pages (a limit that is slowly dissolving into endless cyberspace). Like that choosy peahen, wanting to select just the really good pieces of research (still, just as difficult to assess, and now by peer-reviewers who are full on engaged in the competition to keep a job). There is also another piece to protect – journal ranking. And, let’s not get into financial issues – it leads too fast into the suspicion of the nefarious, and I simply don’t want to go there. I don’t think journals or editors or peer reviewers are nefarious. They are simply, like any good limiting resource (such as peahens) being choosy and discriminate in order to maintain their level of prestige, of eyeballs, of shelf-life, of down-loads, and of citations, so that the scientists will keep vying for their interest.

University positions are similar (oh, the sex metaphor breaks down, unless one think, like Kurt Vonnegut in Slaughterhouse 5, if I remember right, that there is a need for multiple sexes in order to reproduce). Who should they hire? To keep prestige. To keep the research going. To keep the funding coming. Funding agencies alike are looking at how to select who to grant. Good researchers are now a dime a dozen. More. They are endless. They are skilled. They are hard-working. They publish. They publish more and more and more. (As someone mentioned, the body of work Kahneman won the nobel prize for would not get you tenure today).

Being a researcher is slowly becoming similar to being a musician, or an actor or poet, or any of those positions where very few are chosen from myriads of qualified. You do not play the game, and play it well, you are gone, and even if you do, you may still be gone. Most are gone.

The markers, who at one point were reasonable, are also changing to become not so reasonable, because they are not necessarily signs of skill. Evolution sees the emergence of gamers, of mimics and cheats, and sees no way of eliminating them. It is a way of getting competitive advantage, or defending against predation. Mimic the pattern of that bitter butterfly and escape being eaten. Grow the bands of the poisonous snake, and enjoy the protection of the danger signal, despite being harmless. Lay your eggs in the nests of other species and avoid the cost of raising off-spring.

The competition, with the overproduction of scientists, is encouraging the mimics. The outright fraud is the extreme – nothing is likely to protect from that – but there are still the questionable practices, and the underpowered experiments, and the non-reproducible results in fields from cancer research to social cognition.

What does this have to do with Sweden, where there is no overproduction ofPhD’s? The market is global. Research is international. We cannot escape.

The problems with how science is conducted and reported and evaluated and rewarded have been raised for decades. You have the Tukey’s and the Meehls, and the Cohens and the Gigerenzers publishing and critiquing and requesting that we pay attention to the problems of the research and nothing really changes. Yes, we have added a requirement for effect sizes.

But, calls for going against the system in order to do something properly will not be heeded when they go against the practices for even staying in the system. Boom, you are selected out.

We can train the new scientists, but what will they do? They will use those evolved heuristics for social learning that we are endowed with, and look at their successful elders, and try to emulate them. The systems, as they are, are not good at distinguishing whether the signal comes from solid research or problematic research (it was never easy). It also may reward traits that are not conducive to good research, but are conducive to good signaling.

I think a lot of researchers experienced a disconnect between how statistics and methods were taught in the class room, and how it was actually practiced in the lab, and the border between good pragmatic practices, and questionable ones is not a clear one.

I think researchers have been unhappy with the state of affairs for a long time, especially those who have not done well by the system of course, but I think also some of those that have. Because science is based on trust, and if you cannot trust the results, everything breaks down.

Otherwise, there would have been no Tukey’s and Meehls and Cohens and Gigerenzers. Science is more interesting when it is true. And, to paraphrase Uri Simonsohn’s sentiment, one can’t bear being a part of a field of research where there is more flash than substance, because we got into this not for the fame or fortune, but because we wanted to figure out how the world works.

I don’t know if it is different this time.

Posted in Philosophy, Research Practice, Science | 1 Comment