Science is a craft. What is painstakingly, lovingly and carefully crafted are studies and experiments designed to test or capture something interesting. As any craft, it needs practice and honing, and the learning of a wealth of tricks and pragmatics that no text can ever cover. You need to do.
In all the courses I have designed (and they have allowed me to design a lot of courses) I’ve included practical elements. It could be the simple canned 2-hour experiment, which is not the most exciting thing in the world, but gets the students out of the books, and gives them a chance of experiencing how messy results in psychology really are. I often include a project section where the students, usually in small groups, pursue something practical.
Some years back, I was asked to co-design the introductory course for our new International Master’s Program. We already knew that half the new course would be on Theory/Philosophy of science, but we were unsure about what to do with the second part. During a discussion, one of the older lecturers mentioned a time where they had run day-long labs – 8 hour sessions where the students could become immersed in a laboratory project. This was the start of the ideas for the practice section of the introductory course.
I’m now running this course for the third time. So far, the students have enjoyed it (I haven’t gotten evaluations yet, but they were high even the first time)and there has been some evolution in how I handle the course.
It is divided into two parts. In the first part, we have 8 day-labs, and in the second they do a short project that ends in a poster presentation.
For the day-labs I recruited lecturers that are representative of the major research directions at our department. I want the students to be exposed to a broad variety of methods even if they have their heart set on a particular area and a particular methodology. As researchers we need to have some idea about the variety of ways to ask and answer questions. Since the implementation we have a Creativity lab, a personality lab which is focusing a great deal on being a data-detective, a Bayesian statistics lab, an ERP lab, and my two social psychology labs which I will go into in more detail. Our university has a good Industrial/Organizational department, so we have always had at least one day-lab focusing on these types of issues, but here the teachers have shifted. I also hope we will be able to once again have a developmental lab, and possibly to retain our Clinical lab, even if it is not possible to move to clinical psychology from non-clinical directly, the way the educations are set up here.
I, perhaps foolishly, decided to take on two day-labs. In one, participants get to program their own IAT. In the other – in the field – they get to go out and either observe or interfere with the public. Social cognition and the old, venerable tradition of field work.
Just about all the research I have ever done has been done on a computer, which, of course, means that I have programmed up a lot of experiments in different types of presentation software. In fact, the first software used was custom written in Pascal. Since then I have used DMDX (for its ability to record precise reaction times). I wouldn’t want to teach DMDX to students who do not have at least a little bit of computer programming back-ground. That turns out to be the vast and utter majority of my international masters, to my surprise. (I got my Cognitive Psychology degree during the hey-day of the computer metaphor of mind paradigm, so I have programmed in both Lisp and C++)
Our department has a license for e-prime, which, compared to DMDX is a docile and kind software, with both drag-and-drop and self-closing parentheses.
I spent a handful of days teaching myself how to use it prior to my first day-lab.
The first time you do anything is always a pilot and a learning experience for everybody. Here are a few of the take-homes.
- There won’t be enough students with programming experience to take the lead, but this is not important.
- Program in pairs, not in groups, because it is too inviting for social loafing, even in motivated students (the students themselves suggested this afterwards).
- Don’t use pictures the first time you create an IAT. Too much time is spent on trying to figure out how to insert the pictures, and too little on figuring out how to create a functioning IAT. (This wouldn’t apply if the teacher is a very experienced e-prime programmer.).
- You can learn to program your very own IAT from scratch in one day.
The past two times I have run this (and I run it in half class, so four repeats), the day has stabilized as follows:
I book our computer room, where there are enough computers for the students to work in pairs.
The first half hour or so, I lecture on background. First, the rationale for the lab:
- A lot of data is collected via computers. It is good to know how to create such programs so you can tailor them to your needs (and don’t need to engage others).
- A lot of times we read about some neat paradigm that we want to use. If it is described well enough, one can use the methods to re-create it.
I give them the Greenwald Banaji & Nosek 2003 paper where they compare different computations for the outcome, as that was the paper I used when I first programmed an IAT. Of course, I did that all by my lonesome, up hill in the snow both ways in DMDX (cue the violins).
I then tell them about the IAT: what is the rationale behind it, how is it structured and why (learning, fatigue, handedness, etc.). I got tired of using ethnic in-group and out-groups for my examples this time, so instead I used a comparison between chocolate lovers and frugivores. They get to do whatever comparison they want in their own program.
Next, we all go through the very nice tutorial that comes with e-prime. I sit by the teacher’s computer, projecting the programming onto the white-screen. I get to tell them about the importance of case, types of brackets, and frequent saving. The tutorial teaches them to do a one-block priming study. Simple, but still somewhat different from a single block of the IAT.
Once that is done, I let them loose. By now it is usually around 10 am, so they have the rest of the day.
So far, just about everybody has a functioning 7-block IAT by the end of the day. Usually several of the groups also get to test each other’s program. This time there was also a couple of groups that were done with functioning programs quite early – 2 pm, which means that there are 3 hours to go. I really do not want people to up and leave in the middle of the session. Fortunately, there is so much more one can learn once one has a functioning program (and I need to have that more clearly up my sleeve for the next time). Collecting the data is only half of the work. After that, one has to look at the output, extract the important data-points, and create the IAT score! Are you done with that? Work on either figuring out a proper counter-balancing, or how to add pictures to your IAT.
In the field.
When I first decided to do both Social Psychology day-labs, I started becoming a bit panicked about what to do for the second, and how much prep I might have to do. I’ve advised a lot of introductory students on their final short project, and a favorite of theirs is to do various field studies. That could be dropping things in front of single people or crowds to test dilution of responsibility; “stealing” ones bike when either well-dressed or poorly dressed (nobody interferes regardless), or one of my favorites where they left a bag in a public space and someone else later came to pick it up to test if anybody would interfere. (It matters if you are a man or a woman, if I recall right).
Field studies is a staple of Social psychology, and it has turned out this is a fun day. We structure it like this:
I start with a brief introduction where I summarize some of the seed-studies I give them to read beforehand (for inspiration), and summarize some of what earlier students have done. Then they are divided into 3 groups (5-6 people in each) and given until 10 to come up with a study, measures and all, that they can perform that day. I float around and help them shape their ideas, making sure they can do them that day, that they are not planning on doing something that can land them in trouble (a group had the idea of having a woman and a man pushing each other in order to check who intervened – I don’t think I would want to be responsible for that), and make sure they understand they can’t just wing it when it comes to the measures, but to have a very clear idea about what they are going to do. I give them very free hands otherwise.
By 10, we gather together and each group tells the others what they are planning to do. This allows for input from the other students that can help refine the work. Then, off they go to gather data.
I had them come back at 15 this time, but next time, I’ll have them return by 14. It turns out you can collect a lot of data in 4 hours’ time, and still have a nice, leisurely lunch.
When they get back, they summarize their data. I strictly forbid them to do any inferential statistics. The first time I ran this lab, students threw their data into a regression that they didn’t quite remember how to do, and I now have a very firm mindset that you do not get to do inferential stats that you don’t understand, especially not before you have looked at your descriptives. Besides, the exercise is not so much about testing hypothesis as it is about learning and experiencing what it means to do a field study.
This time I allotted 1 hour to do so, but I really think it is necessary to have more time than that.
Finally, each group presents their research and their data. I also ask this crucial question: Now that you have done it, what would you have done differently?
Something that frequently comes up is that it is difficult to keep track of a lot of data in the field. Also, that it is not always easy to pick the right spot or right way to do things the first time over – piloting is needed.
I wanted to share some of the favorite projects that they have done so far. (They are all fun, but some stick out).
The first time, we ended up talking about violating social norms, and how Swedes have this gigantic privacy bubbles. When you get on a bus, you always sit at free seats, as far as you possibly can. You only sit next to a stranger if there are no other options (but you are very polite about it). They decided to see what happens when you violate that norm – not on buses but in other public areas where there are a lot of options for sitting down, but you choose to sit down next to the sole person there.
There really was not much interaction at all. Quiet toleration, but evidently body language that indicated that something was amiss. The only person that talked to the experimenter was a homeless guy, who offered my student some Jägermeister.
Another year, one of the students came in dressed in a dinosaur suit. Their experiment was to approach people on the street and offer free hugs. They recorded approximate age, gender, and whether or not people accepted the hug . (Student was male, so it was dino-boy). In fact, more guys accepted hugs than the girls, although without good controls you don’t know what that means.
This year, a group was interesting in observing giving behavior, and observed some of the beggars that have entered Sweden most likely from Romania. They were particularly interested in when people are giving – on their way in or on their way out of the store, and what proportion of passers-by (categorized by approximate age and gender) were givers. In their “what would we have done different” discussion they brought up the issue with double-counting people going in or out of the store, and how to deal with that problem. It was clear from their observation that there are a reasonable percentage of people that give something – between 2 and 5%. Not always money, but there is giving. They, along with other groups, were also discussing how you make yourself inconspicuous, and whether people noticed that they were out recording data. Clearly there were those that were aware, and that actually asked the students if they were doing some kind of experiment – but this is Lund, and having students out and about doing funny things in public is not out of the ordinary.
But, I also mentioned that often people may not be aware at all. You are focused on your tasks, and the data-collector is as invisible as that proverbial gorilla. Which prompted the students to tell us that at one point they had been so focused on counting passers-by that they didn’t notice that the beggar had left.
The last two weeks of the course the students do a short project. I decided early on that it should be examined with a poster-session, as I didn’t want them to spend a lot of time on writing a paper (neither did I want to spend a lot of time grading papers, nor did I want the advisers spending time on giving feedback on papers either).
Turned out that it was very hard to stop them from doing papers. (They had to do a handout, and tried to turn that into a paper). To stop them from doing this, and to have them focus on the research process instead, I decided to use a portfolio method for grading. And, in true open science fashion, I have turned it into an exercise in what I call “Research Hygiene”. The focus is on keeping good track of the research process so that in the future when Jelte Wicherts write them about something, they can cheerfully share a properly labeled data-set and stimuli without any excuses.
I ask them to use the Open Science Framework to keep track of their project, and to share it with me. That is where their portfolio should reside.
I ask them to provide me with the following:
- A pre-registration (to prevent HARKING). I tell them it is fine to say that it is exploratory, if it is.
- Copies of their materials.
- A Lab book/diary/protocol
- A labeled data-set with code-book
- Graphs and figures of their results.
- A reflection – similar to what they do in “in the field”
They are still forbidden to do inferential statistics. (I may relent for correlations). In this exercise I don’t care if they get enough participants for proper power. The time is too short. It is nice if they show they are aware of it, though.
During the poster-session, I walk around and talk to all the presenters. They also get to peer-review a couple of the other participants posters. (I still have to update that peer-review sheet).
I use a handbook for poster presentations that were done internally by a former Admin. I also link them to Dr. Zen’s poster blog for inspiration and ideas.
We are in the middle of this session right now. It is the third time I’m running it, and only the second time with the Portfolio. Last time I had no idea how that would work out, but it turned out it worked just fine. There was one group that tried to write a paper (and submitted it in portfolio form, which meant that I had to open an awful lot of documents). This time, when I know it works, I think I feel more comfortable grading across the entire range. I expect that the grades will be high, as this really is a measure of conscientiousness.