Can an AI tell how trustworthy long dead people are from their portraits?

I was just set to teach a segment on AI and facial expressions of emotion when twitter started circulating a just published Nature paper, where the authors trained an AI on detecting trustworthiness DISPLAYS, set it loose on the faces of curated sets of historical paintings, and then claimed that this was a way to possibly detect historical shifts in trustworthiness, at least tentatively so. (Safra et al., 2020)

I’ve been thinking of it for a bit, but wanted to take a stab at my thoughts, with a lot of places that is up for critique.

A short summary.

The face-perception literature have identified face-displays that people tend to judge as either trustworthy or not trustworthy. It is important to think of these as displays and stereotypes, rather than indicators of trustworthiness. (Think the gangster-film trope of Baby-face so-and-so who invariably is the nastiest of the lot). Remember, this research is done on contemporary people, many (but not all) from western countries.

The researchers have trained an algorithm on these stereotypical displays, and then let it classify a set of digitized curated historical (and western) paintings. The aim was to see if one could see a shift in the use of the trustworthy (often smiling) display across time. The prototypes are shown in the paper. For anybody that have been mildly involved in face-research, these are face-gen faces*.

They see a shift towards more of the “Trustworthy display” as time approaches modern times. In their discussion, they take it as evidence that trustworthiness has increased across time. As many have pointed out – this is going well beyond the data.


I do faces – kinda. I’m interested in emotions and emotional expressions – and god knows that is an area that is getting complicated, because of course it is.

I’ve always been puzzled by rating faces on trustworthiness. (See my blog post on the “brown eyed people are more trustworthy I participated in a rating exercise once, where the task was to rate a series of faces on trustworthiness. I gave up after a while, which probably was good for the researchers.  I had used that facial data-base before so they were familiar to me. But, also because it felt entirely arbitrary. How the hell can I rate a random person’s trustworthiness? Trustworthiness – at a first glance – is a property of a person.

Thinking a bit more deeply about it – trustworthiness is a property of relationships. As an extreme – think a spy. Trustworthy in the home-country, deceptive in the foreign country. 

The importance of being trustworthy – at least towards your in-group – is socialized into people. Deception is punished, trustworthiness rewarded. We seem to spend a lot of time on teaching why it is important to not lie, to be honest, and not break your promises. (Think the boy who cried Wolf).

Trustworthiness may also be something we afford our (sometimes) extended in-group. You trust a friend, because you believe they value your friendship and thus wouldn’t betray you. You may trust people in your home-town because being untrustworthy would result in a bad reputation, which has a lot of negative consequences. You may trust a distant relative, or the friend of a friend of a friend.

Of course, it is interesting to understand what kind of stereotypes people have about all sorts of things, including faces and face display, especially since people tend to use these to make decisions. I’m reminded of that early Todorov paper where people had to judge who they thought won a particular state-level (US) election based just on the faces of the competing politicians. Taken together, the participants were right at above chance levels. A recent pre-print has connected impressions from faces on traits such as trustworthiness (among others) with the stereotypical beliefs people hold about what different types of faces expresses. We seem to rely on stereotypes to make our judgments, regardless of whether they are correct. Faces matter, even if we really cannot discern anything about the person based on it.

I also suspect stereotypes are moving targets. We spend a lot of time (righly) on the harmful stereotypes, and not that much research on why we have stereotypes at all (apart from Lee Jussim). I figure it is for the same reason we have lots of categories and prototypes – it helps us orienting ourselves in the surroundings. When the surroundings change, the stereotypes are updated.

Smiling norms and immigration

In this course, I also include a paper by Rychlowska et al., 2015 where the group looks at differences in smiling norms between contemporary countries. They test a hypothesis that there will be more smiling in a country consisting of immigrants from many different parts of the world, than in countries that have had little immigration.** Roughly the idea is that in a country with little immigration, norms of behavior, including emotions are stable and predictable. There is then little need to frequently display intentions through, for example, gestures and facial expressions – but as they point out, it does not make them meaningless, but perhaps even more meaningful for those instances when they are used.

However, in a country built on relatively recent immigration, like the English speaking colonies for example, there is much more uncertainty about other peoples intentions given the varying cultural backgrounds, so it becomes quite important to signal positive intent – perhaps trustworthiness.  The smile seems to have come overloaded with social affiliation/social negotiation meaning, being an appeasement signal or reward/affiliation signal. Hence, more smiling.  This is what they find in their paper. (How stable and reasonable this is, I haven’t checked, but it seems more plausible to me than the notion that brown-eyed people are more trustworthy).

This leads me to think that the conclusion of the paper (with all the caveats) should be the opposite. As time went on, and travel became easier and networks broader, the assumption of trustworthiness became more tenuous, so it became increasingly important to signal trustworthiness.

In both papers, to be fair, they bring up that there is separate research on the development of institutions and norms to support trust in large interacting network. The increase in smiling as individual signals of trust could have happened together with these developing institutions. I’m going through Heinrich’s “The weirdest people in the world” concurrently with this, where he lays out his educated thought about this cultural evolution and the extension of trust beyond kin-groups.


I collected some commentary in the form of blogs and twitter-threads collected via the very useful threadreaderapp.

Iris Van Rooij Blogpost on the paper

Iris van Rooij’s twitter comments

“shall I read the computational phrenology paper for your entertainment? By Sonia

Andreas Kirsch Twitter comments

Matteuz Fafinski’s twitter comments

Older medium article on “The new physiognomy” by Blaise Agüera y Arcas, Margaret Mitchell and Alexander Todorov

How Possible

I do think one can use AI as a tool, if it is done carefully, with recognition that it is just a tool – perhaps somewhat more sophisticated than the statistical algorithm’s we gleefully throw at our data with (sometimes) incomplete understanding of how it works.

We come with faces. Faces that give rise to impressions. I have, with age, acquired maybe not quite a resting bitch face, but more a resting tired face. My husband has the resting disapproving eye-brows; thick, dark and down slanting.

As copious research shows, we use other people’s faces to rapidly infer a lot of things about them – even things that we know cannot be inferred, such as trustworthiness.

Faces also come with lots of muscles that we constantly move. Chewing, speaking, expressing emotions. (I find myself moving my face a bit while writing this, and it has nothing to do with communicating to others.) The facial expressions we make can be tied to underlying intentional states. Sometimes we can’t help expressing our emotions, even if we try to suppress them. But, we also have quite a bit of control over our faces, and use them in ways similar to talking – to communicate with others. (See, for example, work by Alan Fridlund). We don’t have to feel happy in order to smile.

We have known for a long time that facial expressiveness is in part learned, and influenced by culture (just see that Rychowska article above) – and what we learn are both displays, and interpreting what we perceive. (We really have to divide these up, and I don’t think that is an easy distinction to grasp, coming from my own experience – although maybe I’m particularly dense).

Looking at how display rules have evolved over time – in a particular class in a particular culture – could be very informative, together with other historical data. I do think this was the impulse that drove the researchers, although I think they have wildly overstated what could be inferred from the changing display preference psychologically.

But there still are some issues. The training set are on modern day stereotypes of “trustworthiness” (see Iris van Rooij’s blog post). That is, they fit some average of face-type that present day living people tend to judge as “trustworthy” (without there being a correspondence to actual trustworthiness, as has also been experimentally shown). So, they are not training on displays (smiles). They are training on physiognomy of a stereotype. So the question they answer, perhaps, is – can we see the development historically of the current stereotype of trustworthiness? Which actually seems rather nonsensical to me, now that I write it out.

*Face-gen being a program where you can create artificial human faces with different expressions. So, lots of stimulus control. I have used a predecessor to this – Poser – to do some work.

**This is in historic time, the past 500 years. Humans have this propensity of traveling around the globe and settle places that seems to stretch at least 40 000 years – or more. In some sense we are mostly immigrant populations already, though some of us immigrated for forever ago).

Posted in Uncategorized | Leave a comment

On efficiency as a virtue

”In Western societies today efficiency is a virtue”

David Hull 1988, Science as a process.

This passage lodged itself in my memory when I re-read his book for the third time.  It was summer. We had just traveled to Australia (so technically winter, but in Brisbane, who would know), and I was spending jet-lag recovery on a sofa on the porch of the gigantic queenslander where we were staying.

I had never thought of it as a virtue, but just as something that just was something given (as we so often do with the norms we culturally inherit). But it isn’t. It is something we have decided to value (hey, more bang for the buck, more things done, more answers, what is not to like?) Nevertheless it is a virtue, rather than a key to discovery.

This was a few years into the reproducibility crisis, and a lot of discussion had been on the waste inherent in producing under powered studies. It was a waste of money. It was a waste of participants time. It was a waste of researcher’s time trying to build on results that might have just as well been so much noise. It could even be harmful and wasteful for the public when interventions were created built on research with questionable results. Think implicit bias training, or inefficient medications with bad side-effects, to bring up some examples.

I agree to some extent. If we are going to go through the considerable effort that goes into running studies, of course we should do what we can to make sure we can get a reasonable answer out of it, even when nature perhaps says “nope”.

But, this virtue also has down-sides, and in the recent symposium from the Center for Open Science, one of the speakers, I think it was David Peterson talked about how meta-science has in part focused on efficiency, how this is strategically good (funders would really get behind an efficiency argument), but that it also has some strong drawbacks. As he pointed out, efficient is something you can be when you have a clear goal, and a good background that can be effectivized, but this is really not always what is conducive for doing good science. The truth may be out there, but we often only have vague guesses about what it may be.

I was always very ambivalent about the efficiency argument. Efficiency is brittle. It works in a highly structured and stable environment, but slight changes and it all comes crashing down. What you want is robust, and robust has a lot of redundancies.

I’m also not sure if efficiency is conducive to do science. I’m reminded of an account I read many years back (the source is since long forgotten) about experimenting with genetic algorithms. As the algorithm was chugging away towards some solution, there were lots of dead branches, things that went nowhere, items that weren’t used. So they tried to remove these things, and run the algorithm again. And, it didn’t end up in the same place. It seems like all those non-producing things were needed in order to find a solution.

I’m also reminded about reading histories of science.  Right now I’m going through Hassok Chang’s book “inventing temperature” (thanks Berna Devezer for the tip). I haven’t come far, but he begins chronicling the work on finding some kind of fixed point that could be used for calibrating thermometers. There were lots of suggestions, where people thought temperature might be constant enough to be used. He spends a lot of time discussing work on figuring out the temperature of boiling water, with stretches into super heating, using different kinds of materials in the vessels, what is meant by boiling – when it first start to move, or when it is full on roiling away? Some of these ending up being side-tracks (how high can you heat water without boiling?). I had no idea so much went into coming up with something I have used since I was a kid, and that I confidently though were on a ratio scale, and even an absolute scale considering the Kelvin measure. But, of course it does. The same thing is the case with discoveries that we now simply teach as facts to high-school students, without going in to all that went into establishing something that could be measured with some precision.

We shouldn’t expect that working at the edge of science – as we likely hope we are doing – should be efficient and not lead into a lot of dead end rabbit holes.

I end with the continuation of the Hull passage that follows the quote above.

“One periodically hears rumbles of discontent about how much money is wasted in science through the support of unproductive scientists. If 95 percent of all citations are to works published by 5 percent of the practicing scientists, why waste so much money supporting all those third-raters? Would not science be improved if we made it more efficient? 

Perhaps it is only mindless romanticism, but the call for efficiency in the production of great works of art is likely to strike most of us as worse than wrong headed.  The same may be true for the more creative aspects of science. It may just be possible that creative processes of all sorts are inherently inefficient and that attempts to make them too efficient are likely to destroy them. Some support is lent to this hypothesis by the character of biological evolution in the creation of new species and the incredible array of adaptations exhibited by organisms. The vast majority of germ cells produce never unite to form zygotes, the vast majority of zygotes never reach sexual maturity, and so on. Of the thousands of sexual adults released from a termite hill, only one or two ever succeed in founding a new colony. The God of the Galapagos is anything but the Protestant God of waste not, want not. “

Posted in Uncategorized | Leave a comment

Passing of a past friend

Half a life-time ago, I had moved into the in-house advertising department at my company. My position was “Production assistant”. Advertising happens in teams, divided into particular tasks. There’s the account manager who talks to the client, the art director, the copy writer, traffic, that coordinates everything, and Production that buys services from outside vendors and make sure that everything goes right – in our case, the printing, stuffing and mailing of materials. I worked with a Production Supervisor who had the main responsibility.

Although I was known as competent, and able to work with somewhat difficult people, there was one supervisor that did me in. She was secretive, and obsessive. One of the few people I have not been able to work with or work with well.

We did have an absolutely fantastic manager though. He was able to match people well, you knew he had your back, and he got the best out of most of his staff.

At this time, Valerie came in as a freelance production supervisor. Our manager moved me to work with her instead, after a work review I had gotten that even he questioned (I was too timid to be upset about it).

Valerie had worked free-lance in the advertising industry in LA for quite a while – it is actually a small kind of incestuous group, where people move from agency to agency. But, she had started out as a pianist, with a BA in music from USC.

With my love of classical music, we got along very well. She was kind and personable. Nine years older than me, but approaching 30, that gap was unimportant.

At the time, there was a new tradition for co-workers to surprise decorate the cubicle or office for someone’s birthday. There was one where the office was filled with colored dots. The owner of the Dalmatians got hers decorated with black dots. A woman got hers decorated with hunks, and a man his decorated with babes – all tying in with what they so very expressly liked.

For my birthday, Valerie and friends decorated my cubicle with dinosaurs. When it was her turn, I cut out 8’th notes, and made a poster where I wrote out the melody for happy birthday in musical notation.

Somewhere, as I was shifting towards more studying, we started to go to concerts and operas together. She knew some of the musicians. She had played in trios with them, so I got to go back-stage a lot to talk to them, which was a lot of fun. Particularly, she was good friends with a tall cellist, and the conductor of the opera orchestra.

One of the bennies I got as a UCLA student was cheap tickets to the opera (She had a subscription). What was so wonderful with Valerie was that she thought of the opera as her home, so when we got there, and had our middling seats, she’d scout out open seats close to the scene, and just move us there, and in the breaks she would chat with Randy, the conductor.

But, as I got more involved with my studies, and met a new boyfriend (now husband), we met more seldom, and the friendship kind of faded. Not for any particular reason. Lives are lived, and things change, and then I moved to Indiana to pursue my doctorate.

Occasionally I would look her up on the net, but she didn’t have the same kind of presence as I did.

I did it again, the other day, and now I found her obituary. She had passed away this summer, 70 years old. She was a kind, generous person. I have a lot of great memories with her. And, even though 70 is respectable, much too young.

Posted in Uncategorized | Leave a comment

Being on the Red Team

Late spring, I participated as one of the reviewers in the Red Team challenge.* I was selected for my knowledge of research on emotional expressions, which now goes back to the time before emotion research was fashionable (as soon as it got, I kind of slid out. I don’t like crowds).

I was so excited about the possibility, that I plowed through the edited volume “ the science of facial expressions”, read the paper and began writing a review before I even knew I had been selected (actually lotted in), and been informed about the format of feedback. (it wasn’t supposed to be the standard review format. Oh well).

It’s a while since I was this excited about something in science!

I wanted to share a bit of my experience.


The challenge was formatted as finding errors, noting them down on a google docs spread sheet. These were to be judged by an independent judge (Ruben Arslan).

For some issues this works well: Computational errors. Confusing parts in text. Sections that seems to not quite represent earlier research.

But, I did find this format itself challenging, as my commentary was not always bite-sized, and not always about errors but about alternative interpretations, questions about methodological choices, references to other theories. I did the best I could with the ones I had and in the end also included the entire review that I wrote – which is not as easy to judge. (I did read the entire spread sheet of issues with Ruben’s and Nicholas’ comments).

Some of my “issues” weren’t so much about errors as it was about bringing up challenges from the field. For example the work assumed a categorical/discrete view of emotions both in posing and in rating of emotional experience. But there is an approaching century old controversy about how to conceptualize emotions – dimensionally or categorically. Another example was the opening statement that most theories of emotions incorporate facial feedback which, in the fashion of a long time essay advisor, I thought was much too broad, as I can think of lots of theoretical accounts surrounding emotions that are completely uninterested in whether there is something like facial feedback. Nothing was made of that. I just wanted something more precise. Is it an error? Not really.

I also came across this chapter in the above mentioned book: Form and Function of facial expressive origins, by Daniel H. Lee and Adam K. Anderson.  Their main interest was in looking at the face as a moving surface, and how different expressions seems to have opposite action (an expansion in fear, a contraction in disgust). On page 1983 they mention how the posing of expressions did not seem to have any effect on pupil size or behavioral changes (measured by looking at eye movements), and concluding from this that there were no evidence for no autonomic feedback. I’m not sure how to incorporate that into the paper.

In the end, I did like the paper. I thought it was a cute, minor contribution, but doesn’t resolve the facial feedback question. Of course, no paper could


Unlike Frank Zappa, I wasn’t in it for the money. I’m a reasonably well paid academic in a secure position. I appreciated it (moar artisanal muesli for me!), but that is not what enticed me. Now, I’m not categorically against using money as an incentive, and I do think we need a better incentive structure, with more acknowledgement of all the work that goes into producing good research.

Scientists as a Social Species

So what excited me about the Red Team challenge?

It was a paper in an area where I have expertise about a topic that I care about. It was trying out a new technique for improving the scientific literature, and science in general, which is also an area I care deeply about. It was also conducted by people I like and care about – in the capacity as scientists and science reformers. I felt I could make a contribution that was deeply meaningful.

Let me expand a bit on this.  My absolute favorite book on meta science is David Hull’s “Science as a process” which I came across in the late 90’s as a graduate student (and then re-read several times). The thesis he advances (with thorough observations) is that science proceeds in an evolutionary manner**. The knowledge that is amassed through the scientific endeavor does not spring from the individual scientists heads, nor does it require that scientists are dispassionate and ostensibly impartial and objective. No, it is a product of the churning between groups of scientists passionately amassing and rebutting evidence and arguments, as described by Sperber and Mercier in The enigma of reason. It is not fool proof. It is not necessarily efficient (Hull points out that “efficiency” is a value, not something necessary), but it does slowly, incrementally approaches something that could be a little t, provisional truth. I’ve recently seen Cordelia Fine mentioning this in an Aeon article***.

In his conceptualization, the individual scientist is inconsequential, unless they belong to a deme, a scientific community where subgroups either champions or rebuts a particular topic. The social is as important as is the desire to understand the world just a little bit better. A single scientists may have fantastic insights, but these won’t matter if shouted into a lonely void, without others engaging with the ideas.

Science need to have active communities engaging with and arguing about ideas for our understanding to deepen. Hull asks towards the end of the book, what would happen if the argumentation was attenuated or stopped? And my thought is that what has been seen in the science crisis is in part what happens when critique and argument is not rewarded, just production of papers. The inquiry is not corrected if critiques are stopped (but that is for another post to develop).

But, to come back to me, this felt deeply meaningful to me as I felt I contributed to two small areas in science (emotion research and meta-psychology), and that my small, prosocial contribution could possibly help making science better. It doesn’t matter if in the future there will be no facial feedback or the scientific process will take up other procedures than the red/blue team style. It is a small step on the way.


*A challenge to find errors in the paper by Nicholas Coles, Brooke Frohlich, Jeff T. Larsen, Lowell Gaertner and get paid for it. Proposed by Daniël Lakens and Nicholas Coles.

** He is regularly cited by researchers in cultural evolution

*** Fiona Fidler @fidlerfm Replying to @siminevazire

“scientific objectivity depends not simply on scientists being coolly detached with respect to their data, but ‘upon the depth and scope of the transformative interrogation that occurs in any given scientific community’.” (Fine quoting Longino)


Posted in Uncategorized | Leave a comment

Diary of a….

It’s Monday,

slither down the greasy pipe, So far so good no one saw you…

Things feel disoriented. Sometimes things are fine, sometimes this is very weird. I hate that I cannot have vague plans for what happens a couple of months from now, because I just don’t know.

Work goes on. Tomorrow we’ll do the ethics committee via Skype. That will be…. Interesting. I’ve summarized my particular cases (that is what we do, each one of us get two to three cases to summarize and validate and give recommendations for). Then we’ll see if we agree.

I had a short zoom meeting last week, which worked very well, but we were only two people. That is about potential doctoral students. Still a bit of work to do on that one.

I also found out that a local conference/hotel/winery that I really like went bankrupt. People stopped scheduling conferences and they just could not handle it. It is a lovely lovely place, and they have also run the restaurants in two other locations in skåne. The restaurants will (temporarily) close, then open with someone else. It actually really bothered me to the point I was having bad dreams about it.

I see a lot of new efforts about doing valuable research on what is happening. I feel too swamped to be part of it, even if I would like to.

Then, I have moments, Like when I’m listening to Daniel Milo’s Good Enough, and it is just so much fun!

Could I just go home and listen to audio-books until June?


Posted in Uncategorized | Leave a comment

Diary of an Academic in a Pandemic


It is very quiet at work. The lunch restaurant in the building across where I usually get my coffee and a breakfast sandwich has closed. No use now that no students are coming.  On the bus this morning, we were 4 passengers spread across the whole length. I tend to cough throughout the winter. The doctors find nothing wrong with me, it is just something I do after I had my first winter cold. I’m now highly aware of that. Yesterday, an older woman (older than me) who sat right in front of me got up and moved several seats ahead. I can’t blame her.

I got some training on using Zoom, in anticipation of my new course that starts in a week. I think that will work, as long as Zoom holds up.

I feel this sadness, and tiredness. A lot of work and uncertainty yesterday, and now there’s the aftermath like a kind of intellectual hangover. I’ll keep coming in, because I have a nice office, and I don’t think I could work from home with the rest of the family there – my husband has always worked from home.

We have set up some reading courses for the doctoral students who would have, but now can’t, go out collecting data. But most of them are mainly in writing and planning stages, and are doing just fine.

Is this becoming the new normal?

Posted in Uncategorized | Leave a comment

Diary of an Academic in a Pandemic. Wednesday

Premiere on-line seminar! The first pass worked using Conferences on Canvas (which uses something called Big Blue Button). Took a bit before I got my microphone working, but after that we could discuss the two papers I have assigned via either talk, or chat.

Second pass. I get the notice “failed to connect with conference” over and over again! Of course, it is in the sweet spot where everybody wants to teach 10-12 so perhaps a capacity issue? There are a lot of these, particularly for the academic ones. The commercial ones like YouTube and messenger and all that works fine! No wonder.

Let me just say, although I’m happy that this option exists, and that I started prepping for it early, online will not be the teaching method of choice for me.

At least, I know how it works now, and won’t have to fret for next weeks seminar. I have one more to go today. See if Conferences work then.

Posted in Uncategorized | Leave a comment

A little later

I’m getting more anxious and overwhelmed, but I keep going. This is my feeling thermometer. Anxious, a bit sad. It will  change soon.

Posted in Uncategorized | Leave a comment

Tuesday after 11

Tuesday at 11.

The government went out with info that all students at University and High-school level should stay home, and do distance teaching. Us teachers have to then create these courses. Glad I had a bit of startup there for my course. This means my two older kids will be home, while the youngest will be at school.

In other news, one of our teachers found out that a student of hers had Corona (this is all hearsay). So, there is a teacher and a student group that may have been exposed. Yeserday, I had a meeting with this teacher, and we regularly see each other in the lunch room. Hypocondria, here I come.

Posted in Uncategorized | Leave a comment

Diary of an Academic in a Pandemic. Tuesday morning.

Overnight, the university decided that anybody that wants to go to on-line teaching can do so.  This is good, because it gives us some choices. When the line was “business as usual” we couldn’t really break rank. (Ok, so some departments did go out early about potentially prepping for on-line, which I think was very good. This switch does take time and a lot of mental energy).

My co-teacher was preparing to lecture via zoom or something similar, because he was not feeling well. Today he has decided to cancel entirely.

I’m suffering from the “what if I threw an online seminar and nobody came” syndrome. Also, how the hell is this going to work? And also a bit of “I don’t want to do this, can’t we all go home and hibernate until it is all over?”

We will have our crisis meeting soon. I’m learning more functionality from Canvas. I wish I had immediate people to discuss my plans with, whether they were reasonable or out in the left field (can I really ask my students to put together small filmed presentations, without any demand of quality, just because my 19 year old daughter has done that effortlessly since she was 14?)

I am also torn with all the other things that are supposed to be done if everything should function well, and an inability to prioritize right now. How important is the ethics committee? Do we have to have a list of top potential doctoral candidates this week, or can we wait a couple of more weeks (and I’m the deciding body here). How half-assed can my new course really be, considering that there is only doctoral students, and they are clever and self-going.  And, what else important am I now forgetting?

I did yoga yesterday. My gym has decided to close all the halls, but those trainers that care to have taken to giving outdoors sessions. I prefer the “Work out doors” anyway, so the past 3 days, I have exercised every day.

Posted in Uncategorized | Leave a comment