Can an AI tell how trustworthy long dead people are from their portraits?

I was just set to teach a segment on AI and facial expressions of emotion when twitter started circulating a just published Nature paper, where the authors trained an AI on detecting trustworthiness DISPLAYS, set it loose on the faces of curated sets of historical paintings, and then claimed that this was a way to possibly detect historical shifts in trustworthiness, at least tentatively so. (Safra et al., 2020)

I’ve been thinking of it for a bit, but wanted to take a stab at my thoughts, with a lot of places that is up for critique.

A short summary.

The face-perception literature have identified face-displays that people tend to judge as either trustworthy or not trustworthy. It is important to think of these as displays and stereotypes, rather than indicators of trustworthiness. (Think the gangster-film trope of Baby-face so-and-so who invariably is the nastiest of the lot). Remember, this research is done on contemporary people, many (but not all) from western countries.

The researchers have trained an algorithm on these stereotypical displays, and then let it classify a set of digitized curated historical (and western) paintings. The aim was to see if one could see a shift in the use of the trustworthy (often smiling) display across time. The prototypes are shown in the paper. For anybody that have been mildly involved in face-research, these are face-gen faces*.

They see a shift towards more of the “Trustworthy display” as time approaches modern times. In their discussion, they take it as evidence that trustworthiness has increased across time. As many have pointed out – this is going well beyond the data.


I do faces – kinda. I’m interested in emotions and emotional expressions – and god knows that is an area that is getting complicated, because of course it is.

I’ve always been puzzled by rating faces on trustworthiness. (See my blog post on the “brown eyed people are more trustworthy I participated in a rating exercise once, where the task was to rate a series of faces on trustworthiness. I gave up after a while, which probably was good for the researchers.  I had used that facial data-base before so they were familiar to me. But, also because it felt entirely arbitrary. How the hell can I rate a random person’s trustworthiness? Trustworthiness – at a first glance – is a property of a person.

Thinking a bit more deeply about it – trustworthiness is a property of relationships. As an extreme – think a spy. Trustworthy in the home-country, deceptive in the foreign country. 

The importance of being trustworthy – at least towards your in-group – is socialized into people. Deception is punished, trustworthiness rewarded. We seem to spend a lot of time on teaching why it is important to not lie, to be honest, and not break your promises. (Think the boy who cried Wolf).

Trustworthiness may also be something we afford our (sometimes) extended in-group. You trust a friend, because you believe they value your friendship and thus wouldn’t betray you. You may trust people in your home-town because being untrustworthy would result in a bad reputation, which has a lot of negative consequences. You may trust a distant relative, or the friend of a friend of a friend.

Of course, it is interesting to understand what kind of stereotypes people have about all sorts of things, including faces and face display, especially since people tend to use these to make decisions. I’m reminded of that early Todorov paper where people had to judge who they thought won a particular state-level (US) election based just on the faces of the competing politicians. Taken together, the participants were right at above chance levels. A recent pre-print has connected impressions from faces on traits such as trustworthiness (among others) with the stereotypical beliefs people hold about what different types of faces expresses. We seem to rely on stereotypes to make our judgments, regardless of whether they are correct. Faces matter, even if we really cannot discern anything about the person based on it.

I also suspect stereotypes are moving targets. We spend a lot of time (righly) on the harmful stereotypes, and not that much research on why we have stereotypes at all (apart from Lee Jussim). I figure it is for the same reason we have lots of categories and prototypes – it helps us orienting ourselves in the surroundings. When the surroundings change, the stereotypes are updated.

Smiling norms and immigration

In this course, I also include a paper by Rychlowska et al., 2015 where the group looks at differences in smiling norms between contemporary countries. They test a hypothesis that there will be more smiling in a country consisting of immigrants from many different parts of the world, than in countries that have had little immigration.** Roughly the idea is that in a country with little immigration, norms of behavior, including emotions are stable and predictable. There is then little need to frequently display intentions through, for example, gestures and facial expressions – but as they point out, it does not make them meaningless, but perhaps even more meaningful for those instances when they are used.

However, in a country built on relatively recent immigration, like the English speaking colonies for example, there is much more uncertainty about other peoples intentions given the varying cultural backgrounds, so it becomes quite important to signal positive intent – perhaps trustworthiness.  The smile seems to have come overloaded with social affiliation/social negotiation meaning, being an appeasement signal or reward/affiliation signal. Hence, more smiling.  This is what they find in their paper. (How stable and reasonable this is, I haven’t checked, but it seems more plausible to me than the notion that brown-eyed people are more trustworthy).

This leads me to think that the conclusion of the paper (with all the caveats) should be the opposite. As time went on, and travel became easier and networks broader, the assumption of trustworthiness became more tenuous, so it became increasingly important to signal trustworthiness.

In both papers, to be fair, they bring up that there is separate research on the development of institutions and norms to support trust in large interacting network. The increase in smiling as individual signals of trust could have happened together with these developing institutions. I’m going through Heinrich’s “The weirdest people in the world” concurrently with this, where he lays out his educated thought about this cultural evolution and the extension of trust beyond kin-groups.


I collected some commentary in the form of blogs and twitter-threads collected via the very useful threadreaderapp.

Iris Van Rooij Blogpost on the paper

Iris van Rooij’s twitter comments

“shall I read the computational phrenology paper for your entertainment? By Sonia

Andreas Kirsch Twitter comments

Matteuz Fafinski’s twitter comments

Older medium article on “The new physiognomy” by Blaise Agüera y Arcas, Margaret Mitchell and Alexander Todorov

How Possible

I do think one can use AI as a tool, if it is done carefully, with recognition that it is just a tool – perhaps somewhat more sophisticated than the statistical algorithm’s we gleefully throw at our data with (sometimes) incomplete understanding of how it works.

We come with faces. Faces that give rise to impressions. I have, with age, acquired maybe not quite a resting bitch face, but more a resting tired face. My husband has the resting disapproving eye-brows; thick, dark and down slanting.

As copious research shows, we use other people’s faces to rapidly infer a lot of things about them – even things that we know cannot be inferred, such as trustworthiness.

Faces also come with lots of muscles that we constantly move. Chewing, speaking, expressing emotions. (I find myself moving my face a bit while writing this, and it has nothing to do with communicating to others.) The facial expressions we make can be tied to underlying intentional states. Sometimes we can’t help expressing our emotions, even if we try to suppress them. But, we also have quite a bit of control over our faces, and use them in ways similar to talking – to communicate with others. (See, for example, work by Alan Fridlund). We don’t have to feel happy in order to smile.

We have known for a long time that facial expressiveness is in part learned, and influenced by culture (just see that Rychowska article above) – and what we learn are both displays, and interpreting what we perceive. (We really have to divide these up, and I don’t think that is an easy distinction to grasp, coming from my own experience – although maybe I’m particularly dense).

Looking at how display rules have evolved over time – in a particular class in a particular culture – could be very informative, together with other historical data. I do think this was the impulse that drove the researchers, although I think they have wildly overstated what could be inferred from the changing display preference psychologically.

But there still are some issues. The training set are on modern day stereotypes of “trustworthiness” (see Iris van Rooij’s blog post). That is, they fit some average of face-type that present day living people tend to judge as “trustworthy” (without there being a correspondence to actual trustworthiness, as has also been experimentally shown). So, they are not training on displays (smiles). They are training on physiognomy of a stereotype. So the question they answer, perhaps, is – can we see the development historically of the current stereotype of trustworthiness? Which actually seems rather nonsensical to me, now that I write it out.

*Face-gen being a program where you can create artificial human faces with different expressions. So, lots of stimulus control. I have used a predecessor to this – Poser – to do some work.

**This is in historic time, the past 500 years. Humans have this propensity of traveling around the globe and settle places that seems to stretch at least 40 000 years – or more. In some sense we are mostly immigrant populations already, though some of us immigrated for forever ago).

About asehelene

... because if I'm in a room with a second person, I want to be reasonably sure I'm the crazier one.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s