Focus on collaboration, not individual fame.

Fame makes a man take things over
Fame lets him loose, hard to swallow
Fame puts you there where things are hollow (fame)
Fame, it’s not your brain, it’s just the flame
That burns your change to keep you insane (fame)

Written by Carlos Alomar, David Bowie, John Lennon • Copyright © Universal Music Publishing Group, BMG Rights Management US, LLC, Tintoretto Music

 

When I read Diederik Stapels autobiography, I was struck by his self-professed hunger for fame – to be one of those researchers fawned upon by grad-students and colleagues at conferences. To reach fame, he crafted intriguing ideas based on the literature, but the data misbehaved. So he crafted better data.

Now, there is a section in Perspectives on Psychological science called “Special Section on Scholarly Merit in Psychological Science”, all invited papers. The introductory summary paper, by Robert Sternberg is called “Am I Famous Yet?” Judging Scholarly Merit in Psychological Science: An Introduction”

Some of the papers are thoughtful. Roddy Roediger shows that fame is elusive and ephemeral – you are better off working on something that interests you. Simonton summarizes those aspects of eminence you may find, after eminence has already been reached, but laments that those indicators, sadly has low predictive validity, and implores “Please don’t ask that question”. Likewise, Feist advises scientists to disregard the hope for fame and instead balance their work between intrinsic and extrinsic rewards (enthusiasm for the research and possibility for reaping rewards). But others, like Ruscio champion indices like the h-index, as if pursuing science is akin to fantasy football, and that there are good metrics with which we for certain can discern the champions from, presumably, the chaff of the Ph.D’s.

All of these are aimed at the individual scientist, as if science is a lonely enterprise of the hopeful geniuses, but I will take the position that focusing on individual is mistaken. The question on assigning merit (and fame) is the wrong question, if we want to have a science that is worth believing in.  What is important is ideas and knowledge, which emerges collaboratively over time. How can we assure that we have a system that promotes knowledge over fame?

Social Proof and science

Who to bet on is, of course, not unique to science. Recording companies, book publishers, film studios, investors, risk capitalists and gamblers also yearn to find winners, and, as anyone involved in any of these businesses knows, it is a bit of a crap-shoot. Analyzing success in hind-sight can’t, much like Simonton suggests, provide one with easy reliable measures for predicting success beforehand.

Duncan Watts took on this question in his book “Everything is obvious once you know the answer”, but prior to writing his popularization, he and Salganik published a couple of experimental papers tackling the question of predictability in cultural market (Salganik & Watts, 2008, 2009 I will focus on the 2009 paper). The question was, why, if it seems so obvious in hindsight that a cultural product like Star Wars or Harry Potter would become break-out successes, was it so difficult for professionals to predict beforehand that these were good bets? After all, both were rejected by studios and publishers multiple times. In addition, they asked, if we re-run the world would they, once again become hits?

Salganik and Watts took advantage of the internet to pursue this question. They created a platform (this was pre-facebook and Spotify), loaded it with 48 songs from real but unknown bands, and invited 2930 participants. The participants could listen to as many songs as they liked, and as a thank you they could down-load whichever songs they wanted. All they had to do was to rate the songs they had listened to. The participants were sorted into several isolated “worlds”. In the control world, there were only the songs, with no information about which songs were popular. In the experimental worlds, participants were given real-time number of downloads.

The control world, could be considered a base-line, where the ratings of the songs indicated their appeal, as they prefer to call it. The down-load frequency of these songs was relatively flat. No clear hits.

The experimental worlds were different, though. As downloads accumulated, there emerged a clear top group of songs that kept becoming more popular. Intriguingly, the top songs differed in the different worlds. Popularity, or Social Proof (to use Cialdini’s terminology) was a clear factor, and furthermore, it was capricious. Songs rose to the top, because others had chosen them too, not because they had better appeal. But, appeal was not un-important. Songs that were rated low in the control world were never among the top hits. Social proof cannot overcome low-appeal, but once appeal is there, the crowd decides, and it will be different every time.

A scienctific enterprise that looks for stars rather than results is likely to behave similarly. In fact, the term Mathew effect was coined by Merton to specifically illustrate this, and, as the Salganik and Watts papers suggests, those who reap the rewards are arbitrary.

But, should science model itself on this type of businesses?

Rank and Yank

Diener talks with admiration about a system of rating for merit at the university of Illinois, which, he claims resulted in very little “deadwood”. This recalls the “rank and yank” method that was wildly popular among corporations in the 90’s but has since softened, if not been discredited. Employees are evaluated on individual performance and ranked. The top rated receive bonuses and rewards. The bottom performers are let go. There’s plenty of writings about the perils of this technique, but I rely, in this paper, on an article from the Economist (November 16, 2013). The logic was to introduce competition within the company, and thus spur performance, but it resulted in people being so concerned about their rank that it instead spurred secrecy and information poaching. It is considered one of the reasons for the downfall of Enron. One of the problems, as mentioned in the article, is that as you continue this culling, you start firing average worker, but lots of good work is done by the average workers. It also discourages cooperation, which is vital for a corporation to function. The competition should, properly, be between businesses, not within. To Diener’s immense credit his solution is not in discarding, but in implementing systems to help faculty develop.

Paula Stephan’s indictment

Even though the university system doesn’t explicitly use the rank-and-yank technique, its overproduction of PhD’s (in the US system – Sweden does not), and obsessive assignment of merit with the rewards falling to the top makes it a de-facto Rank and Yank system.

Paula Stephan, in her book “How Economics Shapes Science” (2012) provides a scathing indictment of the tournament business model for academic research. She summarizes her arguments in a 2012 nature commentary, and I cite the summary points:

  • Science is full of incentives that encourage bad financial choices, such as expanding labs and hiring too many temporary scientists.
  • These incentives hurt both individual scientists and society as a whole, which gets minimal return on its investment when someone is trained for a field with no career prospects.
  • The way forward is to fix incentives that are damaging the system, by considering their true social and personal cost

Her focus is on the bio-medical complex, which has additional problems, but her analysis can easily be applied to other academic fields as some of the incentives are general.

There is an incredible waste of talent, especially of doctoral students and post-docs, and much of this is because the incentives favors paper-production and citations when allocating resources, and resources are allocated to individuals (at best individual labs) in strong competition. The rewards are reaped by the universities, while workers and the public pay the price, as she writes in her final chapter:

“In one sense, U.S. universities behave like high-end shopping malls. They are in the business of building state-of-the art facilities and a reputation that attracts good students and faculty. They then turn around and “rent” the facilities to faculty in the form of indirect costs on grants and the buy-out of salary. Faculty, in turn, create research programs, staffing them with graduate students and postdocs, who contribute to the research enterprise by their labor and the fresh ideas that they bring, but who can also be easily downsized, if and when times get tough. Universities leverage these outcomes into reputation. The amount of funding universities receive, as well as the citations and prizes awarded to their faculty, determine their peer group—the club to which they belong. They also attract donations and students and affect the university’s ranking.”

 

Science as a process.

The gist of the symposium seems to be that it is of great importance to identify and credit meritorious – eminent – individuals, and that considerable time should be taken to perfect this system of credit, but is it really the eminent individuals that drive science forward?

My go-to author on philosophy of science isn’t Kuhn or Lakatos, but David Hull, and specifically his book “Science as a process” from 1988. His thesis is that science advances in an evolutionary manner. A wealth of ideas are produced, only some of these are selected and survive, and which ideas survive depend both on their scientific merit as well as a social process, the process that involves production of papers, citations, and engagement of groups of scientists. Ideas that are not interacted with will die, no matter how profound they are. Ideas that are interacted with by groups of scientists (demes, as he calls them – borrowed from evolutionary biology) will grow and change, and perhaps bring our knowledge closer to the truth. I provide two citations from the preface and first chapter

Preface:

“In the manuscript, Nelson (1973c) complained of the way that the views of Leon Croizat had been treated through the years by such authorities as G.G. Simpson and Ernst Mayr. I decided that the sort of thing Nelson was investigating with respect to Croizat was the sort of thing I would like to do in philosophy of science.  What is the relative importance in science of reason, argument, and evidence on the one hand, and power, prestige, and influence on the other? I thought that answers couched totally in terms of one sort of influence or the other were sure to be wrong and that the interplay between the two was likely to be fascinating.”

Page 3, Chapter 1

The system of cooperation and competition, secrecy and openness, rewards and punishments that has characterized science from its inception is both social and internal to science itself. The conceptual development of science would not have the characteristics it has without this social system. Quite obviously science is a social process, but it is also “social” in a more significant sense. The objectivity that matters so much in science is not primarily a characteristic of individual scientists but of scientific communities. Scientists rarely refute their own pet hypotheses, especially after they have appeared in print, but that is all right. Their fellow scientists will be happy to expose these hypotheses to severe testing. Science is so structured that scientists must, to further their own research, use the work of other scientists. The better they are at evaluating the work of others when it is relevant to their own research, the more successful they will be. The mechanism that has evolved in science that is responsible for its unbelievable success may not be all that “rational,” but it is effective, and it has the same effect that advocates of science as a totally rational enterprise prefer.

(Emphases mine).

This is very far from the focus on eminence and individual fame. Competition is a factor, sure. Some people thrive on competition, and it can be engaging to take on a theory or an experiment to expose its flaws (and merits). But, cooperation is vital. Producing original research is just about always team-work, involving co-authors, researchers, assistants, and in psychology, participants. And, for the ideas to survive, multiple labs need to engage with the ideas either as champions or as severe adversarial testers. In this churning and testing of ideas, we may come closer to understanding our world. (I’m reminded of Mercier & Sperber’s (2012) theory how reasoning is improved via argument)

If we only focus on who may become eminent, or who is eminent, we are losing some of the power of the scientific process. The eminent scientists would be nowhere without the collaborators and the adversaries that are willing to engage with the ideas, and science is littered with these sole ideas that went nowhere. We just don’t know about them, because like failed commercial products, they disappear.

I would also argue, without much evidence that in the focus on production and individual eminence, and protection of reputation, the argument portion – the other scientists “happy to expose these hypotheses to severe testing” has broken down. The tendency to overwhelmingly publishing only positive results in psychology, based on underpowered studies, with no clear avenue for publishing failures to confirm means that as scientists we are not grappling with the real field, and the social churning that Hull describes cannot take place (see Chris Chamber’s recent book).

On a more hopeful note, I’m reminded of the current focus on improving our methods and statistics in psychology. The complaints about business as usual are old. There are continuous reports of authors who complained about p-values, the use of NHST, etc, published 20, 40 or 50 years ago, that were never heeded. Some of these are, of course, eminent (Cohen, Tukey, Meehl), and many of us came across their complaints in graduate school, but then the business of publishing and surviving took over, we copied the social practices of those who had stayed in business (see Richerson & Boyd for a discussion on mechanisms of cultural transmission), and buried our concerns. What may be different this time is that, through interconnectivity via social media, we are no longer lone voices in the wilderness, but can build alternative demes of champions.

Considering that good scientific progress depends on a collaborative social process, it is misguided to focus on potential superstars and the accumulation of individual merit. Instead, look at how to better create a collaborative environment – at least within individual departments (or across virtual academies), where the diverse talents can come to their own right in joint efforts. Dieners suggestions are a start, but I think we need to go further.

The obsession with publications and citations.

Diener mentions that Sternberg has published 1500 papers. Of course this is impressive. Presumably each of these papers involved an action editor and a median of two peer peviewers, as well as an army of co-authors, research assistants and (since it is psychology) participants. Elsewhere he laments the low productivity (1.5 papers a year), and that most papers are never cited. Is this focus on productivity a viable avenue to proceed?

We are flooded in a tsunami of papers. There are more researchers, and higher pressure to publish, and it is now impossible to overview even one’s own field. With so many papers, cumulative science becomes near impossible. I recently rejected a paper because the introduction did not bring up crucial theoretical and empirical papers for their work (the crucial papers were older), and the method they used did not connect with the extensive development of that method thus they used it inappropriately. This is not the first time.

Sure, high productivity can be good fodder for that selection process outlined by Hull, but, to spin further on the evolutionary idea, there seems to roughly be two strategies for passing on genes to the next generation – the r and K selection strategies. In the r-selection strategy many offspring are produced (fish, Birches), little effort is invested and most of the offspring become food. In the K selection, such as humans, few offspring are produced but they are then carefully nurtured. Both are viable strategies, but tend to depend on whether the environment is stable or not (in evolutionary time). In fact, this is echoed in Feist’s “Prescription for a successful scientific career”, especially his figure of productive scientists as adaped from Cole and Cole. He also, importantly, points out that what, in the current system, is good for the individual may not be good for the field. (This tension between what is good for the individual and what is good for the group/field turns up frequently within those areas that take an evolutionary view of different developments, such as Evonomics, Clio-dynamics)

The problem of measurements

But current measurement system in science does not favor the slow, nurturing type of creating and developing ideas, which perhaps is at our peril. Ruscio made much of the objectivity of the h-index as a good selective mechanism for identifying the stars. First, this presupposes that the peer review system assures that papers are reasonably solid, and that citations can be used as a reasonable proxy for quality.

But, as others have also argued, citations seem to work much more like popularity in the Salganik and Watts (2008, 2009) papers, and citations fill multiple roles in a paper, where some are more central than others.

I recently undertook a project where I looked at all of the papers that cited Srull & Wyer(1979) for the first 5 years after publication (53 in total). I extracted all of the citations from the papers (when possible). For the vast majority of the papers, Srull & Wyer were only cited once. I give you a typical example.

However, because of recent theoretical and methodological developments in cognitive psychology, considerable effort is now being made to analyze these operations (Carlston, 1980, Ebbesen, 1980; Hastie & Carlston, 1980; Srull & Wyer, 1979; Wyer & Carlston, 1979; Burnstein, & Schul, 1982.)

Note that the single citations were appropriate! When we write papers, we do a lot of single citations to indicate where we get the ideas even if we are not directly building on them, and this is incredibly useful. But this is where the popularity comes in. We need something to cite for part of the argument building up our introduction, or clarifying our conclusions, and most likely we have a series of go-to papers to cite. This is now built into citation indices.

The h-index is considered good, because it can presumbably not be gamed, but one should more properly say, one hasn’t figured out how to game it yet. But, even that is not true. Dorothy Bishop uncovered an h-index boosting ring involving several authors and editors at separate journals, which she describes in a series of blog-posts.

This focus on rewarding frequent spawning is also one that opens up for poor scientific practices such as salami slicing, corner cutting, questionable research practices and p-hacking. It has been proposed by many, but I think the paper by Smaldino and McElreath (2016) where they model the consequences is illustrative.

A recent paper using net-work modeling that attempted to find when, in a career, a scientists most impactful work occurred, and the distribution is random!(Sinatra, Wang, Deville, & Song, 2016). The measure is still focused on the individual, as one factor is productivity, with the two other factors being a factor Q (which may indicate creativity), and a factor for luck.

As Ulrich Schimmack (among others) have pointed out, there is also no viable disincentive to publish weak and irreproducible work. Sure, the vast majority of papers go unread and uncited, as mentioned by Diener, but they now clog the publication record as so much algae soaking up oxygen. The only predators are the predatory journals, and they simply add to the problem.

I don’t want to take away from Sternbergs impressive productivity (surely he belongs among the eminent), but productivity varies and cannot be used as a proxy for quality. It is necessary to keep spaces open for the lower producers, and perhaps for those broader, cross-disciplinary collaborations with lower yield, but, in the end, perhaps with higher and more long lasting true impact.

Not everyone thrives on competition

My daughter recently quit her elite-team of team-gymnastics, a sport she has enthusiastically, almost obsessively pursued for 10 years. As she approached the top, the stress took all of the fun out of the enterprise, and after the last competition – where her team did well – she felt the effort was no longer worth it.  In many ways, she’s her mother’s daughter.

Competition can be engaging, and for some individuals spur them onto performance, or possibly reaching for the enhancers, but for some it can become demoralizing. Thus, it isn’t clear that competition is the sole way to go in order to maximize performance.

Some years back, I watched an intriguing colloquium by Uri Gneezy (the research is discussed in this paper Gneezy, Leonard & Gist, 2009). The colloquium can be found on Itunes U in the “Center for Behavioral Evolution and Culture” series from UCLA) where they investigated gender-differences in competitiveness. They were particularly interested in teasing apart possible cultural influences, and thus they took the pains to locate a patriarchal tribe (Maasai), and a matrilineal tribe (Khasi). The task was a simple ball-throwing task with the goal of getting as many balls as possible into a bucket. The task was specifically selected because there were no gender differences in ability. Participants could select whether they wanted to do it competitively (the winner took home all the rewards), or piece-meal (you got paid according to how many balls you got in). In the Maasai group, many more men chose to compete, and as Gneezy mentions in the task, you would get a similar result if you tested UCLA students. In the matrilocal group, the pattern was reversed. As he mentioned, there were not enough reliable data to discern who performed better, but the focus was on choosing to compete. He also mentioned research by Dreber and Hoffman that have investigated gender and competitiveness in children across several cultures, and there appears to be clear cultural differences, that seems to co-vary with how egalitarian the culture is.

Now, if we posit that in a group of western men you will find more individuals that elect to compete than in a group of western women, setting up an enterprise so that it rewards those that are more willing to compete (fair or foul) may very well stack the deck against women, even when the sheer intellectual ability and creativity are the same. As Alice Eagly lamented, where are the women among the eminent (and surely there are some). Perhaps it is because they don’t respond to the same incentives. We do know of instances where men have reached glory based in part on work of women (the Watson and Crick story tends to be top of mind). Surely there are more women (and non-competitive men) on whose shoulders those eminent have clambered up upon, without necessarily giving credit where credit is due, distorting our perception.

Via Negativa

So, how are we going to proceed?  In the book “Antifragile” Nassim Taleb lays out a strategy for betting in an uncertain world – optionality. We know that graduate schools attract just about only the best minds for doing science. These are people who want to do scientific work, and are competent enough to be admitted. As Simonton points out, we don’t have reliable indicators on who will end up pursuing the runaway success that will advance a scientific field, and as Taleb points out, there isn’t a way to know. Instead, once the “low appeal” have been sorted away, do an even bet on all. That is, he says, how at least some venture capitalists work when funding start-ups, and is part of his own strategy when investing. This is a non-linear enterprise, and even one wild success can pay for all the non-producing bets.

He calls this via-negativa. You can quickly sort away those that seem unpromising (the non-appealing songs), but then place your bets evenly on the appealing ones.

Perhaps better still, following Hull, don’t bet on individuals. Create collaborative groups working on problems.

Science is high risk and low yield

Like so many others, I have a file-drawer of ideas that didn’t pan out. We are at the edge of knowledge, and most of our attempts are, most likely, carving out what doesn’t work. Putting a productivity demand is, like I have pointed out, likely to distort rather than enhance.

Most of the research work at Universities is tax-payer based. Mariana Mazzucato is currently advancing the idea that the state here functions like a risk capitalist – an entrepreneur (Her book the Entrepreneurial state is a must read, but her Ted talk gives a reasonable, short overview). The internet and GPS (among others) are the results of basic projects financed by tax-payer money (frequently via the military). Like the risk-capitalists I mentioned in the Via Negativa they fund risky ideas where only a few will yield dividends, and most likely far in the future, and the dividends may also be reaped by private entrepreneurs, such as Apple. The pay-off for the tax payers, she proposes, should be in the taxes the eventual successful projects may reap.

But, in the mean-time, scientists need to be allowed to pursue risky projects, with a high likelihood of failure. Treating scientists like factory workers is unlikely to be a good strategy. Scientists shouldn’t be punished for failing to find anything interesting, but perhaps for producing sloppy unreliable work.

The myth of the eminent scientist

I have had students who wistfully said that they could never become as great as a Newton or a Darwin or a Skinner (to pick the one eminent psychologist that Roediger thought most of us knew something about). I pointed out that not even Newton or Darwin or Skinner were the same as their mythological figures. Don’t let the myths stop you from pursuing your dreams.

Hull brings up how we use mythologized older scientists (who may be dead, and thus can’t protest their image) as a rhetorical devise to lend credence to one’s own ideas, which is one reason why we may want to mythologize some that came before.

Perhaps also our human tendency to look for individuals with prestige in order to learn from them has something to do with it, but I think this remains an interesting psychological conundrum to test. (We also have a number of mythologized experiments, as we have discovered, such as Milgram, and the story behind Kitty Genovese, which evidently the journalist responsible for the initial article polished to fit a story he wanted to tell more than the messy facts).

It may also be a way for us to cognitively sort and tag ideas. If I mention Ekman and Russell to fellow emotion researchers they know that they are the proponents of the categorical vs the dimensional theories of emotion, even though they are not the sole researchers pursuing and testing these theories. Perhaps it would be better to move towards theory names rather than researcher names.

A great antidote for the myth of the great man (because they are mostly men) is to read some good historians of science (My favorite is Thony Christie @rmathematicus) where they carefully excavate the real scientist from the bronzed mythology.

Matthew Francis recently wrote a scathing blog post against the Nobel Prize. It was anticipated that the 2016 Nobel in physics would go for work on LIGO (the Laser Interferometer Gravitational-wave Observatory). (In an upset, the Nobel did instead go to work on topological phase transitions). His objection was not that a Nobel wasn’t deserved. He thought the achievement was fantastic. His objection was that the Nobel continues to manifest the mistaken notion that science is advanced by eminent individuals rather than communities of researchers, and this false emphasis on eminence. As he says

“The Nobel Prize is simply … a reminder that despite our advances, we still promote the idea that Science is done by the Lone White Male Genius, maybe with an adoring female assistant standing by to do the thing.”

Let’s move away from Science as an enterprise for fame, from the cultural markets model. Focusing on merit will not fix science, it will not fix the woman problem, and not fix the non-white-male problem. Move more towards the collaborative, cumulative work that seems to define our species (Henrich). As one eminent scientist allegedly said (Newton) he stood on the shoulder of giants. Well, perhaps the shoulders are not of giants, but of all the humans that came before and all the humans that collaborated and to crown a single king of the mountain is to distort what it actually takes to move science forward.

 

Coda

Let’s give Stapel the fame he craved. He wanted it, he faked it, and came clean. He should never work in science again, but let’s enshrine him in the history of psychology, to remind us about the danger of seeking to reward eminence, over the hard collaborative work that really advances science.

Partial references (because it is a blog…)

Ranked and Yanked. The Economisthttp://www.economist.com/news/business/21589866-firms-keep-grading-their-staff-ruthlessly-may-not-get-best-them-ranked-and-yanked November 16, 2013. Pulled December 1, 2016.

http://deevybee.blogspot.se/2015/02/editors-behaving-badly.html Pulled December 1, 2016.

https://galileospendulum.org/2016/10/03/dethroning-the-nobel-prize/ October 3, pulled December 1.

Salganik, Matthew I., & Watts, Duncan J. (2009) Web-Based experiments for the study of collective social dynamics in cultural markets. Topics in Cognitive Sciences 1, 439-468. DOI: 10.1111/j.1756-8765.2009.01030.x

Salganik, Matthew J. & Watts, Duncan J. (2008) Leading the herd astray: An experimental study of self-fulfilling prophecies in an artificial cultural market. Social psychology Quarterly, 71, 338-355.

Stephan, Paula (2012). Perverse Incentives. Nature, 484, 29-31.

Stephan, Paula (2012). How Economics Shapes Science. Harvard University Press.

Advertisements

About asehelene

... because if I'm in a room with a second person, I want to be reasonably sure I'm the crazier one.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s