A 1982 BBs paper on peer review from Peters & Ceci is making the rounds on Twitter. The gist is that they submitted already published papers, with altered author names, to scientific journals, and received rejections that had nothing to do with the papers being basically plagiarized, but more comments on the quality of the research
This shows that publication/peer review may be a bit of a crap-shoot. Maybe. (I’d rather take my chances with publication than with die-throwing though).
But, it reminds me of a couple of papers I’ve assigned to my students in my market psychology course, both from Salganik and Watts. Instead of peer-reviewing paper, they look at the entertainment market – music. (Both papers linked here, and below)
Let me dolly back a bit. (cue Haitian divorce). My course is based on Cialdini’s “influence”. The core here is that people use a number of cues or signatures to decide on their actions rather than taking the time to carefully examine the actual quality or reasons. This works, because the co-variation between the cue and quality are usually quite high, although not absolute. So, for example, when deciding to publish something, you may also rely on the reputation of the author. A well-known, established scientist are reasonably more likely to produce high quality research than a no-name – although not for certain, of course. Everybody does crap, and no-names can do great stuff. Of course, in science we are supposed to not be swayed by this, but we clearly are. And, Authority is one of Cialdini’s ways to yes. We are persuaded by authority, because usually people come by their authority in a reasonable way (long experience, high education, etc).
The music papers looks at another cue to quality – social proof. The “1000 elvises can’t be wrong” cue. Early on they bring up the issue – why is a particular film, or book, or painting, or song popular? And, if you are a fan, you quickly bring up all the good qualities, and the obvious reasons why this piece of art or entertainment has become such a mega-hit. But, and anybody having done any kind of marketing knows this, if the quality was so obvious, how come a script like Star Wars, or a book like Harry Potter were rejected by studio after studio, or publisher after publisher? Is the quality or appeal really that obvious? (The answer of course is, No). So, why do things get popular? Why do things go viral? Is there a magic sauce? An alternative, of course, is that it is popular because it is popular. Social proof. And, of course, marketers have been aware that exposure is important, getting people to start liking something is important. So there has been as long as I lived Lists. Top music charts (I religiously listened to “Svensktoppen” and “tio I top” when I was young, although I never always agreed with what was considered good). And, there has been attempts to manipulate charts and exposures (payola), and counter measures (make payola strictly forbidden). But, even with high promotion, lists, and payola, things have flopped and flopped big, and unknowns have raced to the top.
Right around when I started using these papers, I listened to this interview with Steve Meyers on Econ Talk (did R & D for Capitol records) where he basically illustrated the capriciousness of the business.
It’s been known that social proof is a big factor for a long time in both social psychology and sociology, but studying it in the lab has been, as Salganik and Watts points out, well, hard. How many people can you cram in there? But, these days you can become good pals with a whole bunch of invisible friends all over the planet via the magic of the internet, so why not use that as a lab? (Yeah yeah, that is old news now, of course, but they actually ran their experiments pre-face book and mechanical turk). The cultural product is one that just about all of us know and love – the pop-song. The (original) target market, young people hanging on the net.
The first paper was about exploring the strength of social proof in the form of a popularity list. They had gathered 50 songs from basically unknown bands (so much music, so little time). They recruited participants through an on-line site and told them to please listen to and rate some songs. As a reward, they could download one of them for free!, but they had to listen to them first. And, no demand to listen to all of them of course. Just listen to how many you like.
In non-influence world, the songs were just there, with no ranking. This is a control world of sorts. And, also possibly one where you can get a sense of a songs appeal. They use appeal because they thought it was a bit strange talking about quality when it comes to cultural products. Quality and appeal can be at variance (at least to culture snobs).
Then they created two test-worlds. Basically the same set up, but in world A, the songs were listed and ranked according to the appeal they uncovered in the control world. In world B, the ordering was entirely reversed.
They let them run, with an awful lot of participants. And, yes, being ranked top really had an effect. Very much more listened to and down-loaded. But, appeal was not unimportant. For the reversed list, over time people discovered that those low on the list were better, and they started to recover some popularity. The vast middle was, of course, entirely overlooked.
In the next paper they ask the question – if we could run the world over again, would the kids still ask for light-sabres and know the rules of quiddich?
Similar set up with a control world. Then they created 8 separate worlds, where the ratings and rankings of the songs were visible as they were changing, and let them churn. It resulted in 8 different top hits. Harry Potter may not have made it in a parallel world. They re-ran the experiment, but now with a set of a bit older folks, testing whether this kind of flock-behavior perhaps was peculiar to the young people. But, the results were rather similar. Some differences, but overall different hits, an effect of social proof.
Makes the lives of R & D people difficult. In fact, all developers of new stuff. You just can’t know what will be the new new thing (to go with a Michael Lewis title).
But, there was one thing that they could see, with regards to appeal. The low appeal songs never made it to the top 10. You can tell lack of quality. A quick sort of chaff. What sucks will keep sucking, and won’t be fixed in the mix. But, for those with appeal, it is a gamble. A similar idea comes in Taleb’s Anti Fragile from his notion of Via Negativa. Focus on what to avoid. For the rest, you just can’t predict the future.
We frequently stray into talking about music, downloading and the music-business in this class, even prior to me assigning those papers. I keep seeing some parallels between songs and papers. The argument against the pirating was always that you steal someones intellectual property, and the musicians/song writer ought to be paid. I can kind of see it, but I think it is spurious (I think it is the appeal of a dying business model instead). Of course, I get paid by the tax-payers, but the copyright to the few papers I have produced have been signed away. I freely give away my intellectual property once published (I’m stingy with my damned power points though. Take notes, people. It is good for you! And get off my lawn).
And, I’m thinking, what is the reason, really, to support scientists doing basic science, especially now that there is such a ridiculous over-production of them. Anymore than supporting all other types of arts that can’t generate their own income stream. (There probably are reasons, but for me they tend to be, because, well, it is nice. I like being able to go to the opera, or to a museum, and a vague sense that it is important to preserve this, and support this, but I can’t give a compelling reason which isn’t also deeply biased towards my own preferences of being able to pursue science, and go to the opera, and other nice stuff. But, the tax-payers have to support that. Then again, I think that some of those too big to fail banks really should have gone bankrupt, but let’s not go there).
Will we become garage-band scientists? Set up science gigs and vernisages? (Here, you get to be on the guest-list to my latest experiment). Then again, if you are happy with the one-voxel imaging – or perhaps even fractionate voxels – doing psychological research is cheap. I advised 10 papers this spring (don’t ever do that. Don’t let me ever do that again.) For several of them, I talked them into collecting data for projects I’m interested in. Some are kind of piloty. Others more than that. Mostly collected via Google docs or similar over the internet. The university did not have to shell out anything in support. Sure, there are real limitations to this. Some things you need to do in a lab proper, or with carefully selected samples. but, the advantage, as Salganik and Watts show, is that you can now have large labs with lots of participants.
And peer review? Already, there are interesting comments on the net, on blogs, in comment sections (as I have talked about). And, I’m sure all Cialdini’s factors will be involved – Reciprocity, Authority, Social Proof, Liking, Committment and Consistency (my theory is Right, and I stand by that), and Scarcity. But, some quality will still matter. That little t truth.
I’ve mostly summarized Salganik & Watts papers from memory. But, here are the references and links to their papers (most likely pay-walled)
Salganik, Matthew I., & Watts, Duncan J. (2009) Web-Based experiments for the study of collective social dynamics in cultural markets. Topics in Cognitive Sciences 1, 439-468. DOI: 10.1111/j.1756-8765.2009.01030.x
Salganik, Matthew J. & Watts, Duncan J. (2008) Leading the herd astray: An experimental study of self-fulfilling prophecies in an artificial cultural market. Social psychology Quarterly, 71, 338-355.
Or, get Duncan Watt’s book, “Everything is obvious, once you know the answer.” (I have it as a sound book, which he reads, in a nice broad aussie accents. To me, Aussie accents have quite the appeal).