Tag Archives: Publication

On Publishing and Cost Benefit Analysis

On his blog, Gilles Saint-Paul comments on the publication process in economics.

Of course I was wrong in all accounts. The publication process in economics is not a publication process, it is a validation process by which we acquire a certain rank in a certain pecking order. Submitting a paper to a journal has nothing to do with research dissemination, it is far more similar to taking an exam or participating in a sports competition. The actual dissemination takes place mostly orally, in seminars and conferences; these seminars and conferences are also important validation events, because they allow authors to signal some of their characteristics that may influence their position in the pecking order, while not being easy to infer from their papers.

Now, when you take an exam as a student, you are graded by your professor, not by a fellow student – who would be a competitor if this exam is actually a contest. …

Yet this is the way our own profession is organized. Each submission is “peer reviewed’, that is, it has to be accepted by anonymous referees who happen to be participating in the same beauty contest as the author(s), most often in the same subcategory. At a minimum, as believers of cost-benefit analysis, we should consider that the journal editors and referees themselves perform a cost-benefit analysis when deciding whether or not to publish a paper. I must say that if I apply such a theory to explain my own experience with acceptances and rejections, I easily get an R2 of 80 %.

Doubts about Empirical Research

The Economist reports about research by Paul Smaldino and Richard McElreath indicating that studies in psychology, neuroscience and medicine have low statistical power (the probability to correctly reject a null hypothesis). If, nevertheless, almost all published studies contain significant results (i.e., rejections of null hypotheses), then this is suspicious.

Furthermore, Smaldino and McElreath’s research suggests that

the process of replication, by which published results are tested anew, is incapable of correcting the situation no matter how rigorously it is pursued.

With the help of a model of competing research institutes, Smaldino and McElreath simulate how empirical scientific research  progresses. Labs that find more new results also tend to produce more false positives. More careful labs try to rule out false positives but publish less. More “successful” labs are allowed to replicate. As a consequence, less careful labs spread out. Replication—repetition of randomly selected findings—does not stop this process.

poor methods still won—albeit more slowly. This was true in even the most punitive version of the model, in which labs received a penalty 100 times the value of the original “pay-off” for a result that failed to replicate, and replication rates were high (half of all results were subject to replication efforts).

Smaldino and McElreath conclude that “top-performing laboratories will always be those who are able to cut corners”—even in a world with frequent replication. The Economist concludes that

[u]ltimately, therefore, the way to end the proliferation of bad science is not to nag people to behave better, or even to encourage replication, but for universities and funding agencies to stop rewarding researchers who publish copiously over those who publish fewer, but perhaps higher-quality papers.