OATAO - Open Archive Toulouse Archive Ouverte Open Access Week

Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees

Cabanac, Guillaume and Preuss, Thomas Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees. (2013) Journal of American Society for Information Science and Technology, 64 (2). 405-415. ISSN 1532-2882

[img]
Preview
(Document in English)

PDF (Author's version) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
1MB

Official URL: http://dx.doi.org/10.1002/asi.22747

Abstract

Peer review supports scientific conferences in selecting high quality papers for publication. Referees are expected to evaluate submissions equitably according to objective criteria (e.g., originality of the contribution, soundness of the theory, validity of the experiments). We argue that the submission date of papers is a subjective factor playing a role in the way they are evaluated. Indeed, program committee (PC) chairs and referees process submission lists that are usually sorted by paperIDs. This order conveys chronological information, as papers are numbered sequentially upon reception. We show that order effects lead to unconsciously favor early-submitted papers to the detriment of later-submitted papers. Our point is supported by a study of 42 peer-reviewed conferences in Computer Science showing a decrease in the number of bids placed on submissions with higher paperIDs. It is advised to counterbalance order effects during the bidding phase of peer review by promoting the submissions with fewer bids to potential referees. This manipulation intends to better share bids out among submissions in order to attract qualified referees for all submissions. This would secure reviews from confident referees, who are keen on voicing sharp opinions and recommendations (acceptance or rejection) about submissions. This work contributes to the integrity of peer review, which is mandatory to maintain public trust in science.

Item Type:Article
Additional Information:Thanks to Wiley editor. The definitive version is available at http://onlinelibrary.wiley.com/doi/10.1002/asi.22747/abstract
HAL Id:hal-01123585
Audience (journal):International peer-reviewed journal
Uncontrolled Keywords:
Institution:French research institutions > Centre National de la Recherche Scientifique - CNRS (FRANCE)
Université de Toulouse > Institut National Polytechnique de Toulouse - INPT (FRANCE)
Other partners > Fachhochschule Brandenburg (GERMANY)
Université de Toulouse > Université Toulouse III - Paul Sabatier - UPS (FRANCE)
Université de Toulouse > Université Toulouse - Jean Jaurès - UT2J (FRANCE)
Université de Toulouse > Université Toulouse 1 Capitole - UT1 (FRANCE)
Laboratory name:
Statistics:download
Deposited By: IRIT IRIT
Deposited On:05 Mar 2015 10:18

Repository Staff Only: item control page