For a change, this post is actually related to NLP. It’s about rules, but not the linguistic kind, but those that govern our research communities.
Not all proceedings are proceedings
For ACL 2017, the Annual Meeting of the Association for Computational Linguistics,1 the program chairs have created an accompanying blog, the PC Chairs Blog, in order to increase transparency and communication. This is a great initiative to base the further development the field’s premier conference on a broader basis.2
One heavily debated topic is the handling of papers already published as preprints, in particular on arXiv.org; see the posts Statistics on Submissions and Status Update, arXiv and the Future of Double-Blind Conference Reviewing, and arXiv and Double-Blind Reviewing Revisited. The current policy is that such papers are permitted, if the fact is declared; the current policy is also that papers that are not anonymized or that have been published as preprints, but where the authors fail to declare this, are rejected without review.
As area chair for the Multidisciplinary Track, I had the opportunity to experience the problems with this policy at first hand. One of the papers submitted to our track had already been presented—in identical form—at conference X and was available online on the conference Web site. Now, the ACL 2017 call says:
The ACL 2017 conference invites the submission of long and short papers on substantial, original, and unpublished research in all aspects of automated language processing.
However, this is followed by a “Multiple Submission Policy,” which states:
Preprint servers such as arXiv.org and workshops that do not have published proceedings are not considered archival for purposes of submission.
I find it striking that in this section we’re suddenly no longer talking about “publishing” but about “archival.” I would think that the primary goal of preprint servers and workshops is public dissemination, not archiving. I have never seen ACL conferences advertising “archival” either. What is more, I have not seen any explanation of what “archival for purposes of submission” is supposed to mean, because the requirements mention “substantial, original, and unpublished,” but not “unarchived.” I have the suspicion that “archival” here really means “published.” So what the Multiple Submission Policy really says is that if you publish your paper on a preprint server or at a workshop that does not have proceedings, it is considered unpublished for purposes of submission.
In order to verify the status of the paper mentioned above, the program chairs contacted the organizers of conference X. They responded:
We do not consider work submitted to conference X as published work and do thus not publish any proceedings. So the conference X webpage should not be seen as an archival page.
Here is a quote from the call for papers: “The conference does not publish any proceedings, but accepted contributions will be made available on the conference web page as extended abstracts.”
So the paper in question is acceptable for submission to ACL 2017, as conference X declares that it does “not publish any proceedings.” I think the discussion on the PC Chairs Blog has already shown that the current policy is not tenable, and this is another good example. For all practical purposes, the paper has been published, but since conference X states that it “does not publish any proceedings” it’s OK. After all, the papers are only “made available on the conference web page.” Please, what exactly is the difference between a Web page containing links to the full text of all submissions and proceedings? Suppose another conference has a Web page containing links to the full text of all submissions, but puts the title “Proceedings” on top of the page—a paper presented at that conference would not be acceptable for submission to ACL 2017.
In my view, this is not only arbitrary and unfair. It is also highly problematic in other respects, because it is no longer ACL that defines what’s acceptable, but the acceptability of papers is now determined by other conferences’ self-definitions. One can have different opinions about how to best handle preprints, about the best approach to peer review, whether it must be double-blind or not, etc. In any case, however, you must specify clear, unambiguous rules based on objective criteria, which must then be followed without exception. Only this can ultimately guarantee a level playing field for the whole community.
Is this a trend?
At the recent general assembly of DHd, the German-language digital humanities association, it was decided that, while the primary language of the association’s conference is German, submissions can now be made in any language. Yes, any language. The organizers would then have to see whether they would find qualified reviewers, otherwise the paper would be rejected. A lot could be said about (and in particular: against) this decision,3 but what strikes me is the similarity of both policy decisions. Clear rules, such as “submissions must be previously unpublished” or “submissions must be in German,” are being replaced by vague verbiage, often in the name of promoting “openness” and other laudable goals.
The potential side effects, however, are completely ignored. Suppose you submit your paper to DHd 2018 in Swahili. Unfortunately, the chairs fail to find a sufficient number of reviewers (say, three) that speak Swahili, are competent in the subject, and have no conflicts of interest. The paper is eventually rejected without review. Somebody else submits a paper written in, say, Italian. Luckily for the author, suitable reviewers are found and the paper is accepted. If any language is ostensibly acceptable, why is one paper rejected without review due to its language, whereas the other is accepted? Again, the decision is arbitrary and unfair.
Is this part of a larger societal trend? The policies discussed above remind me of the German government’s Commissioner for Integration’s “master plan,” which demands that, in order to promote integration of migrants, society must accept that the rules for living together must be renegotiated on a daily basis. Again, this may sound great at first, but it would effectively mean the end of the rule of law—a very severe side effect. Abandoning explicit rules does not automatically lead to more fairness and openness; to the contrary, it leads to widespread uncertainty, arbitrariness, and unfairness.4 It encourages deceit and punishes those who don’t clamor loud enough for special, i.e., preferential, treatment. I don’t think this is what we should aim for. It’s good to see such issues openly discussed in the CL/NLP community.
- If you’re not working in computational linguistics or natural language processing, the ACL conference is the A* conference in a field where it’s easier to get a paper accepted in the best journal (see conference acceptance rates).
- This post is actually an extended version of a comment I made on the blog.
- See Ulrike Wuttke’s blog post for an extensive discussion in German.
- I’m not saying that there can’t be unfair rules. The point here is explicitness; rules that are explicit can be learned, and they can be challenged, discussed, and—if necessary—changed. Backroom decisions, however well-meaning they may be, are inaccessible to democratic processes.