Rules

For a change, this post is actually related to NLP. It’s about rules, but not the linguistic kind, but those that govern our research communities.

Not all proceedings are proceedings

For ACL 2017, the Annual Meeting of the Association for Computational Linguistics,1 the program chairs have created an accompanying blog, the PC Chairs Blog, in order to increase transparency and communication. This is a great initiative to base the further development the field’s premier conference on a broader basis.2

One heavily debated topic is the handling of papers already published as preprints, in particular on arXiv.org; see the posts Statistics on Submissions and Status Update, arXiv and the Future of Double-Blind Conference Reviewing, and arXiv and Double-Blind Reviewing Revisited. The current policy is that such papers are permitted, if the fact is declared; the current policy is also that papers that are not anonymized or that have been published as preprints, but where the authors fail to declare this, are rejected without review.

As area chair for the Multidisciplinary Track, I had the opportunity to experience the problems with this policy at first hand. One of the papers submitted to our track had already been presented—in identical form—at conference X and was available online on the conference Web site. Now, the ACL 2017 call says:

The ACL 2017 conference invites the submission of long and short papers on substantial, original, and unpublished research in all aspects of automated language processing.

However, this is followed by a “Multiple Submission Policy,” which states:

Preprint servers such as arXiv.org and workshops that do not have published proceedings are not considered archival for purposes of submission.

I find it striking that in this section we’re suddenly no longer talking about “publishing” but about “archival.” I would think that the primary goal of preprint servers and workshops is public dissemination, not archiving. I have never seen ACL conferences advertising “archival” either. What is more, I have not seen any explanation of what “archival for purposes of submission” is supposed to mean, because the requirements mention “substantial, original, and unpublished,” but not “unarchived.” I have the suspicion that “archival” here really means “published.” So what the Multiple Submission Policy really says is that if you publish your paper on a preprint server or at a workshop that does not have proceedings, it is considered unpublished for purposes of submission.

In order to verify the status of the paper mentioned above, the program chairs contacted the organizers of conference X. They responded:

We do not consider work submitted to conference X as published work and do thus not publish any proceedings. So the conference X webpage should not be seen as an archival page.

Here is a quote from the call for papers: “The conference does not publish any proceedings, but accepted contributions will be made available on the conference web page as extended abstracts.”

So the paper in question is acceptable for submission to ACL 2017, as conference X declares that it does “not publish any proceedings.” I think the discussion on the PC Chairs Blog has already shown that the current policy is not tenable, and this is another good example. For all practical purposes, the paper has been published, but since conference X states that it “does not publish any proceedings” it’s OK. After all, the papers are only “made available on the conference web page.” Please, what exactly is the difference between a Web page containing links to the full text of all submissions and proceedings? Suppose another conference has a Web page containing links to the full text of all submissions, but puts the title “Proceedings” on top of the page—a paper presented at that conference would not be acceptable for submission to ACL 2017.

In my view, this is not only arbitrary and unfair. It is also highly problematic in other respects, because it is no longer ACL that defines what’s acceptable, but the acceptability of papers is now determined by other conferences’ self-definitions. One can have different opinions about how to best handle preprints, about the best approach to peer review, whether it must be double-blind or not, etc. In any case, however, you must specify clear, unambiguous rules based on objective criteria, which must then be followed without exception. Only this can ultimately guarantee a level playing field for the whole community.

Is this a trend?

At the recent general assembly of DHd, the German-language digital humanities association, it was decided that, while the primary language of the association’s conference is German, submissions can now be made in any language. Yes, any language. The organizers would then have to see whether they would find qualified reviewers, otherwise the paper would be rejected. A lot could be said about (and in particular: against) this decision,3 but what strikes me is the similarity of both policy decisions. Clear rules, such as “submissions must be previously unpublished” or “submissions must be in German,” are being replaced by vague verbiage, often in the name of promoting “openness” and other laudable goals.

The potential side effects, however, are completely ignored. Suppose you submit your paper to DHd 2018 in Swahili. Unfortunately, the chairs fail to find a sufficient number of reviewers (say, three) that speak Swahili, are competent in the subject, and have no conflicts of interest. The paper is eventually rejected without review. Somebody else submits a paper written in, say, Italian. Luckily for the author, suitable reviewers are found and the paper is accepted. If any language is ostensibly acceptable, why is one paper rejected without review due to its language, whereas the other is accepted? Again, the decision is arbitrary and unfair.

Is this part of a larger societal trend? The policies discussed above remind me of the German government’s Commissioner for Integration’s “master plan,” which demands that, in order to promote integration of migrants, society must accept that the rules for living together must be renegotiated on a daily basis. Again, this may sound great at first, but it would effectively mean the end of the rule of law—a very severe side effect. Abandoning explicit rules does not automatically lead to more fairness and openness; to the contrary, it leads to widespread uncertainty, arbitrariness, and unfairness.4 It encourages deceit and punishes those who don’t clamor loud enough for special, i.e., preferential, treatment. I don’t think this is what we should aim for. It’s good to see such issues openly discussed in the CL/NLP community.

  1. If you’re not working in computational linguistics or natural language processing, the ACL conference is the A* conference in a field where it’s easier to get a paper accepted in the best journal (see conference acceptance rates). []
  2. This post is actually an extended version of a comment I made on the blog. []
  3. See Ulrike Wuttke’s blog post for an extensive discussion in German. []
  4. I’m not saying that there can’t be unfair rules. The point here is explicitness; rules that are explicit can be learned, and they can be challenged, discussed, and—if necessary—changed. Backroom decisions, however well-meaning they may be, are inaccessible to democratic processes. []

2 thoughts on “Rules

  1. Hi,

    I think there could be several differences between archival and non-archival release of papers.

    (1) Copyright. Most publishers hold copyright, some of which are exclusive. Even they don’t, a permission of archiving is typically more rigorous than just posting a non-archiving paper (which could be called whatever you like, e.g., a technical report).

    But yes, this is not core of the problem. Copyright is not fair to researchers, who apply grants, work hard, pay the conference/publication fees, but have to transfer the copyright at no charge (and even have to pay when reading peer researcher’s work).

    (2) Credit. Researchers *earn significantly different credit* when “posting” (either publishing or preprinting) archival and non-archival papers. Undergraduates need papers to get admitted, PhD students need papers to get graduated, and faculty need papers to get promoted. If he/she publishes papers in a well-established conference/journal, he will earn much credit for the work. But if he/she posts papers at non-archival workshops/conferences, or preprint servers, yes, he/she will get known for the work, but won’t earn much credit for his/her career development.

    Because of this, dual submission to archival venues is unethical. Otherwise, one can earn multiple credits for a study (except that, in computer science, journals publish extended conference papers with explicit permission in CFP).

    I cannot imagine other differences between “archival” and “non-archival” papers. (1) There’s no such a gallery in the world that collects all and only “archival” papers.
    (2) An archival paper does not necessarily mean higher quality: There are a large number of low-quality, low-impact, low-visibility archival journals/conferences. On the contrary, an arXiv paper is generally more interesting (maybe less rigorous than its final archival version).

    I personally don’t arXiv 3-low archival papers, which are mainly for pragmatic purposes (e.g., some undergraduates want a paper, but they have no hope of publishing top-tier papers and I might be a coauthor).

    The trend is the machine learning community is becoming more and more open.
    NIPS-16 creates a gallery of rejected papers (of the authors’ own accord); ICLR is open-review and open-access for even submissions. So there’s no real difference between acceptance and rejection in terms of public availability. An acceptance, is more about a tag assigning credits to the author, although it brings further visibility and influence. So is the difference between archival and non-archival papers.

  2. Thanks for your comments! I think we agree here. I maybe should have made it more explicit that I do *not* think that we need to stop academic publishing from developing further. The problem I see is that we have a system that used to be somehow consistent, but the environment has drastically changed, and now we’re starting to patch it here and there, but that won’t really fix the system but only introduces inconsistencies. In fact, I’m open to radical changes if they help to give us a better system.

    So, I’m not necessarily opposed to accepting non-archival papers, but then we need to consistently talk about archival/non-archival and make clear what we mean. The current ACL policy doesn’t do this.

Leave a Reply

Your email address will not be published. Required fields are marked *