The share of a reproduction paper is based on analyses of plus in insights into current methods and problems—plus the additional certainty that is sold with validating past outcomes.

  1. Relevance: Is it paper highly relevant to COLING?
  2. Readability/clarity: could be the paper well-written and well-structured?
  3. Data/code access: could be the data/code (as appropriate) open to the study community or perhaps is here a compelling reason provided why this isn’t feasible?
  4. Analysis: In the event that paper surely could replicate the outcomes of the previous work, does it plainly set down exactly exactly what must be filled in to do therefore? If it wasn’t in a position to reproduce the outcome of previous work, does it obviously recognize just what information ended up being missing/the most likely factors?
  5. Generalizability: Does the paper rise above replicating the outcomes regarding the initial to explore if they could be reproduced an additional environment? Instead, in instances of non-replicability, does the paper talk about the wider implications of the result?
  6. Informativeness: To what level does the analysis reported into the paper deepen our knowledge of the methodology used or even the nagging problem approached? Will the information within the paper assistance professionals using their selection of technique/resource?
  7. Significant contrast: along with determining the experimental results being replicated, does the paper motivate why these specific email address details are a target that is important reproduction and exactly exactly what the near future implications are of these having been reproduced or been discovered to be non-reproducible?
  8. General suggestion: there are lots of good submissions contending for slots at COLING 2018; essential could it be to feature that one? Will people discover a complete great deal by scanning this paper or seeing it presented? Please be decisive—it is way better to vary from other reviewers than to grade every thing at the center.

Site paper

Documents in this track provide a language resource that is new. This may be a corpus, but in addition might be an annotation standard, device, an such like.

  1. Relevance: Is it paper highly relevant to COLING? Will the resource presented likely be of good use to the community?
  2. Readability/clarity: Through the real method the paper is created, could you inform the way the resource ended up being produced, the way the quality of annotations (if any) ended up being assessed, and exactly why the resource ought to be of great interest?
  3. Originality: Does the resource fill a need into the collection that is existing of resources? Remember that originality could possibly be when you look at the range of language/language variety or genre, when you look at the design of this annotation scheme, into the scale regarding the resource, or nevertheless other parameters.
  4. Site quality: what type of quality control ended up being performed? If appropriate, had been inter-annotator contract calculated, and in case therefore, with appropriate metrics? Otherwise, how many other assessment ended up being carried out, and exactly how acceptable were the outcome?
  5. Site accessibility: could it be simple for researchers to down load or otherwise access the resource to be able to make use of it in their own personal work? As to the level could work predicated on this resource be provided? answers to incorporate: Yes, i’ve confirmed
  6. Metadata: perform some authors explain whoever language use is captured when you look at the resource also to exactly just what populations results that are experimental in the resource could be generalized to? In instance of annotated resources, will be the demographics regarding the annotators also characterized?
  7. Meaningful contrast: could be the brand new resource situated psychology research paper example with regards to current operate in the industry, including similar resources it took motivation from or improves on? Will it be clear what exactly is unique about the resource?
  8. General suggestion: there are numerous good submissions competing for slots at COLING 2018; essential will it be to feature this 1? Will people learn a complete great deal by scanning this paper or seeing it presented? Please be decisive—it is way better to vary from other reviewers rather than grade every thing at the center.

Position paper

A situation paper presents a challenge to mainstream thinking or perhaps a futuristic vision that is new. It might open a brand new area or unique technology, propose changes in existing research, or provide a brand new pair of ground guidelines.

  1. Relevance: Is it paper highly relevant to COLING?
  2. Readability/clarity: Is it clear just just just what the positioning is the fact that the paper is arguing for? Will be the arguments because of it laid down in an understandable method?
  3. Soundness: Are the arguments presented when you look at the paper coherent and relevant? May be the eyesight well-defined, with success requirements? (Note: it ought to be feasible to provide a higher rating right here even though you don’t buy into the place taken by the writers)
  4. Imagination: How bold or novel may be the position drawn in the paper? Does it express well-thought through and imaginative brand new ground?
  5. Range: How much range for brand brand brand new scientific studies are exposed by this paper? Exactly just just What effect could it have on current areas and concerns?
  6. Significant contrast: may be the paper well-situated with regards to past work, both place documents (taking exactly the same or opposing side on a single or comparable dilemmas) and appropriate theoretical or experimental work?
  7. Substance: Does the paper have sufficient substance for the full-length paper? Could be the problem sufficiently crucial? Would be the arguments adequately thoughtful and diverse?
  8. General suggestion: there are numerous submissions that are good for slots at COLING 2018; essential can it be to feature that one? Please be decisive—it is way better to vary from other reviewers rather than grade every thing at the center.
  9. A study paper provides an organized summary of the literary works up to now on a topic that is specific assists the reader understand the kinds of concerns being inquired about this issue, the different approaches which were used, the way they relate with one another, and just what further research areas they start. A conference-length study paper should sufficiently be about a concentrated subject so it can do that effectively with when you look at the web web page limits.

    1. Relevance: could be the paper highly relevant to COLING?
    2. Readability/clarity: may be the paper generally speaking an easy task to follow and well organized?
    3. Organization: Does the paper organize the appropriate literary works in a narrative and determine typical strands of inquiry?
    4. Scope: Does the paper determine a fairly concentrated area to review?
    5. Thoroughness: because of the region identified to survey, does the paper cover every one of the relevant literary works? Could be the literary works evaluated represented accurately?
    6. Outlook: Does the paper recognize areas for future work and/or plainly explain exactly what isn’t yet managed inside the literary works surveyed?
    7. Context: Does the paper situate current research appropriately within its historic context? (We don’t expect papers in the first place Pa?ini, yet in the exact same time one thing that just cites work from 2017 most likely doesn’t capture just just how current work pertains to the larger image.)