Difference between revisions of "Evaluating a research paper"
| (5 intermediate revisions by the same user not shown) | |||
| Line 16: | Line 16: | ||
===Context=== | ===Context=== | ||
| − | * Provides adequate context: why is there need for the paper? | + | * Provides adequate context: why is there a need for the paper? |
* Cites relevant prior work | * Cites relevant prior work | ||
* Makes reasonable and explicitly stated assumptions | * Makes reasonable and explicitly stated assumptions | ||
| − | * | + | * Has a thesis statement: the 'message' of the paper |
===Contributions=== | ===Contributions=== | ||
| Line 25: | Line 25: | ||
* Makes a non-trivial contribution | * Makes a non-trivial contribution | ||
** Focus is not too narrow | ** Focus is not too narrow | ||
| − | * Does not overstate contributions | + | * Does not overstate contributions |
| − | * If this is an implementation paper, the work is implementable by others | + | * If this is an implementation paper, the work is implementable by others, at least in principle |
* Explicitly identifies limitations | * Explicitly identifies limitations | ||
| − | === | + | === Uses sound methodology=== |
* Sufficiently evaluates contributions | * Sufficiently evaluates contributions | ||
** The larger the claim, the more the need for careful evaluation | ** The larger the claim, the more the need for careful evaluation | ||
| Line 37: | Line 37: | ||
* Compares results with that from prior work or a well-chosen, non-trivial benchmark | * Compares results with that from prior work or a well-chosen, non-trivial benchmark | ||
* Is mathematically correct | * Is mathematically correct | ||
| − | |||
* Is reasonably complete: does not have major unaddressed issues or gaps | * Is reasonably complete: does not have major unaddressed issues or gaps | ||
* Validates tools used in the work, such as simulators | * Validates tools used in the work, such as simulators | ||
** Does not do 'proof by simulation' | ** Does not do 'proof by simulation' | ||
Latest revision as of 13:32, 16 February 2016
Contents
Evaluating a research paper
A workshop I recently attended led me to consider the question of "What is a good quality paper?" Here is my attempt at creating a checklist to judge paper quality. In creating this list, I would like to acknowledge feedback from the participants of the Dagsthul workshop on "Publication Culture of Computing Research", from the ISS4E research group, and from Timothy Roscoe's excellent paper on "Writing reviews for systems conferences."
Attributes of a good paper
Clarity
- Is grammatically correct
- Explicitly states the research question
- Has good mathematical notation
- Uses standard terminology
- Easy to understand
- Good flow
- Well-chosen examples
- Clear figures with descriptive captions
- Each section of the paper lays out its relation to the rest of the paper
Context
- Provides adequate context: why is there a need for the paper?
- Cites relevant prior work
- Makes reasonable and explicitly stated assumptions
- Has a thesis statement: the 'message' of the paper
Contributions
- Makes a novel contribution over prior work either in the solution approach or the problem domain
- Makes a non-trivial contribution
- Focus is not too narrow
- Does not overstate contributions
- If this is an implementation paper, the work is implementable by others, at least in principle
- Explicitly identifies limitations
Uses sound methodology
- Sufficiently evaluates contributions
- The larger the claim, the more the need for careful evaluation
- Uses an appropriate data set
- Uses appropriate statistical techniques in reporting results
- Has justifiable and well-chosen metrics to evaluate performance
- Compares results with that from prior work or a well-chosen, non-trivial benchmark
- Is mathematically correct
- Is reasonably complete: does not have major unaddressed issues or gaps
- Validates tools used in the work, such as simulators
- Does not do 'proof by simulation'