getResponsePanel() to return an HttpPanelResponse. The Javadoc above it still says @return org.parosproxy.paros.view.HttpPanel. The build passes, the tests pass, the review looks at the code. The next person to read that comment believes it.
A linter can catch part of this. Java’s own documentation checker reports a @param that names no parameter. It cannot tell whether “Creates elastic node as single member of a cluster” is still what a method does after a commit rewrote it. That is a judgment about meaning, and it is the kind of rule this cookbook turns into code: one spec, measured on real commits, and a script that runs it on a change.
The rows
Panthaplackel et al. (AAAI 2021) (MIT) collected commits from open-source Java projects on GitHub. Each row is a method’s comment, the method before the commit, and the method after it. The test split has 3,944 rows from hundreds of projects, for three kinds of comment: the summary sentence (1,066),@param lines (1,038) and @return lines (1,840).
Each row’s label says whether the commit left the comment wrong, and it is a heuristic: “wrong” means the developer edited the comment in the same commit. Developers also reword comments that were fine, and forget ones that weren’t. The authors checked a random sample by hand, found 17–20% of it noisy (11% wrong labels, 3% uncertain, 6% poor examples such as a method commented out), removed those, and kept 300 clean rows: 50 wrong and 50 fine of each kind. Those 300 are the gold here (gold_checked), and the heuristic label (gold_stale) is kept beside them.
The paper also reports its own models on the same 300. Its best, trained on the dataset’s training split, is right 87.8% of the time; a rule comparing the comment’s words with what the commit deleted, 75.7%; a CodeBERT classifier that sees only the comment and the current code, 66.9%.
What the question needs to see
The question is the same in every spec below: is the comment now wrong about the method? What changes is what the model sees.code_only/stale_comment.yml shows it the comment and the method as it is now, which is all a linter has:
code_only/stale_comment.yml
before_after/stale_comment.yml adds the method before the commit, and says the comment was written for it. hunch diff compares the two on every row:
Code finds the facts, the model judges
By kind, the before-and-after spec was right on 88 summaries, 84@return lines and 75 @param lines of each 100. The @param lines are the odd ones out, and they are also where a plain rule works: a @param names a parameter, and a parser can tell exactly whether that parameter is gone, renamed or retyped. A rule that says “stale” when the named parameter is gone or has a new type is right on 87 of the 100.
So the parser’s facts go to the model. rows.py is the spec’s source: it reads the rows and adds signature_change, what the commit did to the parameters and the return type:
stale_comment.yml
On
@param the facts fixed 13 rows and broke none (p < 0.001). On summaries and @return the moves (3 fixed and 2 broken; 3 and 7) are within noise on 100 rows. The paper’s best model is level with the spec on summaries and @return and ahead on @param, where the paper’s own rule-based baseline is better still, at 94.0: for @param comments, a rule is the right tool.
Across the 300 the spec is at 85.7%, and the paper’s model at 87.8%. With 300 rows the spec’s interval is about four points either way, so the two can’t be told apart here. The spec was not trained on this dataset; the paper’s model was, and its figure is an average over three training runs.
The parser’s facts and the
@param rule were designed after seeing the per-kind results on these same 300 rows, and are measured on them: there is no other hand-checked set. That makes the numbers for the last spec somewhat optimistic.How sure it is
@return true if it's ok above a method the commit changed to return void, and @param context The application's context. above one whose context parameter the commit removed. Both are labelled fine, because nobody edited the comment.
Run it on a change
check_change.py takes two versions of a Java file. For every method whose code changed while its Javadoc did not, it judges each part of the Javadoc (the summary sentence, each @param, the @return), the same units the spec was measured on, and exits 1 when one is surely stale. example/Before.java and example/After.java hold two methods from the dataset with their comments as the dataset has them, wrapped in a class:
@return fails the check. @param w given widget is stale too, since w is now toolItem, but at 0.79 it is below the bar, so it is listed for a person rather than failing the build. The summary “Checks if toolitem is selected” was not flagged: the rename didn’t make it wrong. Both of those calls overrule the heuristic label, which says the @return was fine (nobody edited it) and the summary was stale (somebody did). The two cases are pinned in the spec’s examples, so a later edit to the question can’t quietly change either.
In CI, the two versions are the file at the base of a pull request and at its head (git show origin/main:path/Foo.java).
What did not work
Facts helped only where a rule already works.signature_change fixed @param comments and made no measurable difference to summaries or @return lines. For @param, the paper’s rule-based baseline beats every model, this spec included.
Java only, and the parser is a sketch. rows.py reads method headers with regular expressions: types lose their generics, so a change inside angle brackets is invisible, and a renamed method is not reported. check_change.py finds methods the same way: it skips methods without a body (interfaces, abstract methods) and overloaded names, and expects each Javadoc tag to start a line.
One person checked the gold. The 300 were checked by one of the paper’s authors. The spec is wrong on 43 of them; with a single checker, some of those may be the gold’s.
A whole Javadoc was never measured. check_change.py judges each part of a Javadoc separately, as the rows were measured. The parts of a comment can depend on each other. The demo’s answers were asked afresh: the script cuts a method’s text slightly differently from the dataset, so its inputs are not byte-for-byte the dataset’s rows.