Catch comments a change left wrong

A commit changes a Java method and leaves its documentation behind; ask, for each comment, whether it still holds, and fail the build when the answer is surely no.

A commit changes getResponsePanel() to return an HttpPanelResponse. The Javadoc above it still says:

/** @return org.parosproxy.paros.view.HttpPanel */

Another commit renames a parameter from w to toolItem. The sentence above that method, “Checks if toolitem is selected”, is as true as it ever was.

Both commits passed their builds and their tests. One of them left a comment that now misleads whoever reads it next. How would your code tell which?

Documentation drifts from code wherever the two live side by side: Javadoc, docstrings, the README next to a function. A linter can check that a @param names a real parameter. Whether a sentence still describes what a method does is a judgment, and that’s the part hunch is for.

​

Try it first

Each comment below sat above a method that a commit changed, and the commit left the comment alone. Is it still true? Guess, then see what hunch said and what a person who checked it by hand decided.

Some of these a rule could catch: a parameter that is gone, a return type that changed. Others need someone to read the sentence and the code. The last one is a comment hunch was sure about and got wrong.

​

1. Collect the changes

The rows come from a research dataset of real commits to open-source Java projects (Panthaplackel et al., AAAI 2021). Each row is one comment and the method it documents, before and after a commit:

Column	What it holds
comment	the summary sentence, one @param line or the @return line
old_code, new_code	the method before and after the commit
gold_checked	yes if the commit left the comment wrong, checked by hand

The known answer is called gold. Here only 300 of the 3,944 rows have it, because the authors checked those by hand; every score on this page is measured on them.

​

2. Let code find the facts

Some of what changed is a plain fact. A parser can say exactly whether a parameter was renamed or a return type changed, so the model shouldn’t have to work it out. rows.py reads the rows and adds that fact as a new column, signature_change:

# simplified from rows.py, which also adds a plain rule for comparison
def rows():
    for r in csv.DictReader(open(HERE / "comments.csv")):
        r["signature_change"] = change(r["old_code"],
                                       r["new_code"])
        yield r

change compares the two method headers. Its output reads like this:

summary | Creates elastic node as single member of a cluster. | parameter settings (Settings) added
summary | Marshal the aggregator values of to a JSONArray that w… | return type JSONArray → byte[]
return  | @return org.parosproxy.paros.view.HttpPanel            | return type HttpPanel → HttpPanelResponse

​

3. Write the question

One file says what to ask about each comment and what the model may read. source: py(rows.py:rows) tells hunch to take its rows from that function rather than from a CSV file:

stale_comment.yml

judgment: stale_comment
model: jev-1.13.0
source: py(rows.py:rows)
key: id
state: [comment, signature_change, old_code, new_code]
clip: {old_code: 6000, new_code: 6000}
questions:
  stale:
    type: noul          # yes or no, with how sure
    instructions: >-
      `comment` is the documentation of a Java method,
      written for `old_code`. A commit changed the method to
      `new_code` and left `comment` as it was.
      `signature_change` lists what the commit changed in
      the method's signature, found by a parser: renamed,
      added or removed parameters, changed types. Is
      `comment` now wrong or out of date about `new_code`:
      does it describe a name, type, parameter, return value
      or behaviour that `new_code` no longer has?
    gold: gold_checked  # the known answer
    act: 0.9            # below this, a person decides

​

4. Run it and test it

hunch run prototype/examples/stale_comments/stale_comment.yml \
  --max-cost 0.25
hunch test prototype/examples/stale_comments/stale_comment.yml

Every answer is kept, so running it again is free. hunch test scores the answers against the 300 hand-checked comments:

PASS accuracy 85.7% (min 82%)
PASS AUROC 0.917 (150 yes / 150 no; 0.5 = coin toss; unaffected by base rate) (min 0.88)

When it is sure a comment is stale, it is nearly always right. When it says a comment is fine, it is less reliable, and a “fine” needs no action anyway. So the check in the next step acts on “stale” only.

​

5. Check a change

check_change.py takes two versions of a Java file. For every method whose code changed while its Javadoc didn’t, it asks the question about each part of the Javadoc. It exits with an error when one part is surely stale. The example files hold the two methods from the top of this page:

uv run python prototype/examples/stale_comments/check_change.py \
  prototype/examples/stale_comments/example/Before.java \
  prototype/examples/stale_comments/example/After.java

STALE  getResponsePanel: @return org.parosproxy.paros.view.HttpPanel   (p=0.96; return type HttpPanel → HttpPanelResponse)
CHECK  isSelected: @param w given widget   (p=0.79; parameter w renamed to toolItem)

The @return fails the check. The @param w line is stale too, but hunch is less sure, so it is listed for a person instead. The summary sentence isn’t mentioned: the rename didn’t make it wrong.

​

The result

The same question, asked three times with more to read each time. Of the 54 comments the last version was sure were stale, one wasn’t.

​

What the model sees decides most of it

The first version of the question saw what a linter sees: the comment and the method as it is now. It could not know that getResponsePanel() used to return an HttpPanel, so it couldn’t know the @return had ever matched. It was like proofreading a reply without the letter it answers.

The second version added the method before the commit, and said the comment had been written for it. hunch diff asked both versions about every row: on the hand-checked ones, 52 answers went from wrong to right and 23 the other way. A comment goes wrong because of a change, and now the change was in view.

That left the @param lines where they were. They are also the one kind of comment a plain rule handles well, since a parser can tell whether the parameter a comment names is gone, renamed or retyped. So the third version passed the model the parser’s facts. On @param lines it fixed 13 and broke none. On summaries and @return lines, nothing moved that 100 rows can tell apart from noise.

What hunch asks is one part of the design; what it is shown is the other. Give the model what code knows for certain, let it judge the rest, and measure which part each one helped.

​

Use it on your data

git show origin/main:src/Foo.java > /tmp/Foo.java
uv run python prototype/examples/stale_comments/check_change.py \
  /tmp/Foo.java src/Foo.java

Run it on each changed file of a pull request to fail the build on a surely stale comment; review a few dozen of your own comments first, and hunch test tells you how often it’s right on your code.

​

How it was measured

The data

The test split of the just-in-time comment inconsistency dataset of Panthaplackel, Li, Gligoric and Mooney, “Deep Just-In-Time Inconsistency Detection Between Comments and Source Code”, AAAI 2021 (repository, MIT licence). Its rows are Java methods and their comments from 357 open-source projects on GitHub, each under its own project’s licence, reproduced unmodified for evaluation (see NOTICE.md in the example folder). There are 3,944 rows of three kinds: the summary sentence (1,066), @param lines (1,038) and @return lines (1,840).

Each row’s dataset label, gold_stale, is a heuristic: “wrong” means the developer edited the comment in the same commit. Developers also reword comments that were fine, and forget ones that weren’t. The authors checked a random sample by hand and found 17–20% of it noisy (11% wrong labels, 3% uncertain, 6% poor examples such as a method commented out). They removed those and kept 300 clean rows, 50 wrong and 50 fine of each kind. Those 300 are the gold here, gold_checked; the heuristic label is kept beside them.

The three questions, in full

The question is the same in all three specs; only what the model sees changes. code_only/stale_comment.yml shows the comment and the method as it is now:

code_only/stale_comment.yml

state: [comment, new_code]
questions:
  stale:
    type: noul
    instructions: >-
      `comment` is the documentation of the Java method
      `new_code`. Is `comment` wrong or out of date about
      `new_code`: does it describe a name, type, parameter,
      return value or behaviour that `new_code` does not
      have?
    gold: gold_checked

before_after/stale_comment.yml adds old_code and says the comment was written for it. Against the first:

stale: 982/3944 rows flip (245 within noise band)
  gold accuracy on the 300 shared rows with gold 72.7% → 82.3%  (✓ 52 fixed, ✗ 23 broken, 0 wrong both times, 907 without gold)
  paired sign test p=0.001 → significant

stale_comment.yml adds signature_change. Against the second:

stale: 361/3944 rows flip (185 within noise band)
  gold accuracy on the 300 shared rows with gold 82.3% → 85.7%  (✓ 19 fixed, ✗ 9 broken, 0 wrong both times, 333 without gold)
  paired sign test p=0.087 → NOT significant: could be noise, get more gold rows

Over all 300 the last gain is not significant. By kind it is one clear effect and two non-effects:

Kind (100 rows each)	Code only	Before and after	With the facts	Paper’s best model
Summary	70	88	89	90.0
@param	74	75	88	92.0
@return	74	84	80	81.3

On @param the facts fixed 13 rows and broke none (p < 0.001). On summaries and @return the moves (3 fixed and 2 broken; 3 and 7) are within noise on 100 rows. The paper found the same about the change itself: every model that saw the change beat every model that didn’t.

Against the paper's models

The paper reports its own models on the same 300. Its best, trained on the dataset’s training split, is right 87.8% of the time, an average over three training runs; a rule comparing the comment’s words with what the commit deleted, 75.7%; a CodeBERT classifier that sees only the comment and the current code, 66.9%.

The spec is at 85.7%. With 300 rows its interval is about four points either way, so the two can’t be told apart here. The spec was not trained on this dataset; the paper’s model was. By kind (table above), the paper’s best model is level with the spec on summaries and @return and ahead on @param, where the paper’s own rule-based baseline is better still, at 94.0. For @param comments, a rule is the right tool.

How sure it is

stale (noul, 3944 rows)
  gold: 300 rows (300 from source, 0 from review)
  PASS accuracy 85.7% (min 82%)
  PASS AUROC 0.917 (150 yes / 150 no; 0.5 = coin toss; unaffected by base rate) (min 0.88)
       dial   act on yes: automated  wrong   │  act on no: automated  wrong
       0.80                  33.0%   5.1%   │                23.0%   8.7%
       0.90                  18.0%   1.9%   │                 9.0%   7.4%  ← act yes  ← act no
       0.95                   9.7%   3.4%   │                 2.0%  16.7%

AUROC measures how well the answers rank stale comments above fine ones, whatever the bar. At act: 0.9, a sure “stale” covers 54 of the 300 rows, and 1 of them is wrong. A sure “fine” covers 9.0% of the rows and 7.4% of those are wrong, so check_change.py fails on a sure “stale”, lists an unsure one for a person, and stays quiet otherwise. The tests’ bars sit below what was measured, to catch a regression.

Against the heuristic label on all 3,944 rows it agrees 77.2% of the time. If it is right 85.7% of the time and 11% of the labels are wrong, agreement around 78% is what to expect. Its most confident disagreements are the label’s: @return true if it's ok above a method the commit changed to return void, and @param context The application's context. above one whose context parameter the commit removed. Both are labelled fine, because nobody edited the comment.

The facts were designed on the rows they are scored on

The parser’s facts and the @param rule were designed after seeing the per-kind results on these same 300 rows, and are measured on them: there is no other hand-checked set. That makes the numbers for the last spec somewhat optimistic. The plain @param rule in rows.py says “stale” when the named parameter is gone or has a new type, and is right on 87 of the 100 @param rows (python rows.py in the example folder prints it).

The demo and its pinned answers

example/Before.java and example/After.java hold two methods from the dataset with their comments as the dataset has them, wrapped in a class. None of them is among the 300 hand-checked rows. Both of the script’s calls overrule the heuristic label, which says the @return was fine (nobody edited it) and the summary “Checks if toolitem is selected” was stale (somebody did). The two cases are pinned in the spec’s examples, so a later edit to the question can’t quietly change either. The demo’s answers were asked afresh: the script cuts a method’s text slightly differently from the dataset, so its inputs are not byte-for-byte the dataset’s rows.

In CI, the two versions are the file at the base of a pull request and at its head (git show origin/main:path/Foo.java).

What did not work

Facts helped only where a rule already works. signature_change fixed @param comments and made no measurable difference to summaries or @return lines. For @param, the paper’s rule-based baseline beats every model, this spec included.

Java only, and the parser is a sketch. rows.py reads method headers with regular expressions: types lose their generics, so a change inside angle brackets is invisible, and a renamed method is not reported. check_change.py finds methods the same way: it skips methods without a body (interfaces, abstract methods) and overloaded names, and expects each Javadoc tag to start a line.

One person checked the gold. The 300 were checked by one of the paper’s authors. The spec is wrong on 43 of them; with a single checker, some of those may be the gold’s.

A whole Javadoc was never measured. check_change.py judges each part of a Javadoc separately (the summary sentence, each @param, the @return), as the rows were measured. The parts of a comment can depend on each other.