AI code review metrics: a 30-day scorecard

Vendor benchmarks score AI reviewers on other people’s pull requests. The question you actually have is narrower: is the bot on our repo worth the noise it adds? You can answer that in an afternoon, from data GitHub already stores, with a short script and some honest hand-labelling.

This guide gives you the AI code review metrics that hold up, a gh + jq script that pulls them for any 30-day window, a four-label method for the part a script can’t judge, and a worked example on this site’s own repo, where the automatic number turned out to be wrong in an instructive way.

The numbers worth tracking

Most AI review dashboards lead with volume: comments posted, issues “found”. Volume is a cost, not a result. These are the six numbers that tell you whether a reviewer earns its place, split by whether a script can compute them.

Action rate and false discovery rate are the pair that matters. A reviewer can post nothing and never be wrong, or post everything and catch every bug. You want a high share of comments that change code and a low share that are simply mistaken.

Step 1: pull 30 days of merged PRs

The script below asks GitHub’s GraphQL API for every PR merged in a window, with its reviews and inline review threads, then prints the automatic half of the scorecard and writes one CSV row per bot thread for you to label. It needs the GitHub CLI (logged in) and jq, and nothing else.

Download ai-review-scorecard.sh or copy it:

#!/usr/bin/env bash
# ai-review-scorecard.sh — score an AI PR reviewer on your own merged PRs.
# Usage: ./ai-review-scorecard.sh OWNER/REPO BOT_LOGIN 2026-09-01 2026-09-30
# Needs: gh (logged in) and jq. Writes scorecard/prs.json and scorecard/threads.csv.
set -euo pipefail
REPO=$1 BOT=$2 SINCE=$3 UNTIL=$4
OUT=${OUT:-scorecard}
mkdir -p "$OUT"

# 1. Every merged PR in the window, with reviews and inline review threads.
gh api graphql --paginate \
  -f q="repo:$REPO is:pr is:merged merged:$SINCE..$UNTIL" \
  -f query='
query($q: String!, $endCursor: String) {
  search(query: $q, type: ISSUE, first: 25, after: $endCursor) {
    pageInfo { hasNextPage endCursor }
    nodes { ... on PullRequest {
      number createdAt mergedAt author { login }
      reviews(first: 100) { nodes { author { login __typename } submittedAt } }
      reviewThreads(first: 100) { nodes {
        isResolved isOutdated path
        comments(first: 1) { nodes { author { login } url body } }
      } }
    } }
  }
}' | jq -s '[.[].data.search.nodes[]]' > "$OUT/prs.json"

# 2. One row per bot thread, with an empty label column for hand-scoring.
jq -r --arg bot "$BOT" '
  ["pr","state","path","url","excerpt","label"],
  (.[] as $pr | $pr.reviewThreads.nodes[]
   | select(.comments.nodes[0].author.login? == $bot)
   | [$pr.number,
      (if .isOutdated then "changed" elif .isResolved then "resolved_unchanged" else "untouched" end),
      .path, .comments.nodes[0].url,
      (.comments.nodes[0].body | gsub("\\s+"; " ") | .[0:140] | if test("^[=+@-]") then "\u0027" + . else . end), ""])
  | @csv' "$OUT/prs.json" > "$OUT/threads.csv"

# 3. The automatic half of the scorecard.
jq -r --arg bot "$BOT" '
  def hours($a; $b): ((($b | fromdateiso8601) - ($a | fromdateiso8601)) / 3600 * 10 | round) / 10;
  def median: sort | if length == 0 then "n/a" elif length % 2 == 1 then .[length / 2 | floor] else ((.[length / 2 - 1] + .[length / 2]) / 2 * 10 | round) / 10 end;
  def pct($n; $d): if $d == 0 then "n/a" else "\($n * 1000 / $d | round / 10)%" end;
  ([.[] | . as $pr | .reviewThreads.nodes[]
    | select(.comments.nodes[0].author.login? == $bot)]) as $t
  | ($t | map(select(.isOutdated)) | length) as $changed
  | ($t | map(select(.isResolved and (.isOutdated | not))) | length) as $resolved
  | ([.[] | select(any(.reviews.nodes[]; .author.login? == $bot))]) as $covered
  | [ ["Merged PRs", length],
      ["PRs the bot reviewed", "\($covered | length) (\(pct($covered | length; length)))"],
      ["Bot inline threads", ($t | length)],
      ["Comments per reviewed PR", (if ($covered | length) == 0 then "n/a" else (($t | length) * 10 / ($covered | length) | round / 10) end)],
      ["Changed after comment", "\($changed) (\(pct($changed; $t | length)))"],
      ["Resolved without a change", "\($resolved) (\(pct($resolved; $t | length)))"],
      ["Untouched", "\(($t | length) - $changed - $resolved) (\(pct(($t | length) - $changed - $resolved; $t | length)))"],
      ["Median hours to first bot review",
        ([$covered[] | . as $pr | [.reviews.nodes[] | select(.author.login? == $bot) | .submittedAt] | min
          | hours($pr.createdAt; .)] | median)],
      ["Median hours to first human review",
        ([.[] | . as $pr | [.reviews.nodes[] | select(.author.__typename? == "User" and .author.login != $pr.author.login?) | .submittedAt]
          | select(length > 0) | min | hours($pr.createdAt; .)] | median)],
      ["Median hours to merge", ([.[] | hours(.createdAt; .mergedAt)] | median)]
    ][] | "\(.[0]): \(.[1])"' "$OUT/prs.json"

echo "Label a sample in $OUT/threads.csv: real-fixed, real-wontfix, wrong, noise."

Make it executable once, then run it with the repository, the bot’s login, and the window:

chmod +x ai-review-scorecard.sh
./ai-review-scorecard.sh your-org/your-repo coderabbitai 2026-09-01 2026-09-30

Find the bot’s login on any PR it reviewed. GraphQL reports bot logins without the [bot] suffix you see in the UI and REST API, so use coderabbitai, not coderabbitai[bot].

How “changed after comment” works

The script can’t read minds, so it leans on a flag GitHub already sets. A review thread is isOutdated when the lines it was attached to changed in a later commit. That is a decent proxy for “someone acted on this”. Threads resolved without a change are counted separately, as likely dismissals. Everything else is “untouched”.

The proxy has known blind spots, and the worked example below shows how big they can be. A fix made on a different line, or in a different file, leaves the thread untouched. An unrelated edit to the same hunk marks it changed. Treat the automatic action rate as an estimate until your labels confirm it.

Step 2: label a sample by hand

Open scorecard/threads.csv. If there are fewer than 50 rows, label all of them. Otherwise take a random 50, so you don’t end up labelling only the memorable ones:

(head -1 scorecard/threads.csv; tail -n +2 scorecard/threads.csv | awk 'BEGIN{srand()} {print rand() "\t" $0}' | sort -n | cut -f2- | head -50) > sample.csv

Open each comment’s link, read it against the code as it was at merge, and give it exactly one of four labels:

• real-fixed: the finding was correct and the code changed because of it, in that PR or a direct follow-up.
• real-wontfix: correct, but the team made a deliberate call not to change it. Still a useful comment.
• wrong: the claim is false. The file exists, the race can’t happen, the API doesn’t behave that way.
• noise: not false, but not worth anyone’s attention. Generic advice, “verify this before deploy”, or speculation with no concrete failure.

Label quickly, and write one line of evidence in a notes column when you mark something wrong. You will need those notes in step 3.

Step 3: compute and read the scorecard

• Action rate = (real-fixed) ÷ labelled. Compare it with the script’s “changed after comment” share. The gap tells you how far to trust the automatic number on future months.
• Useful rate = (real-fixed + real-wontfix) ÷ labelled.
• False discovery rate = wrong ÷ labelled.
• Noise rate = noise ÷ labelled.
• Cost per acted-on comment = monthly spend ÷ (action rate × bot threads). It is a blunt number, but it is the one a budget conversation will ask for.

Then do the step most teams skip: group the wrong findings by cause. Ten wrong comments with ten different causes is a reviewer that misreads code. Ten wrong comments with one cause is usually a configuration problem you can fix in an afternoon.

Worked example: 17 comments on this site’s repo

We ran the script on the repository behind prcodereview.com: every merged PR from its launch in August to mid-October 2026. The reviewer is Mergestorm’s Vortex. We build Mergestorm, so read this as a demo of the method, not a benchmark: one small repo, one bot, our own code.

Merged PRs: 19
PRs the bot reviewed: 18 (94.7%)
Bot inline threads: 17
Comments per reviewed PR: 0.9
Changed after comment: 7 (41.2%)
Resolved without a change: 0 (0%)
Untouched: 10 (58.8%)
Median hours to first bot review: 0
Median hours to first human review: n/a
Median hours to merge: 0.2

With only 17 threads we labelled all of them, checking each finding against the PR’s files and later commits:

What the labels changed

The automatic action rate was too low, not too high. The script said 7 of 17 comments led to a change (41%). Hand labels found 10 (59%). All 7 threads the script marked “changed” were genuine fixes, so the proxy was precise. It missed three: a privacy-copy fix and a homepage-copy fix landed on lines other than the ones commented on, and a CSS fix answered a file-level comment that had no line to go outdated. On this repo, read “changed after comment” as a floor.

The wrong findings shared one cause. All four were the same mistake: the reviewer didn’t see binary files (images, a favicon) in the diff snapshot and reported them missing. All four were in the repo’s first five merged PRs, and none of the fourteen PRs since had one. A 24% false discovery rate sounds damning. “One blind spot, since gone” is a different conversation, and you only get to it by grouping causes.

The empty row is a finding too. “Median hours to first human review: n/a” means no human ever left a formal review on this repo. It is a small site maintained by a team that ships through agents, so the bot is the review gate. That raises the stakes on its false discovery rate, and it means the speed comparison you would normally make (bot vs human turnaround) doesn’t apply here.

Volume is a cost, not a result. Count the comments that changed code, and group the ones that were wrong by cause.

Keep, tune, or drop

There is no industry threshold for these numbers, and anyone quoting one is selling something. These are the rules of thumb we use. They are starting points, not standards.

• Keep when roughly half or more of the comments are useful, wrong findings stay under about one in five, and they don’t cluster on one cause you are ignoring.
• Tune when wrong findings cluster: exclude generated or binary paths, tell the reviewer what your framework does, or turn off a category that keeps misfiring. When noise is the problem, raise the severity floor or switch off style suggestions. Then re-run the scorecard on the next 30 days.
• Drop or replace when fewer than about one comment in five leads to a change even after a tuning round, or your team has started resolving bot threads without reading them. That habit is worse than having no bot, because it trains people to ignore review.

If you are deciding what to tune, compare your wrong and noise buckets against what AI code review should flag on a pull request. Most noise is a reviewer commenting on things that list says to skip.

What this scorecard can’t tell you

Escaped defects. The most important number is the bugs that shipped anyway, and no API holds it. For a rough check, list reverts and hotfixes merged in the window (gh pr list --state merged --search "revert in:title"), trace each to the PR that caused it, and see whether the bot commented on the line. One escaped bug the bot flagged and a human dismissed teaches more than a month of averages.

The counterfactual. A real-fixed comment might have been caught by a human ten minutes later. If you have human reviewers, sample a few real-fixed findings and ask whether the human review left the same comment.

Who wrote the code. Agent-written PRs and human-written PRs fail differently. If you have both, add an authorship column and score them separately before you average anything.

A 30-day plan you can steal

1. Run the script on the last 30 days and save the output as your baseline.
2. Label up to 50 threads. Note the evidence for every wrong.
3. Compare the hand-labelled action rate with the script’s estimate. Write down the gap.
4. Group wrong findings by cause and fix the biggest cluster in the reviewer’s configuration.
5. Check reverts and hotfixes from the window for escaped defects.
6. Run it again in 30 days. Decide keep, tune, or drop from the trend, not one month.

Before you trust the numbers

• The bot login matches what GraphQL reports (no [bot]).
• The window has fewer than 1,000 merged PRs, or you split it.
• The labelled sample was random, not hand-picked.
• Every “wrong” label has a line of evidence.
• You compared the automatic action rate against your labels.

A reviewer that passes this scorecard has earned the right to be automatic on every PR. For how to run more than one reviewer without paying for the whole fleet on every push, read How to structure an AI PR review loop.