Design prompts with explicit criteria to improve precision and reduce false positives

The single highest-leverage move for precision is replacing vague quality language with explicit, categorical criteria. A prompt that says "check that comments are accurate" leaves the model to invent its own definition of accuracy, so it flags stylistic quibbles, stale TODOs, and harmless imprecision. A prompt that says "flag a comment ONLY when the behavior it claims contradicts what the code actually does" names the exact condition, so the model applies a testable rule instead of a mood.

Explicit criteria work because they convert a judgment call into a decision procedure. Instead of asking "is this good enough," the model asks "does this input meet the stated condition," which is far more reproducible across files and across runs.

Concrete, testable predicates such as "contradicts actual code behavior," "writes past the end of the buffer," or "logs a secret" outperform abstract adjectives such as "accurate," "clean," or "safe" every time. The exam's canonical contrast is exactly this pair: flag comments only when claimed behavior contradicts actual code behavior versus check that comments are accurate.

General meta-instructions like "be conservative" or "only report high-confidence findings" do not improve precision. They feel like a knob, but they do not move the decision boundary. The model's self-assessed confidence is poorly calibrated, so the same false positives it produced before are still ones it feels confident about. In practice the phrase either suppresses true and false positives together, or just adds hedging language to the same wrong output.

This is the same calibration weakness that makes self-reported confidence scores an unreliable escalation trigger elsewhere in the exam. The fix is not a better adjective, it is a specific categorical rule. "Report an issue only if it falls into one of these named categories" gives the model something to check; "be careful" does not.

False positive rates are not judged category by category in the real world, they are judged at the product level. If one category (say, comment accuracy or naming) is noisy, developers stop reading the tool's output entirely, and they dismiss even the accurate categories like security and data-loss findings along with the noise. A high false positive category undermines confidence in the categories that are actually working.

That means precision is partly a trust problem, not only a prompt problem. The cost of a bad category is not limited to its own comments; it spills over and poisons the well. This is why the correct operational response to a noisy category is often to remove it from production first, then improve it, rather than leaving it live while you iterate.

Rather than relying on the model to self-filter by confidence, write explicit report and skip lists that scope the surface area. Name the classes of issue worth a comment (bugs, security, data loss) and the classes to ignore (minor style, formatting, naming, patterns that are consistent with the surrounding file).

REPORT (post a finding):
- Bugs: logic errors, off-by-one, null/undefined dereference
- Security: injection, missing authorization, secrets in code
- Data loss: destructive migrations, unbounded deletes

SKIP (never post):
- Naming, formatting, import order, style preferences
- Patterns consistent with the surrounding file

This is more reliable than confidence filtering because it targets the actual source of noise: the model reporting subjective, low-value issues. A skip list explicitly tells the model that a whole class of observations is out of scope, which removes the ambiguity that produced false positives in the first place.

Consistent severity classification is a precision problem too: a review that labels the same pattern CRITICAL in one file and MEDIUM in another is noise. Adjectives alone ("critical," "major," "minor") do not anchor the model. Define explicit severity levels and attach a concrete code example to each one so the model has a reference point.

CRITICAL: e.g. db.exec("DELETE FROM orders") with no WHERE clause
HIGH:     e.g. assignment (=) used where equality (==) was intended in an auth check
MEDIUM:   e.g. missing null check on an optional field used later
LOW:      (do not report unless it also breaks a REPORT category)

Anchoring each level to a worked example turns severity from a vibe into a comparison. The model asks "is this input more like the CRITICAL example or the MEDIUM example," which produces reproducible classification across the whole review.

When a category is generating so many false positives that developers are ignoring the tool, the fastest way to restore trust is to temporarily disable that category entirely, keep the reliable categories running, and improve the disabled category's criteria offline. Re-enable it only once its precision is acceptable on a validation sample.

This is a deliberate trust-recovery play, not an admission of defeat. Leaving a broken category live while you "fix it later" keeps eroding confidence in everything else the tool says. Pulling it out stops the bleeding immediately, and because trust is a product-level property, the accurate categories become useful again the moment the noise stops.

A useful supporting practice is to tag each posted finding with its category (and, for analysis, a detected_pattern field). Tracking which categories developers dismiss most tells you exactly which one to disable and iterate on, rather than guessing.