Teaching an AI reviewer to review a design document, not code

I switched on a shared AI merge-request reviewer across a set of our repositories. On the code repos it behaved roughly as advertised: it read the diff, made some reasonable points, made some obvious ones, and occasionally caught something worth catching.

Then I pointed it at the architecture repository — and it had almost nothing useful to say.

That repository is documentation-as-data. There is no application code in it at all. It holds a single schema-validated YAML file describing a platform’s architecture, which CI renders into a Markdown view, an HTML site, and a PDF for the governance portal. A merge request against it changes prose and structured metadata, not behaviour.

The reviewer’s first pass on that repo asked where the unit tests were.

This is not a failing of the tool. Given a diff and no other information, an AI reviewer infers what kind of review to perform, and the overwhelming prior is “this is code”. If the artefact is not code, you have to say so — and, more usefully, you have to say what a good review of that artefact actually looks like. So this post is about doing exactly that: what I told it, what worked, and the recommended fix that cost me a broken build.

Start from the human review, not from the tool

The instinct is to configure an AI reviewer by thinking about what an AI reviewer can do. That produces a shopping list of generic checks and a lot of noise.

The better question is: when a competent architecture reviewer opens this merge request, what do they actually do? Write that down first, in plain language, and only then translate it into configuration.

For a design document it turned out to split cleanly in two:

• Mechanical gates — objective things that must pass, and where a wrong answer is simply wrong.
• Judgement checks — the things that make the review worth a human’s time: internal contradictions, missing governance records, claims that no longer match reality.

Those two halves want very different instructions, and getting the reviewer to understand which mode it is in for a given finding turned out to matter more than any individual rule.

Half one: the mechanical gates

These duplicate what CI already enforces, which sounds redundant. It is worth doing anyway, because a failed pipeline job tells the author that something is wrong, and the reviewer tells them what and why in a comment attached to the line.

focus_areas:
  - "The document must remain valid against the design schema and pass yamllint"
  - "Preserve the schema declaration, YAML hierarchy, key names, and multiline
     block scalars (|) with their indentation"
  - "Do not drop business identifiers (application, service, capability, or
     governance IDs) unless explicitly requested"
  - "Do not rename section-title values that begin with markdown heading markers
     (e.g. ### 1 Overview) unless asked"

The structural-preservation rules exist because of how this document is authored. It is a schema-bound YAML file, so a well-meaning tidy-up — reflowing a block scalar, renaming a section title, dropping an identifier that looks like clutter — is almost always a mistake, and almost always invisible in review until the render breaks. Telling the reviewer these are load-bearing turns “why is this weird ID here?” into “do not remove this”.

I also encode the repository’s change-management convention, because it is mechanical and frequently forgotten:

  - "The document version field must equal the governance ticket ID in the branch
     name (e.g. feature/DESIGN-1234 -> DESIGN-1234)"
  - "The document date must be the date of the most recent change, in DD/Mon/YYYY"
  - "CHANGELOG.md must contain a single section matching the branch's ticket ID;
     do not hand-edit it — it is generated"

Note the last clause. Half the value of these instructions is stopping the reviewer from recommending a change that will be overwritten by automation.

Half two: the judgement checks

This is where an AI reviewer earns its place on a document repo, because these checks are genuinely tedious for a human and genuinely hard to do well when you are the twelfth reviewer of a hundred-page document.

The document contradicting itself

A long design document accretes contradictions. Section 2 says there are no virtual machines outside the managed platforms; section 5 describes a standalone VM. One section states an RTO of four hours, another eight. An availability-zone count differs between the narrative and a diagram.

  - "Flag statements that contradict each other elsewhere in the document
     (e.g. 'no VMs outside the managed platforms' vs a described standalone VM;
     conflicting RPO/RTO values or availability-zone counts)"
  - "A component, interface, or connectivity path shown in a diagram must be
     reflected in the relevant narrative section, and vice versa"

An LLM reading a whole document at once is genuinely good at this, and a human reviewer reading a diff is genuinely bad at it — because the contradiction is usually between the diff and a section nobody opened.

The body contradicting a recorded decision

This is the check I would keep if I could keep only one.

Architecture decision records exist so that a choice, once made and justified, does not get quietly re-litigated. But nothing enforces the link: the decisions live in one section of the document, and the narrative that must obey them lives in twenty others. A change can perfectly plausibly describe using technology X for a workload where a decision record explicitly chose Y over X — and it will pass every mechanical gate.

  - "Every claim in the body must be consistent with the recorded architectural
     decisions; if the body does X but a decision record chose Y over X, cite both
     the section and the decision record"

Checking this by hand means holding the decision list in your head while reading the change. Checking it mechanically is impossible. It is exactly the shape of problem a language model is suited to, and it is the finding type that has produced the most “oh — good catch” reactions.

Absence: should this change have produced a new record?

A diff shows you what changed. The hardest review question is what should have changed and did not.

  - "When a change introduces something architecturally new (a new technology,
     integration pattern, external dependency, or network path, or a materially
     different way of doing something an existing decision record covers), check
     whether a new decision record is warranted — recommend one, do NOT invent
     its content"
  - "When a change introduces or exposes a risk not already recorded (new external
     dependency, new single point of failure, DR limitation, security or compliance
     exposure), recommend a new risk entry with a candidate description, mitigation,
     residual rating and owner — leave numbering to the author"
  - "Flag existing risks whose target remediation date has passed, or whose
     mitigation this change has invalidated"

That last one is a small gift. Risk registers rot silently; a reviewer that reads the whole register on every merge request will notice a remediation date that slipped past two quarters ago.

Stale questionnaire answers

Most governance documents contain a compliance questionnaire — a set of fixed questions with short answers, often booleans with explanatory notes. These go stale faster than anything else in the document, because they were filled in once and nobody re-reads them.

  - "Sanity-check the production-environment answers against the rest of the
     document — e.g. a change that adds a VM while the servers question still
     answers 'None', compute answers that no longer match the described platform,
     or a boolean answer whose accompanying notes say the opposite"

“A boolean whose notes say the opposite” is a real pattern, and a genuinely embarrassing one to have found in an audit rather than in review.

The instruction that matters most: recommend, never fabricate

An AI that invents a plausible architecture decision record is far worse than an AI that says nothing. It produces something that reads like governance, will be skimmed and approved, and records a justification nobody actually made.

So this is stated repeatedly and in several places:

coding_standards: |
  - Do not fabricate decision-record options, risk numbers, or reference URLs.
    Recommend that the author adds the governance record and links it, rather than
    inventing its content.
  - This is documentation-as-data, not code: quote the relevant key or section
    number in each finding, and do NOT rewrite the document unless asked.
  - When a claim cannot be confirmed from available evidence, mark it "unverified,
    please confirm" rather than asserting it is correct.

That third line generalises well beyond this use case. The default failure mode of an AI reviewer on a factual claim is confident agreement. Giving it an explicit, low-effort way to express uncertainty — a fixed phrase it is told to use — converts a silent false negative into a visible prompt for the author.

Prose style, and the trap that broke the build

The document is written in British English, held to a specific style guide. That is a reasonable thing to ask a language model to check, and it does it well:

  - "Prose is British English: Oxford -ize endings (realize, organize, recognize),
     keeping the fixed -ise set (advertise, comprise, exercise, supervise, surprise)"
  - "Single quotation marks as primary; doubles only for a quote within a quote"
  - "Spaced en dash for parenthetical asides, unspaced en dash for ranges
     (1939-45); never an em dash"
  - "Use the serial comma before the final item in a list of three or more"
  - "Spell out numbers one to ninety-nine in running prose; numerals for 100 and
     above, except measurements, percentages, versions and identifiers"
  - "Flag inconsistent spelling, hyphenation, or capitalisation of the same term
     across the document"

Then it suggested a fix that would have failed the schema validation.

The document contains fields whose values are constrained to a fixed schema enumeration. One of them is a criticality tier whose exact enum string includes a hyphenated range — something of the shape Tier 2: RTO 4-8 hours. Applied naively, the “unspaced en dash for ranges” rule says that hyphen should be an en dash. Make that change and the value no longer matches the enum, and the schema validation job fails.

The fix is an exemption list, and it needs to be explicit rather than implied:

  - "Exempt machine-formatted values from prose style rules: schema URNs, version
     and date fields, section-title markers (### 1.9), code and identifier strings,
     resource names, URLs, and any value constrained to a fixed schema enumeration
     (e.g. a criticality tier containing 'RTO 4-8 hours' — keep the hyphen, it must
     match the enum exactly)"

The general lesson is worth stating plainly, because it applies to every AI style check you will ever configure: when you ask a model to enforce style, you must also tell it where style does not apply. A document is not uniformly prose. It contains identifiers, enums, paths, and formatted values that look like text and are not. The model cannot infer the boundary, and the cost of getting it wrong is asymmetric — a missed style nit is trivial, a “corrected” enum value is a broken pipeline.

Worth noting too: style findings are the highest-volume and lowest-value category, so say so. Mine are explicitly ranked as subordinate to schema, governance and consistency findings, and capped.

Configuration mechanics worth knowing

A handful of small things that made a disproportionate difference:

Ignore your rendered outputs. This repo commits the generated Markdown and PDF alongside the source. Without ignore patterns, the reviewer reviews the same change three times and leaves comments on generated files the author cannot meaningfully fix.

ignore_patterns:
  - "docs/design.md"
  - "docs/design.pdf"
  - "reviews/*.html"
  - "**/.venv/**"

Turn off code examples. Most reviewers default to illustrating findings with code snippets. On a document repo that produces nonsense; you want quoted keys and section numbers.

require_code_examples: false

Cap suggestions per category. Uncapped, style findings drown everything else and reviewers learn to skim the whole comment.

max_suggestions_per_type:
  style: 8
  consistency: 12
  governance: 12
  security: 15

Give per-file-type instructions. The design source, the changelog, and the ordinary markdown docs each want different treatment — the changelog in particular should be left alone because it is generated.

What it is good at, and what it is not

Good at: whole-document consistency, cross-referencing claims against a decision register, noticing absence, spotting stale dates, and applying a style guide uniformly. All the things that are boring, mechanical-feeling, and that human reviewers skip when the document is long.

Not good at: knowing whether the architecture is right. It has no opinion worth having on whether the chosen technology suits the workload, whether the risk appetite is appropriate, or whether the design will survive contact with the operational team. That is the human review, and it always was.

Occasionally wrong in predictable ways. Mine reliably flags passing a secret from one CI variable into another as a hardcoded-credential risk. It is a false positive — the platform masks by value, so the secret stays redacted — but it is a reasonable false positive, and the right response is to resolve it in the thread rather than contort the configuration to suppress it. Configuring away every false positive eventually configures away the true ones too.

It also caught something I would have got wrong: that the Conventional Commits specification puts the breaking-change ! immediately before the colon — feat(scope)!:, not feat!(scope):. A small thing, but exactly the kind of spec detail that a reviewer with the specification in its training data will beat a human on.

The transferable version

If you are pointing an AI reviewer at something that is not code:

1. Tell it what the artefact is. A paragraph of project description does more than any individual rule.
2. Write down what your human reviewers actually do, then encode that — rather than accepting the tool’s defaults.
3. Separate the mechanical gates from the judgement checks, and let the mechanical ones duplicate CI. The explanation is the value, not the pass/fail.
4. Look for absence, not just change. “Should this have produced a governance record?” is the highest-value question and the one no linter can ask.
5. Forbid fabrication explicitly, and give the model a stock phrase for uncertainty.
6. If you ask for style, define where style does not apply — enums, identifiers, and formatted values will otherwise get “corrected” into a broken build.
7. Ignore generated files, cap the noisy categories, and check the config is somewhere the tool will actually find it.

The end state is a little bit different from a typical code review. The goal is not an AI that approves merge requests. The goal is a left-shift to save my time and that of others by capturing some categories of mistakes early. If the AI reviewer is happy with my latest changes, I can at least be reasonably confident that any human reviewer it goes to opens a document which is already schema-valid, internally consistent, correctly stamped and stylistically clean — and can spend their entire attention on whether the architecture is any good.