The Aviator team began integrating four times as many pull requests with the same nine engineers, according to its account in The New Stack. The median time to integration fell from 2.1 days to 3.4 hours, and reverts dropped by 40%. The change wasn’t handing review over to an AI: a person still approved every PR. What changed was what they examined.
The proposal is to review intent, acceptance criteria, invariants, decisions, and tests, instead of trying to read every line of the diff. We’ll implement a small, executable version in Python: a report that checks that every criterion has evidence, that the specified paths exist, and that pending decisions remain visible. It doesn’t decide whether the change is correct; it prepares a more focused human review.
The problem: a diff doesn’t explain the purpose
In the case described by Aviator, weekly PRs went from around 100 to more than 400, and the median AI-generated PR reached 640 lines. The company also tried an automated reviewer, a four-hour deadline, and a self-review skill. The bot found one real issue among many minor comments; the deadline sped up initial reviews but emptied out approvals; and self-review still meant looking at the diff after writing the code.
The workflow we’ll build collects criteria and decisions in a file, checks each criterion against local evidence, and generates a readable result. In the example, the evidence will be a path to a file and a line citation. In a real project, that proof could also be an API result, a screenshot, or an execution log: Aviator mentions these forms of evidence in its description of the process.
flowchart TD
A["Criterios y decisiones"] --> B["Verificador local"]
C["Evidencias del cambio"] --> B
D["Invariantes del equipo"] --> B
B --> E["Informe para la persona"]
E --> F["Resolver decisiones pendientes"]Prepare criteria and evidence
We’ll use only Python’s standard library. Save the following file as review.py and run it with python review.py. The program creates a demo case in a temporary directory, so it doesn’t depend on preexisting files. Each criterion has a description and a concrete test; a decision records its status and the person who resolved it.
The example deliberately keeps the data alongside the checker so it can run immediately. In a repository, it’s a good idea to separate the tool’s configuration and store the criteria in an artifact associated with the PR.
from pathlib import Path
from tempfile import TemporaryDirectory
criteria = [
{
"id": "AC-1",
"description": "La función de cálculo está presente",
"evidence": "src/pricing.py",
"line": 1,
},
{
"id": "AC-2",
"description": "La prueba de cálculo está presente",
"evidence": "tests/test_pricing.py",
"line": 1,
},
]
def main():
with TemporaryDirectory() as temp_dir:
root = Path(temp_dir)
(root / "src").mkdir()
(root / "tests").mkdir()
(root / "src/pricing.py").write_text("def total(price, tax):\n return price + tax\n")
(root / "tests/test_pricing.py").write_text("def test_total():\n assert 2 + 1 == 3\n")
for item in criteria:
path = root / item["evidence"]
print(f"{item['id']}: {'OK' if path.is_file() else 'FALTA'} — {item['description']}")
if __name__ == "__main__":
main()Check each criterion against its evidence
The existence of a file doesn’t prove that the criterion is satisfied. The next step also checks that the cited line exists. Replace the previous content with this complete program: it retains the sample data, but now produces a structured result. The status only means that the specified evidence is available; it isn’t equivalent to semantic approval.
This distinction matters. In Aviator’s experiment with a settings function, for example, verification involved saving a setting and checking the audit log, then saving without changes to confirm that no additional entry appeared. The second check provides different evidence: testing the favorable case isn’t enough.
from pathlib import Path
from tempfile import TemporaryDirectory
criteria = [
{"id": "AC-1", "description": "Función presente", "evidence": "src/pricing.py", "line": 1},
{"id": "AC-2", "description": "Prueba presente", "evidence": "tests/test_pricing.py", "line": 1},
{"id": "AC-3", "description": "Ruta inexistente", "evidence": "docs/pricing.md", "line": 1},
]
def check(root, item):
path = root / item["evidence"]
if not path.is_file():
return {"id": item["id"], "status": "MISSING_FILE", "evidence": item["evidence"]}
lines = path.read_text(encoding="utf-8").splitlines()
line = item["line"]
if line < 1 or line > len(lines):
return {"id": item["id"], "status": "MISSING_LINE", "evidence": item["evidence"]}
return {"id": item["id"], "status": "EVIDENCE_PRESENT", "evidence": item["evidence"], "line": line}
def main():
with TemporaryDirectory() as temp_dir:
root = Path(temp_dir)
(root / "src").mkdir()
(root / "tests").mkdir()
(root / "src/pricing.py").write_text("def total(price, tax):\n return price + tax\n", encoding="utf-8")
(root / "tests/test_pricing.py").write_text("def test_total():\n assert 2 + 1 == 3\n", encoding="utf-8")
for item in criteria:
print(check(root, item))
if __name__ == "__main__":
main()Make invariants and decisions visible
Rules repeated in reviews can become invariants. The New Stack reports that Aviator analyzed 11,400 comments, produced 212 candidate rules, and approved 64 invariants. Some ran as deterministic rules; others still required additional checks. The following example adds a simple invariant: every criterion must include evidence. It also adds unresolved decisions to the report, rather than hiding them among comments.
Save the complete code as review.py. For this teaching example, the invariant is evaluated against the data, not the project code. In a repository, style rules or patterns can be applied with tools suited to the language; a design decision, on the other hand, should remain visible to a person.
from pathlib import Path
from tempfile import TemporaryDirectory
criteria = [
{"id": "AC-1", "description": "Función presente", "evidence": "src/pricing.py", "line": 1},
{"id": "AC-2", "description": "Prueba presente", "evidence": "tests/test_pricing.py", "line": 1},
{"id": "AC-3", "description": "Ruta inexistente", "evidence": "docs/pricing.md", "line": 1},
]
decisions = [
{"question": "¿Debe redondearse el total?", "status": "pending", "owner": None},
{"question": "¿Se mantiene la firma actual?", "status": "settled", "owner": "revisor"},
]
def check(root, item):
if not item.get("evidence"):
return {"id": item["id"], "status": "NO_EVIDENCE"}
path = root / item["evidence"]
if not path.is_file():
return {"id": item["id"], "status": "MISSING_FILE", "evidence": item["evidence"]}
lines = path.read_text(encoding="utf-8").splitlines()
line = item["line"]
if line < 1 or line > len(lines):
return {"id": item["id"], "status": "MISSING_LINE", "evidence": item["evidence"]}
return {"id": item["id"], "status": "EVIDENCE_PRESENT", "evidence": item["evidence"], "line": line}
def main():
with TemporaryDirectory() as temp_dir:
root = Path(temp_dir)
(root / "src").mkdir()
(root / "tests").mkdir()
(root / "src/pricing.py").write_text("def total(price, tax):\n return price + tax\n", encoding="utf-8")
(root / "tests/test_pricing.py").write_text("def test_total():\n assert 2 + 1 == 3\n", encoding="utf-8")
print("Criterios:")
for item in criteria:
print(check(root, item))
print("Decisiones pendientes:")
for decision in decisions:
if decision["status"] == "pending":
print(f"- {decision['question']}")
if __name__ == "__main__":
main()Turn the result into a review gate
The final adjustment makes the program exit with an error code if evidence is missing or a decision is pending. That way, it can be run as a local check or later integrated into CI. The reviewer still has to check whether the test addresses the criterion and resolve product or design decisions.
In the experience Aviator described, reviewers examined what had already been resolved, handled what agents couldn’t decide, and reopened anything they would judge differently. Automation organizes the work; it doesn’t replace that judgment. The team itself put the transition cost at duplicated work during the first month, before the invariants began reducing the workload in the second.
from pathlib import Path
from tempfile import TemporaryDirectory
criteria = [
{"id": "AC-1", "description": "Función presente", "evidence": "src/pricing.py", "line": 1},
{"id": "AC-2", "description": "Prueba presente", "evidence": "tests/test_pricing.py", "line": 1},
]
decisions = [
{"question": "¿Debe redondearse el total?", "status": "pending", "owner": None},
]
def check(root, item):
if not item.get("evidence"):
return False, f"{item['id']}: falta la ruta de evidencia"
path = root / item["evidence"]
if not path.is_file():
return False, f"{item['id']}: no existe {item['evidence']}"
lines = path.read_text(encoding="utf-8").splitlines()
line = item["line"]
if line < 1 or line > len(lines):
return False, f"{item['id']}: no existe la línea {line}"
return True, f"{item['id']}: evidencia disponible en {item['evidence']}:{line}"
def main():
with TemporaryDirectory() as temp_dir:
root = Path(temp_dir)
(root / "src").mkdir()
(root / "tests").mkdir()
(root / "src/pricing.py").write_text("def total(price, tax):\n return price + tax\n", encoding="utf-8")
(root / "tests/test_pricing.py").write_text("def test_total():\n assert 2 + 1 == 3\n", encoding="utf-8")
failed = False
for item in criteria:
ok, message = check(root, item)
print(message)
failed |= not ok
for decision in decisions:
if decision["status"] != "settled":
print(f"PENDIENTE: {decision['question']}")
failed = True
raise SystemExit(1 if failed else 0)
if __name__ == "__main__":
main()With the sample decision unresolved, execution ends with an error; change its status to settled and assign an owner for the gate to pass. That change doesn’t certify the code: it makes clear what’s missing before someone can approve it with context.
Sources: The New Stack, Aviator, Aviator Blog, Aviator Blog
