An AI-generated patch can pass tests and still change an API response, touch a file unrelated to the task, or weaken an assertion. Akshat Raj’s proposal is to require every change to come with an explicit report: which checks passed, which failed, and what remains unverified.
We’ll build that gate with Python and Git. It won’t prove that the program is correct; it will make acceptance depend on reproducible evidence, not on the agent’s confidence in its own work. The gate will compare a base revision with the current commit, limit the modified files, run a test command, and write a JSON report.
flowchart TD
A["Parche en HEAD"] --> B["Comparar con base"]
B --> C["Revisar alcance y diff"]
C --> D["Ejecutar pruebas"]
D --> E["Guardar informe JSON"]
E --> F["Aceptar o rechazar"]Prepare the repository and the gate
The example is intended for Python 3.10 or later and a Git repository. It doesn’t require a model API or external packages. The comparison is made between the specified revision and HEAD; therefore, before running the gate, you must commit the patch you want to review. Uncommitted changes are excluded from that comparison.
Save the following program as proofgate.py at the repository root. The test command is passed as arguments to the program and is run without going through a shell interpreter.
#!/usr/bin/env python3
"""Comprobaciones básicas para un parche Git y un comando de pruebas."""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import time
from pathlib import Path
def run_command(command: list[str], timeout: int) -> dict:
"""Ejecuta un proceso sin shell y conserva una salida acotada."""
started = time.monotonic()
try:
result = subprocess.run(
command,
capture_output=True,
text=True,
timeout=timeout,
check=False,
shell=False,
)
return {
"command": command,
"exit_code": result.returncode,
"stdout": result.stdout[-12000:],
"stderr": result.stderr[-12000:],
"timed_out": False,
"duration_seconds": round(time.monotonic() - started, 3),
}
except subprocess.TimeoutExpired as exc:
stdout = exc.stdout or ""
stderr = exc.stderr or ""
if isinstance(stdout, bytes):
stdout = stdout.decode(errors="replace")
if isinstance(stderr, bytes):
stderr = stderr.decode(errors="replace")
return {
"command": command,
"exit_code": None,
"stdout": stdout[-12000:],
"stderr": stderr[-12000:],
"timed_out": True,
"duration_seconds": round(time.monotonic() - started, 3),
}
def git(*arguments: str) -> dict:
return run_command(["git", *arguments], timeout=30)
def main() -> int:
parser = argparse.ArgumentParser(description="Verifica un parche Git.")
parser.add_argument("--base", required=True, help="Revisión base de Git.")
parser.add_argument("--max-files", type=int, default=20)
parser.add_argument("--allow-prefix", action="append", default=[])
parser.add_argument("--test-timeout", type=int, default=300)
parser.add_argument(
"--test", nargs=argparse.REMAINDER,
help="Comando de pruebas y argumentos; debe ir al final.",
)
args = parser.parse_args()
if args.max_files < 1 or args.test_timeout < 1:
parser.error("--max-files y --test-timeout deben ser positivos")
test_command = args.test or []
if test_command and test_command[0] == "--":
test_command = test_command[1:]
report = {
"schema_version": 1,
"base": args.base,
"checks": [],
"changed_files": [],
"status": "failed",
}
def record(name: str, passed: bool, details: dict) -> None:
report["checks"].append(
{"name": name, "passed": passed, "details": details}
)
base_result = git("rev-parse", "--verify", args.base)
record("base_revision", base_result["exit_code"] == 0, base_result)
if base_result["exit_code"] != 0:
report["reason"] = "No se pudo resolver la revisión base."
print(json.dumps(report, indent=2, ensure_ascii=False))
return 1
diff = git("diff", "--name-only", "-z", f"{args.base}...HEAD")
if diff["exit_code"] != 0:
record("changed_files", False, diff)
print(json.dumps(report, indent=2, ensure_ascii=False))
return 1
changed_files = [name for name in diff["stdout"].split("\0") if name]
report["changed_files"] = changed_files
record("nonempty_patch", bool(changed_files), {"count": len(changed_files)})
record(
"file_count_limit",
len(changed_files) <= args.max_files,
{"actual": len(changed_files), "maximum": args.max_files},
)
if args.allow_prefix:
prefixes = [prefix.strip("/") for prefix in args.allow_prefix]
violations = [
name for name in changed_files
if not any(name == prefix or name.startswith(prefix + "/")
for prefix in prefixes)
]
record("allowed_paths", not violations, {"violations": violations})
diff_check = git("diff", "--check", f"{args.base}...HEAD")
record("git_diff_check", diff_check["exit_code"] == 0, diff_check)
if not test_command:
record("tests", False, {"error": "No se proporcionó --test."})
else:
result = run_command(test_command, timeout=args.test_timeout)
record(
"tests",
result["exit_code"] == 0 and not result["timed_out"],
result,
)
passed = all(check["passed"] for check in report["checks"])
report["status"] = "passed" if passed else "failed"
print(json.dumps(report, indent=2, ensure_ascii=False))
return 0 if passed else 1
if __name__ == "__main__":
sys.exit(main())The argument list and shell=False prevent the test command from being interpreted as a shell expression. That doesn’t make the process safe: the executable and the code it tests still run with the environment’s permissions. Output is capped so that a noisy process can’t inflate the report indefinitely.
Run the checks
First, commit the patch on a branch and make sure the base revision exists locally. From the repository root, a run for a Python project might look like this:
python proofgate.py \
--base origin/main \
--allow-prefix src \
--allow-prefix tests \
--max-files 20 \
--test-timeout 300 \
--test python -m pytest -qThe allowed scope in this example is the src and tests directories; adjust these prefixes to match the project’s actual structure. For another language, change the command after --test, for example to go test ./.... The process returns exit code 0 only if all checks pass; it returns 1 if a check fails or no test command is provided.
The comparison uses base...HEAD, which represents the branch’s changes relative to its merge base with the base. If the specified revision is unavailable, the gate fails before running the tests. The report distinguishes checks by name instead of reducing everything to an opaque score.
To review the behavior without relying on a particular run, this small consumer checks that the JSON report has an approved status and that every check is marked as passed. Save the code as check_report.py; pass it the JSON file generated by redirecting the gate’s output.
#!/usr/bin/env python3
"""Valida la estructura básica de un informe de proofgate."""
import json
import sys
from pathlib import Path
def main() -> int:
if len(sys.argv) != 2:
print("Uso: python check_report.py informe.json", file=sys.stderr)
return 2
try:
report = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as error:
print(f"No se pudo leer el informe: {error}", file=sys.stderr)
return 1
checks = report.get("checks")
valid = (
report.get("schema_version") == 1
and isinstance(checks, list)
and bool(checks)
and report.get("status") == "passed"
and all(check.get("passed") is True for check in checks)
)
print("Informe aprobado" if valid else "Informe rechazado")
return 0 if valid else 1
if __name__ == "__main__":
raise SystemExit(main())The consumer provides a simple check of the report’s structure and status; it doesn’t replace test validation or certify that the report comes from a trustworthy run. To save and check a run in sequence:
python proofgate.py \
--base origin/main \
--allow-prefix src \
--allow-prefix tests \
--test python -m pytest -q > verification.json
python check_report.py verification.jsonMake it a CI rule
A gate is more consistent when it runs on every proposed change. This GitHub Actions workflow is a template: adapt the paths and dependency installation to the repository. The full history lets Git compare the branch with the base.
name: Verificación del parche
on:
pull_request:
push:
branches:
- main
permissions:
contents: read
jobs:
verify:
runs-on: ubuntu-latest
steps:
- name: Obtener el repositorio
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Configurar Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Instalar dependencias de pruebas
run: python -m pip install pytest
- name: Verificar el parche
run: >-
python proofgate.py
--base origin/main
--allow-prefix src
--allow-prefix tests
--max-files 20
--test python -m pytest -qThe CI check should block integration when a required check fails, not turn the failure into a warning. In projects with supply-chain policies, third-party actions should also be pinned according to those policies; this example uses version tags and doesn’t prescribe a pinning strategy.
What it covers and what it doesn’t
Git detects certain formatting problems with diff --check; the file limit and prefix list enforce scope rules; the runner records the exit code and timeout. None of these results proves that the tests are adequate or that behavior meets every requirement. OWASP SAMM recommends verifying important rules independently of the instructions given to the assistant, preferably through deterministic checks such as static analysis, tests, or automated policies.
The verification itself can also be manipulated: for example, a test could be modified so it no longer checks the expected behavior. Protect test files, CI workflows, and the script with appropriate review, and add regression tests that express the contract from a perspective independent of the implementation. The proposed gate doesn’t detect every way a test can be weakened on its own.
The time limit isn’t a sandbox either. Running generated code can allow access to files, credentials, or the network, depending on the environment’s permissions. Don’t provide production secrets to jobs that run untrusted changes; use an isolated environment and restrict permissions and resources.
OpenAI distinguishes between inspecting execution traces to debug an agent’s behavior and using datasets and evaluation runs when repeatability is needed. In this gate, the JSON report serves a more limited purpose: it records the checks configured for a specific patch. A next step is to save the exact revisions, test command, and tool versions alongside the report.
The result is deliberately modest: a proposal passes because the defined checks ran and left evidence, not because the agent claimed it was done. Before integrating, the team is left with a concrete question: do the configured checks cover the behavior this patch was supposed to preserve?
Sources: DEV Community, OpenAI Developers, OWASP SAMM, arXiv
