AI Code Review in GitHub CI Pipeline — Tutorial

Learn how to wire an LLM into GitHub Actions to automatically catch security anti-patterns, style drift, and logic smells on every pull request — with real YAML and prompt templates.

Most code review bottlenecks are not about effort — they are about attention. A reviewer who has already read 400 lines of diff will miss the subtle SQL injection risk on line 401. An AI reviewer never gets tired. Wiring one into your CI pipeline is not a novelty; it is a force multiplier for every engineer on your team.

This tutorial walks through a production-ready approach: a GitHub Actions workflow that posts AI-generated review comments directly on pull requests, using either the OpenAI API or a self-hosted LLM such as Ollama running a Code Llama model.

What You Are Actually Building

The workflow does three things on every pull request:

Extracts the unified diff for changed files
Sends that diff to an LLM with a structured prompt targeting security, style, and logic issues
Posts the model's response as a PR comment via the GitHub API

No third-party SaaS. No proprietary plugins. Just a Python script, a GitHub Actions YAML file, and a prompt template you control.

Setting Up the GitHub Actions Workflow

Create .github/workflows/ai-review.yml in your repository:

name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install openai PyGithub

      - name: Generate diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD \
            -- '*.py' '*.ts' '*.js' '*.go' \
            > diff.patch

      - name: Run AI review
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO_NAME: ${{ github.repository }}
        run: python scripts/ai_review.py

A few design decisions worth noting here. The fetch-depth: 0 is critical — without the full history, git diff cannot resolve the base branch. The file glob filters keep the diff focused; sending auto-generated lock files to an LLM wastes tokens and produces noise.

The Review Script

The Python script at scripts/ai_review.py loads the diff, calls the LLM, and posts results back:

import os
from openai import OpenAI
from github import Github

PROMPT_TEMPLATE = """
You are a senior software engineer performing a security-focused code review.
Analyze the following git diff and identify:

1. SECURITY: Hardcoded secrets, injection risks, insecure defaults, missing auth checks.
2. LOGIC: Off-by-one errors, unhandled edge cases, incorrect conditional branching.
3. STYLE: Inconsistent naming, overly complex functions, missing error handling.

For each issue, state:
- Severity: HIGH / MEDIUM / LOW
- File and approximate line number
- A one-sentence explanation
- A suggested fix

If the diff looks clean, say so briefly. Do not invent issues.

--- DIFF START ---
{diff}
--- DIFF END ---
"""

def main():
    with open("diff.patch") as f:
        diff = f.read()

    if len(diff.strip()) == 0:
        print("Empty diff, skipping review.")
        return

    # Truncate to ~12,000 chars to stay within context limits
    diff = diff[:12000]

    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(diff=diff)}],
        temperature=0.2,
    )

    review_text = response.choices[0].message.content

    gh = Github(os.environ["GITHUB_TOKEN"])
    repo = gh.get_repo(os.environ["REPO_NAME"])
    pr = repo.get_pull(int(os.environ["PR_NUMBER"]))
    pr.create_issue_comment(f"## AI Code Review\n\n{review_text}")

if __name__ == "__main__":
    main()

The temperature: 0.2 setting is intentional. Code review is a precision task, not a creative one. Lower temperature keeps the model grounded in what it actually sees in the diff.

Using a Self-Hosted LLM Instead

If your code is confidential or you want zero data egress, swap the OpenAI client for a call to a local Ollama instance running codellama:13b. In your Actions runner (or a self-hosted runner on your own infrastructure), start Ollama as a service and point the script at http://localhost:11434/api/chat. The prompt structure stays identical; only the client call changes.

Self-hosted runners on a private VPC mean the diff never leaves your network. For teams building fintech or healthtech products — common in the Ghanaian and broader West African SaaS market — this is often a compliance requirement, not just a preference.

Prompt Engineering for Accuracy

The prompt template is where most teams leave performance on the table. A few principles that hold up in practice:

Constrain the output format. Ask for severity labels explicitly. Unstructured prose is hard to scan.
Tell the model what not to do. "Do not invent issues" significantly reduces hallucinated findings.
Focus the scope. A prompt that tries to review security, performance, documentation, and test coverage simultaneously produces mediocre results across all four. Pick two.
Iterate per language. A Python-specific prompt that references PEP 8 and common Django anti-patterns outperforms a generic prompt for Python repositories.

Handling Noise and False Positives

No LLM produces a zero false-positive review. The right mental model is: this is a junior reviewer who has read every OWASP guide but lacks project context. You will want to:

Add a .aireview-ignore file pattern to skip generated or vendored code
Tune the prompt over time based on which findings your team finds useful
Consider posting the review as a collapsible comment or a check annotation rather than a blocking status, at least initially

As the signal-to-noise ratio improves through prompt iteration, you can graduate it to a required check.

Why This Matters for Your Project

Whether you are a two-person startup or a growing engineering team, the pull request queue is where velocity dies and quality debt accumulates. An AI review layer does not replace human judgment — it raises the floor. Trivial issues get caught automatically, freeing reviewers to focus on architecture, product logic, and the things that actually require human context. Integrating this into your CI/CD pipeline is a one-day investment that pays back on every sprint thereafter.