Building an AI Code Reviewer with Claude API and RAG

The Problem

Code reviews are bottlenecks. At our company, senior developers spent 4-6 hours daily reviewing pull requests. We needed an AI system that could:

• Understand our entire codebase context
• Catch bugs before human review
• Enforce coding standards automatically
• Provide intelligent, context-aware suggestions

Solution Architecture

I built CodeSage AI - an enterprise-grade code review assistant using:

1. RAG (Retrieval-Augmented Generation)

# Vector embeddings for semantic code search
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import PGVector

# Embed entire codebase
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
vectorstore = PGVector.from_documents(
    documents=code_chunks,
    embedding=embeddings,
    connection_string=DATABASE_URL
)

2. Claude API for Code Analysis

import anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

# Analyze PR with context
message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": f"""Analyze this pull request:

Context from codebase:
{retrieved_context}

Changes:
{pr_diff}

Check for: bugs, security issues, performance problems, style violations.
"""
    }]
)

3. GitHub Integration

from fastapi import FastAPI, Request
import hmac

app = FastAPI()

@app.post("/webhook/github")
async def handle_pr(request: Request):
    # Verify webhook signature
    payload = await request.json()
    
    if payload["action"] == "opened":
        pr_number = payload["pull_request"]["number"]
        
        # Analyze PR
        analysis = await analyze_pr(pr_number)
        
        # Post comment
        await post_github_comment(pr_number, analysis)

Key Challenges & Solutions

Challenge 1: Token Limits

Problem: Large PRs exceeded Claude’s context window
Solution: Implemented smart chunking strategy

def intelligent_chunking(diff, max_tokens=8000):
    # Prioritize changed functions
    functions = extract_changed_functions(diff)
    
    # Get surrounding context from vector DB
    context = vectorstore.similarity_search(
        query=functions,
        k=5
    )
    
    # Fit within token limit
    return optimize_context(functions, context, max_tokens)

Challenge 2: API Costs

Problem: $500/month in API calls
Solution: Intelligent caching + batching

from functools import lru_cache
import redis

redis_client = redis.Redis()

@lru_cache(maxsize=1000)
def get_embedding(code_snippet):
    # Check Redis cache
    cached = redis_client.get(f"emb:{hash(code_snippet)}")
    if cached:
        return cached
    
    # Generate and cache
    embedding = embeddings.embed_query(code_snippet)
    redis_client.setex(
        f"emb:{hash(code_snippet)}",
        86400,  # 24 hours
        embedding
    )
    return embedding

Result: Reduced costs by 40% ($500 → $300/month)

Challenge 3: False Positives

Problem: AI flagged valid patterns as bugs
Solution: Custom prompt engineering + few-shot learning

SYSTEM_PROMPT = """You are an expert code reviewer.

Guidelines:
- Only flag REAL issues, not stylistic preferences
- Consider project context and patterns
- Provide actionable suggestions with code examples
- Rank issues: Critical > High > Medium > Low

Examples of VALID patterns (do not flag):
{few_shot_examples}
"""

Results

After 3 months in production:

Metric	Result
Bugs Detected	47 critical issues caught
Time Saved	60% reduction (6h → 2.4h/day)
Accuracy	92% (measured vs human reviews)
PRs Analyzed	15,000+
False Positives	<5%
Prevented Incidents	8 major production issues

Technical Implementation

Full Architecture

┌─────────────────┐
│  GitHub Webhook │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   FastAPI App   │◄────┐
└────────┬────────┘     │
         │              │
         ▼              │
┌─────────────────┐     │
│  Vector Search  │     │
│   (pgvector)    │     │
└────────┬────────┘     │
         │              │
         ▼              │
┌─────────────────┐     │
│   Claude API    │     │
│  Code Analysis  │     │
└────────┬────────┘     │
         │              │
         ▼              │
┌─────────────────┐     │
│  Post Comment   │─────┘
│   to GitHub     │
└─────────────────┘

Database Schema

CREATE TABLE code_embeddings (
    id UUID PRIMARY KEY,
    file_path TEXT NOT NULL,
    function_name TEXT,
    code_snippet TEXT,
    embedding vector(1536),  -- pgvector
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX ON code_embeddings 
USING ivfflat (embedding vector_cosine_ops);

Performance Optimization

# Async processing for speed
import asyncio

async def analyze_pr(pr_number: int):
    tasks = [
        fetch_pr_diff(pr_number),
        get_similar_code(pr_number),
        get_coding_standards(),
    ]
    
    diff, context, standards = await asyncio.gather(*tasks)
    
    # Parallel Claude calls for different aspects
    analyses = await asyncio.gather(
        check_bugs(diff, context),
        check_security(diff),
        check_performance(diff),
        check_style(diff, standards)
    )
    
    return merge_analyses(analyses)

Lessons Learned

1. RAG is Essential: Without codebase context, accuracy was 65%. With RAG: 92%
2. Prompt Engineering Matters: Spent 2 weeks optimizing prompts. Reduced false positives from 18% → 5%
3. Caching Saves Money: 40% cost reduction through smart caching
4. Async is Fast: Reduced analysis time from 45s → 8s

What’s Next

Planning to add:

• Multi-language support (currently Python/JavaScript only)
• Automatic fix suggestions (with diffs)
• Learning from accepted/rejected suggestions
• Integration with VS Code extension

Code

Full source code: github.com/vaibhav7k/codesage-ai

Conclusion

Building an AI code reviewer taught me:

• LLMs + RAG = powerful combination
• Context is everything for code analysis
• Prompt engineering is an art and science
• Production AI requires careful cost/performance trade-offs

The system has become indispensable to our team. It catches bugs humans miss, teaches junior developers, and saves hours daily.

Tech Stack: Python, FastAPI, Claude API, LangChain, PostgreSQL (pgvector), Redis, Docker

---

Have questions? Tweet @vaibhav7k or email me