This content originally appeared on DEV Community and was authored by TOMOAKI ishihara
Introduction
Code reviews take time, and large teams often struggle with PR . PR-Agent (now Qodo Merge) uses AI to automatically review pull requests, generate descriptions, and suggest improvements.
This guide shows you how to set up PR-Agent with OpenRouter, giving you cost-effective access to multiple AI models (GPT-4o, Claude, Gemini, etc.) through a unified API.
Why OpenRouter?
Challenges with Direct OpenAI API
- Single provider dependency (OpenAI models only)
- Complex management of multiple provider APIs
- Difficult to compare pricing across different model
OpenRouter Benefits
- Unified Multi-Model API – Access GPT-4o, Claude, Gemini, Llama, etc. through one interface
- Flexible Model Selection – Choose the right model for each task, optimizing cost-effectiveness
- Early Access to Latest Models – Quick access to new models like GPT-5
- Transparent Pricing – Clear pricing for each model (5.5% fee on credit purchases)
Prerequisites
- GitHub repository (private or public)
- OpenRouter account (free to create)
- GitHub Actions permissions
Step 1: OpenRouter Setup
1.1 Account Creation and API Key
- Visit OpenRouter and create an account
-
Purchase Credits (Important)
- Go to the “Credits” page in your dashboard
- Minimum purchase is $5 (5.5% fee applies to credit purchases)
- Auto-recharge option available for convenience
Free Tier: New accounts receive a small free credit allowance, but additional purchases are needed for production use(FAQ: What free tier options exist?)
- In the dashboard, go to “API Keys” and generate a new key
- Copy and securely store the key
1.2 Available Models
OpenRouter provides access to models like:
openai/gpt-5 # Latest model (2025 release)
openai/gpt-4o # Balanced performance
anthropic/claude-3-opus-20240229 # High-quality analysis
google/gemini-pro # Google's model
meta-llama/llama-2-70b-chat # Open source
Check pricing at OpenRouter Models.
Step 2: GitHub Actions Workflow Setup
2.1 Configure Secrets
In your GitHub repository:
- Go to Settings → Secrets and variables → Actions
- Click “New repository secret”
- Add the following secret:
Name: OPENROUTER_API_KEY
Value: or-xxxxxxxx (your OpenRouter API key)
2.2 Create Workflow File
Create .github/workflows/pr_agent.yml
name: Code Review(OpenRouter)
on:
pull_request:
pull_request_review_comment:
types: [created]
# Concurrency control
concurrency:
group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}-${{ github.event_name == 'pull_request_review_comment' && 'pr_comment' || 'pr' }}
cancel-in-progress: ${{ github.event_name != 'pull_request_review_comment' }}
jobs:
pr_agent_job:
if: |
github.event.sender.type != 'Bot' &&
github.event_name == 'pull_request' &&
github.event.pull_request.changed_files < 50
runs-on: ubuntu-latest
permissions:
issues: write
pull-requests: write
contents: write
id-token: write
name: Run pr agent on every pull request
steps:
- name: PR Agent action step with OpenRouter
id: pragent
uses: qodo-ai/pr-agent@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENROUTER.KEY: ${{ secrets.OPENROUTER_API_KEY }} # OpenRouter API認証
github_action_config.auto_review: "true" # enable\disable auto review
github_action_config.auto_describe: "true" # enable\disable auto describe
github_action_config.auto_improve: "true" # enable\disable auto improve
Step 3: Configuration File
Create .pr_agent.toml
in your repository root.
[config]
model = "openai/gpt-5" # I specified gpt-5, but an error occurred. The correct setting is "openai/gpt-5".
fallback_models = ["openai/gpt-4o", "anthropic/claude-opus-4.1"]
ai_timeout = 300
custom_model_max_tokens = 200000 # Critical! Errors without this
response_language = "en-US"
ignore_pr_title = ["^\\[Auto\\]", "^Auto"]
ignore_pr_labels = ['invalid']
[ignore]
glob = ['dist/**']
[github_action_config]
# set as env var in .github/workflows/pr-agent.yaml
#auto_review = true
#auto_improve = true
#auto_describe = true
[pr_reviewer]
extra_instructions = """\
(1) Act as a highly experienced software engineer
(2) Provide thorough review of code, documents, and articles
(3) Suggest concrete code snippets for improvement
(4) **Never** comment on indentation, whitespace, blank lines, or other purely stylistic issues unless they change program semantics.
(5) **Priority Review Areas - Check systematically:**
- **Security**: Plaintext passwords, SQL injection, input validation, exception leaks
- **Error Handling**: Bare except clauses, missing try-catch, silent error suppression
- **Resource Management**: Missing context managers (with statements), unclosed connections/files
- **Type Safety**: Missing type hints, incorrect type usage, unjustified Any types
- **Performance**: Inefficient algorithms (O(n²) or worse), unnecessary loops, memory leaks
- **Code Quality**: Magic numbers, unclear variable names, unused imports/variables
- **API Design**: Missing input validation, no error responses, required field checks
- **Architecture**: Single responsibility violations, tight coupling, global state usage
(6) Focus on concrete, actionable issues with specific code examples and fix recommendations.
"""
num_code_suggestions = 5
inline_code_comments = true
ask_and_reflect = true
[pr_description]
extra_instructions = "Generate clear and comprehensive descriptions"
generate_ai_title = true
add_original_user_description = false
publish_description_as_comment = true
[pr_code_suggestions]
extra_instructions = "Provide actionable code suggestions with examples"
commitable_code_suggestions = true
demand_code_suggestions_self_review=true
Step 4: Testing the Setup
4.1 Create Test PR
Create a PR with intentionally problematic code.
# Missing type hints
def fizz_buzz(n):
for i in range(1, n):
if i % 15 == 0:
print("FizzBuzz")
elif i % 3 == 0:
print("Fizz")
elif i % 5 == 0:
print("Buzz")
else:
printo(i) # Intentional bug
# Security issue
def add_user(name, password):
users = []
user = {"name": name, "password": password} # Plaintext password
users.append(user)
4.2 Expected Output
PR-Agent will provide.
Auto-generated PR Description:
- Summary of changes
- File-by-file change details
- Impact analysis
Code Review Comments:
- Fix for
printo(i)
→print(i)
- Type hint suggestions
- Security warning about plaintext passwords
- Performance improvement suggestions
Step 5: Advanced Configuration
5.1 Cost Optimization
Use different models for different tasks.
[config]
model = "openai/gpt-4o-mini" # For lightweight tasks
model_turbo = "openai/gpt-5" # For complex analysis
fallback_models = ["anthropic/claude-3.7-sonnet", "google/gemini-2.5-pro"]
5.2 Team-Specific Rules
[pr_reviewer]
extra_instructions = """\
(1) Company-specific checks:
- Use structured logging for all log outputs
- External API calls must have timeout settings
- Database access must use transactions
- Never hardcode secrets (API keys, passwords, etc.)
"""
5.3 File Exclusions
[ignore]
glob = ['dist/**', '*.min.js', 'node_modules/**', '__pycache__/**']
[config]
ignore_pr_title = ["^\\[Auto\\]", "^WIP:", "^Draft:"]
ignore_pr_labels = ['work-in-progress', 'do-not-review']
Troubleshooting
Common Issues and Solutions
1. Model not recognized
Error: Model openai/gpt-5 is not defined in MAX_TOKENS
Solution: Add custom_model_max_tokens = 200000 to config
This error occurs when using a model that PR-Agent doesn’t recognize. The models supported by PR-Agent are listed in this file. If your chosen model isn’t in this list, you need to set the custom_model_max_tokens configuration.
2. Inconsistent language responses
[config]
response_language = "en-US" # Be specific
[pr_reviewer]
extra_instructions = "Always respond in English and explain technical terms clearly"
3. GitHub Actions permission errors
permissions:
issues: write
pull-requests: write
contents: write # Add this line
id-token: write
4. Timeouts on large PRs
[config]
ai_timeout = 600 # Extend to 10 minutes
large_patch_policy = "clip" # Split large PRs
Cost Management Best Practices
1. Appropriate Model Selection
Task | Recommended Model | Pricing (approx.) |
---|---|---|
Simple reviews | gpt-4o-mini | $0.15/1M tokens |
Detailed analysis | gpt-4o | $2.5/1M tokens |
Highest quality | openai/gpt-5 | $1.25/1M tokens |
2. PR Size Limits
if: github.event.pull_request.changed_files < 50 # Limit file count
3. Control Automatic Execution
[config]
ignore_pr_labels = ['draft', 'wip', 'skip-ai-review']
ignore_pr_title = ["^\\[WIP\\]", "^Draft:", "^Auto"]
Impact Measurement
Before/After Comparison Metrics
Quantitative Metrics:
- Reduced code review time
- Increased bug detection
- Earlier security issue identification
- Faster PR merge times
Qualitative Metrics:
- Reduced reviewer burden
- Improved code quality
- Learning effect (developer skill improvement)
Real-World Results Example
Before: Average review time 2 hours/PR
After: Average review time 45 minutes/PR (62% reduction)
Security issue detection:
- Before: 2 issues/month average
- After: 8 issues/month average (4x increase)
Conclusion
Setting up PR-Agent with OpenRouter provides these benefits.
Cost Performance – Use multiple models through OpenRouter for cost optimization
Quality Improvement – Consistent review standards reduce oversights
Efficiency – Significant reduction in review time
Learning Effect – Developers learn best practices from AI feedback
Flexibility – Customizable for team-specific requirements
We recommend starting with a small project for testing and gradually expanding. The configuration file allows extensive customization to match your team’s needs.
References
Hope this guide helps improve your code review process! Feel free to share questions or suggestions in the comments.
This content originally appeared on DEV Community and was authored by TOMOAKI ishihara