swe-bench-lite
Quick-start command to run SWE-bench Lite evaluation with sensible defaults.
Best use case
swe-bench-lite is best used when you need a repeatable AI agent workflow instead of a one-off prompt.
Quick-start command to run SWE-bench Lite evaluation with sensible defaults.
Teams using swe-bench-lite should expect a more consistent output, faster repeated execution, less prompt rewriting.
When to use this skill
- You want a reusable workflow that can be run more than once with consistent structure.
When not to use this skill
- You only need a quick one-off answer and do not need a reusable workflow.
- You cannot install or maintain the underlying files, dependencies, or repository context.
Installation
Claude Code / Cursor / Codex
Manual Installation
- Download SKILL.md from GitHub
- Place it in
.claude/skills/benchmark-swe-lite/SKILL.mdinside your project - Restart your AI agent — it will auto-discover the skill
How swe-bench-lite Compares
| Feature / Agent | swe-bench-lite | Standard Approach |
|---|---|---|
| Platform Support | Not specified | Limited / Varies |
| Context Awareness | High | Baseline |
| Installation Complexity | Unknown | N/A |
Frequently Asked Questions
What does this skill do?
Quick-start command to run SWE-bench Lite evaluation with sensible defaults.
Where can I find the source code?
You can find the source code on GitHub using the link provided at the top of the page.
SKILL.md Source
# Instructions
This skill provides a streamlined way to run the SWE-bench Lite benchmark with pre-configured defaults.
## What This Skill Does
This skill runs a quick SWE-bench Lite evaluation with:
- 5 sample tasks (configurable)
- Verbose output for visibility
- Results saved to `results.json`
- Report saved to `report.md`
## Prerequisites Check
Before running, verify:
1. **Docker is running:**
```bash
docker ps
```
2. **API key is set:**
```bash
echo $ANTHROPIC_API_KEY
```
3. **Config file exists:**
- Check for `mcpbr.yaml` in the current directory
- If missing, run `mcpbr init` to generate it
## Default Command
The default command for SWE-bench Lite:
```bash
mcpbr run -c mcpbr.yaml --dataset SWE-bench/SWE-bench_Lite -n 5 -v -o results.json -r report.md
```
## Customization Options
Users can customize the run by modifying:
- **Sample size:** Change `-n 5` to any number (or remove for full dataset)
- **Config file:** Change `-c mcpbr.yaml` to point to a different config
- **Verbosity:** Use `-vv` for very verbose output
- **Output files:** Change `results.json` and `report.md` to different paths
## Example Variations
### Minimal quick test (1 task)
```bash
mcpbr run -c mcpbr.yaml -n 1 -v
```
### Full evaluation (all ~300 tasks)
```bash
mcpbr run -c mcpbr.yaml --dataset SWE-bench/SWE-bench_Lite -v -o results.json
```
### MCP-only (skip baseline)
```bash
mcpbr run -c mcpbr.yaml -n 5 -M -v -o results.json
```
### Specific tasks
```bash
mcpbr run -c mcpbr.yaml -t astropy__astropy-12907 -t django__django-11099 -v
```
## Expected Runtime & Cost
For 5 tasks with default settings:
- **Runtime:** 15-30 minutes (depends on task complexity)
- **Cost:** $2-5 (depends on task complexity and model used)
## What to Do If It Fails
1. **Docker not running:** Start Docker Desktop
2. **API key missing:** Set with `export ANTHROPIC_API_KEY="sk-ant-..."`
3. **Config missing:** Run `mcpbr init` to generate default config
4. **Config invalid:** Check that `{workdir}` placeholder is in the `args` array
5. **MCP server fails:** Test the server command independently
## After the Run
Once complete, you'll have:
- **results.json:** Full evaluation data with metrics, token usage, and per-task results
- **report.md:** Human-readable summary with resolution rates and comparisons
- **Console output:** Real-time progress and summary table
Review the results to see how your MCP server performed compared to the baseline!
## Pro Tips
- Start with `-n 1` to verify everything works before running larger evaluations
- Use `--log-dir logs/` to save detailed per-task logs for debugging
- Compare multiple runs by changing the MCP server config between runs
- Use `--baseline-results baseline.json` to detect regressions between versionsRelated Skills
run-benchmark
Run an MCP evaluation using mcpbr on SWE-bench or other datasets.
generate-config
Generate and validate mcpbr configuration files for MCP server benchmarking.
performing-sqlite-database-forensics
Perform forensic analysis of SQLite databases to recover deleted records from freelists and WAL files, decode encoded timestamps, and extract evidence from browser history, messaging apps, and mobile device databases.
performing-kubernetes-cis-benchmark-with-kube-bench
Audit Kubernetes cluster security posture against CIS benchmarks using kube-bench with automated checks for control plane, worker nodes, and RBAC.
performing-docker-bench-security-assessment
Docker Bench for Security is an open-source script that checks dozens of common best practices around deploying Docker containers in production. Based on the CIS Docker Benchmark, it audits host confi
hardening-windows-endpoint-with-cis-benchmark
Hardens Windows endpoints using CIS (Center for Internet Security) Benchmark recommendations to reduce attack surface, enforce security baselines, and meet compliance requirements. Use when deploying new Windows workstations or servers, remediating audit findings, or establishing organization-wide security baselines. Activates for requests involving Windows hardening, CIS benchmarks, GPO security baselines, or endpoint configuration compliance.
hardening-linux-endpoint-with-cis-benchmark
Hardens Linux endpoints using CIS Benchmark recommendations for Ubuntu, RHEL, and CentOS to reduce attack surface, enforce security baselines, and meet compliance requirements. Use when deploying new Linux servers, remediating audit findings, or establishing security baselines for Linux infrastructure. Activates for requests involving Linux hardening, CIS benchmarks for Linux, server security baselines, or Linux configuration compliance.
auditing-cloud-with-cis-benchmarks
This skill details how to conduct cloud security audits using Center for Internet Security benchmarks for AWS, Azure, and GCP. It covers interpreting CIS Foundations Benchmark controls, running automated assessments with tools like Prowler and ScoutSuite, remediating failed controls, and maintaining continuous compliance monitoring against CIS v5 for AWS, v4 for Azure, and v4 for GCP.
sqlite-ops
Patterns for SQLite databases in Python projects - state management, caching, and async operations. Triggers on: sqlite, sqlite3, aiosqlite, local database, database schema, migration, wal mode.
golang-benchmark
Golang benchmarking, profiling, and performance measurement. Use when writing, running, or comparing Go benchmarks, profiling hot paths with pprof, interpreting CPU/memory/trace profiles, analyzing results with benchstat, setting up CI benchmark regression detection, or investigating production performance with Prometheus runtime metrics. Also use when the developer needs deep analysis on a specific performance indicator - this skill provides the measurement methodology, while golang-performance provides the optimization patterns.
phase1-lite
Ultra-compact Phase 1 requirements output. HARD CAP: 500 tokens.
SQLite Memory Access
No description provided.