cicd-pipeline

# CI/CD Pipeline (GitHub Actions)

Set up and manage CI/CD pipelines using GitHub Actions. Covers workflow creation, testing, deployment, release automation, and debugging.

## When to Use

- Setting up automated testing on push/PR
- Creating deployment pipelines (staging, production)
- Automating releases with changelogs and tags
- Debugging failing CI workflows
- Setting up matrix builds for cross-platform testing
- Managing secrets and environment variables in CI
- Optimizing CI with caching and parallelism

## Quick Start: Add CI to a Project

### Node.js project

```yaml
# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm test
      - run: npm run lint
```

### Python project

```yaml
# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest
      - run: ruff check .
```

### Go project

```yaml
# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.22"
      - run: go test ./...
      - run: go vet ./...
```

### Rust project

```yaml
# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo test
      - run: cargo clippy -- -D warnings
```

## Common Patterns

### Matrix builds (test across versions/OSes)

```yaml
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node-version: [18, 20, 22]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test
```

### Conditional jobs

```yaml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test

  deploy:
    needs: test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh
```

### Caching dependencies

```yaml
# Node.js (automatic with setup-node)
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm  # or yarn, pnpm

# Generic caching
- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      ~/.cargo/registry
      node_modules
    key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-deps-
```

### Artifacts (save build outputs)

```yaml
- uses: actions/upload-artifact@v4
  with:
    name: build-output
    path: dist/
    retention-days: 7

# Download in another job
- uses: actions/download-artifact@v4
  with:
    name: build-output
    path: dist/
```

### Run on schedule (cron)

```yaml
on:
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6 AM UTC
  workflow_dispatch:  # Also allow manual trigger
```

## Deployment Workflows

### Deploy to production on tag

```yaml
name: Release

on:
  push:
    tags:
      - "v*"

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run build
      - run: npm test

      # Create GitHub release
      - uses: softprops/action-gh-release@v2
        with:
          generate_release_notes: true
          files: |
            dist/*.js
            dist/*.css
```

### Deploy to multiple environments

```yaml
name: Deploy

on:
  push:
    branches: [main, staging]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: |
          if [ "${{ github.ref }}" = "refs/heads/main" ]; then
            ./deploy.sh production
          else
            ./deploy.sh staging
          fi
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
```

### Docker build and push

```yaml
name: Docker

on:
  push:
    branches: [main]
    tags: ["v*"]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
```

### npm publish on release

```yaml
name: Publish

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          registry-url: https://registry.npmjs.org
      - run: npm ci
      - run: npm test
      - run: npm publish --provenance
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
```

## Secrets Management

### Set secrets via CLI

```bash
# Set a repository secret
gh secret set DEPLOY_TOKEN --body "my-secret-value"

# Set from a file
gh secret set SSH_KEY < ~/.ssh/deploy_key

# Set for a specific environment
gh secret set DB_PASSWORD --env production --body "p@ssw0rd"

# List secrets
gh secret list

# Delete a secret
gh secret delete OLD_SECRET
```

### Use secrets in workflows

```yaml
env:
  # Available to all steps in this job
  DATABASE_URL: ${{ secrets.DATABASE_URL }}

steps:
  - run: echo "Deploying..."
    env:
      # Available to this step only
      API_KEY: ${{ secrets.API_KEY }}
```

### Environment protection rules

Set up via GitHub UI or API:
- Required reviewers before deployment
- Wait timers
- Branch restrictions
- Custom deployment branch policies

```bash
# View environments
gh api repos/{owner}/{repo}/environments | jq '.environments[].name'
```

## Workflow Debugging

### Re-run failed jobs

```bash
# List recent workflow runs
gh run list --limit 10

# View a specific run
gh run view <run-id>

# View failed job logs
gh run view <run-id> --log-failed

# Re-run failed jobs only
gh run rerun <run-id> --failed

# Re-run entire workflow
gh run rerun <run-id>
```

### Debug with SSH (using tmate)

```yaml
# Add this step before the failing step
- uses: mxschmitt/action-tmate@v3
  if: failure()
  with:
    limit-access-to-actor: true
```

### Common failures and fixes

**"Permission denied" on scripts**
```yaml
- run: chmod +x ./scripts/deploy.sh && ./scripts/deploy.sh
```

**"Node modules not found"**
```yaml
# Make sure npm ci runs before npm test
- run: npm ci     # Install exact lockfile versions
- run: npm test   # Now node_modules exists
```

**"Resource not accessible by integration"**
```yaml
# Add permissions block
permissions:
  contents: write
  packages: write
  pull-requests: write
```

**Cache not restoring**
```yaml
# Check cache key matches - use hashFiles for lockfile
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
# NOT: key: ${{ runner.os }}-node-${{ hashFiles('package.json') }}
```

**Workflow not triggering**
- Check: is the workflow file on the default branch?
- Check: does the trigger event match? (`push` vs `pull_request`)
- Check: is the branch filter correct?
```bash
# Manually trigger a workflow
gh workflow run ci.yml --ref main
```

## Workflow Validation

### Validate locally before pushing

```bash
# Check YAML syntax
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" && echo "Valid"

# Use actionlint (if installed)
actionlint .github/workflows/ci.yml

# Or via Docker
docker run --rm -v "$(pwd):/repo" -w /repo rhysd/actionlint:latest
```

### View workflow as graph

```bash
# List all workflows
gh workflow list

# View workflow definition
gh workflow view ci.yml

# Watch a running workflow
gh run watch
```

## Advanced Patterns

### Reusable workflows

```yaml
# .github/workflows/reusable-test.yml
name: Reusable Test
on:
  workflow_call:
    inputs:
      node-version:
        required: false
        type: string
        default: "20"
    secrets:
      npm-token:
        required: false

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
      - run: npm ci
      - run: npm test
```

```yaml
# .github/workflows/ci.yml - caller
name: CI
on: [push, pull_request]
jobs:
  test:
    uses: ./.github/workflows/reusable-test.yml
    with:
      node-version: "20"
```

### Concurrency (prevent duplicate runs)

```yaml
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true  # Cancel previous runs for same branch
```

### Path filters (only run for relevant changes)

```yaml
on:
  push:
    paths:
      - "src/**"
      - "package.json"
      - "package-lock.json"
      - ".github/workflows/ci.yml"
    paths-ignore:
      - "docs/**"
      - "*.md"
```

### Monorepo: only test changed packages

```yaml
jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'packages/api/**'
            web:
              - 'packages/web/**'

  test-api:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/api && npm ci && npm test

  test-web:
    needs: changes
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/web && npm ci && npm test
```

## Tips

- Use `workflow_dispatch` on every workflow for manual triggering during debugging
- Pin action versions to SHA for supply chain security: `uses: actions/checkout@b4ffde...`
- Use `continue-on-error: true` for non-critical steps (like linting)
- Set `timeout-minutes` on jobs to prevent runaway builds (default is 360 minutes)
- Use job outputs to pass data between jobs: `outputs: result: ${{ steps.step-id.outputs.value }}`
- For self-hosted runners: `runs-on: self-hosted` with labels for targeting specific machines