optimizing-sql-queries

# SQL Query Optimizer

## Overview

Rewrite SQL queries for maximum performance by eliminating anti-patterns, restructuring JOINs, leveraging window functions, and applying database-specific optimizations for PostgreSQL and MySQL. This skill takes a slow query and its execution plan as input and produces an optimized version with measurable improvement, along with any supporting index changes needed.

## Prerequisites

- The slow SQL query text and its current execution time
- `EXPLAIN ANALYZE` output (PostgreSQL) or `EXPLAIN FORMAT=JSON` output (MySQL) for the query
- Table row counts and approximate data distribution for involved tables
- `psql` or `mysql` CLI for testing rewrites
- Knowledge of the application's acceptable result ordering and NULL handling requirements

## Instructions

1. Examine the original query structure and identify common anti-patterns:
   - `SELECT *` instead of specific columns (forces unnecessary I/O)
   - `WHERE column IN (SELECT ...)` that can be rewritten as `JOIN` or `EXISTS`
   - `DISTINCT` used to mask duplicate rows from incorrect JOINs
   - Functions applied to indexed columns in WHERE clauses (`WHERE UPPER(name) = 'FOO'`)
   - `OR` conditions that prevent index usage
   - `NOT IN` with nullable columns (produces wrong results and poor plans)

2. Analyze the execution plan to identify the most expensive operation nodes. Focus optimization effort on the node consuming the most time or processing the most rows.

3. Rewrite subqueries as JOINs where possible. Convert correlated subqueries to lateral joins (PostgreSQL) or derived tables. Replace `IN (SELECT ...)` with `EXISTS (SELECT 1 ...)` for existence checks since EXISTS short-circuits after the first match.

4. Optimize JOIN ordering for the query planner: place the most selective table (fewest matching rows after WHERE filters) as the driving table. Use `JOIN` hints only as a last resort since the optimizer usually picks the correct order with accurate statistics.

5. Replace multiple OR conditions on the same column with `IN (...)`: change `WHERE status = 'active' OR status = 'pending'` to `WHERE status IN ('active', 'pending')`. For OR across different columns, consider UNION ALL of two simpler queries.

6. Apply window functions to replace self-joins or correlated subqueries. Use `ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...)` for top-N-per-group queries instead of `GROUP BY` with subqueries.

7. Leverage CTEs (Common Table Expressions) for readability but be aware that PostgreSQL versions before 12 materialize all CTEs. For performance-critical queries on older PostgreSQL, inline the CTE as a subquery.

8. Optimize aggregation queries by filtering before grouping (`WHERE` is more efficient than `HAVING` for non-aggregate conditions), using partial indexes for filtered aggregates, and considering materialized views for expensive recurring aggregations.

9. Test the rewritten query with `EXPLAIN ANALYZE` and compare execution time, row estimates vs. actuals, and buffer usage against the original. The optimized version should show fewer rows processed, index scans replacing sequential scans, and lower total execution time.

10. Document each change made, the reason for the change, and the measured impact so the development team understands and can apply similar patterns to future queries.

## Output

- **Optimized SQL query** with comments explaining each structural change
- **Before/after execution plans** showing performance improvement
- **Index recommendations** (CREATE INDEX statements) needed to support the optimized query
- **Anti-pattern report** listing issues found in the original query with explanations
- **Performance metrics comparison** (execution time, rows scanned, buffer hits)

## Error Handling

| Error | Cause | Solution |
|-------|-------|---------|
| Rewritten query returns different results | JOIN type change (INNER vs LEFT) or NULL handling difference | Verify result sets match with `EXCEPT` query; preserve original JOIN types; handle NULLs explicitly with `COALESCE` |
| Optimized query slower than original | Statistics outdated causing planner to choose wrong plan | Run `ANALYZE` on involved tables; compare `estimated rows` vs `actual rows` in EXPLAIN; consider `SET enable_seqscan = off` to test alternative plans |
| CTE materialization hurting performance | PostgreSQL <12 materializes CTEs preventing predicate pushdown | Inline the CTE as a subquery; upgrade PostgreSQL; add `AS NOT MATERIALIZED` hint in PostgreSQL 12+ |
| Window function query uses excessive memory | Large partition sizes with ORDER BY in window specification | Add `LIMIT` to outer query; use index matching the PARTITION BY and ORDER BY columns; increase `work_mem` for the session |
| UNION ALL produces duplicates | Overlapping conditions in constituent queries | Add mutually exclusive WHERE conditions to each branch; or use UNION (with dedup cost) if overlap is unavoidable |

## Examples

**Converting correlated subquery to JOIN**: Original: `SELECT * FROM orders WHERE customer_id IN (SELECT id FROM customers WHERE region = 'US')` taking 8 seconds with sequential scan on orders. Rewrite: `SELECT o.* FROM orders o JOIN customers c ON o.customer_id = c.id WHERE c.region = 'US'` using index on `orders.customer_id` reduces to 120ms.

**Top-N per group with window function**: Original uses self-join to find the 3 most recent orders per customer (15 seconds). Rewrite: `SELECT * FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY created_at DESC) AS rn FROM orders) sub WHERE rn <= 3` with index on `(customer_id, created_at DESC)` completes in 400ms.

**Eliminating DISTINCT from incorrect JOIN**: `SELECT DISTINCT o.* FROM orders o JOIN line_items li ON o.id = li.order_id WHERE li.amount > 100` scans all line items. Rewrite: `SELECT o.* FROM orders o WHERE EXISTS (SELECT 1 FROM line_items li WHERE li.order_id = o.id AND li.amount > 100)` eliminates the deduplication step and halves execution time.

## Resources

- PostgreSQL query planning: https://www.postgresql.org/docs/current/planner-optimizer.html
- MySQL query optimization: https://dev.mysql.com/doc/refman/8.0/en/optimization.html
- SQL anti-patterns reference: https://use-the-index-luke.com/sql/where-clause
- Window functions tutorial: https://www.postgresql.org/docs/current/tutorial-window.html
- Modern SQL features: https://modern-sql.com/