nw-investigation-techniques

# Investigation Techniques

## Problem Categorization

### Technical Problems

| Category | Sub-Category | Common Symptoms |
|----------|-------------|-----------------|
| System Failures | App crashes, memory leaks, deadlocks, data corruption | Service unavailability, resource exhaustion, integrity errors |
| System Failures | Hardware, network, database, security | Connectivity loss, capacity limits, access failures |
| Performance | Response time: slow queries, latency, algorithmic inefficiency | High p95/p99, user-reported slowness |
| Performance | Throughput: thread pool exhaustion, connection limits, queue backlog | Reduced capacity, growing queues |
| Integration | Internal: component comms, data format, version conflicts | Interface errors, serialization failures |
| Integration | External: third-party availability, API changes, auth failures | Timeouts, contract violations |

### Operational Problems

| Category | Common Symptoms |
|----------|-----------------|
| Deployment: script failures, config drift, migration errors | Failed releases, environment inconsistencies |
| Monitoring: alerting gaps, backup failures, incident response | Missed incidents, slow recovery |
| Human factors: communication gaps, knowledge silos, skill gaps | Repeated mistakes, slow onboarding |

## Evidence Collection

### Technical Evidence Sources

**Logs**: application (timestamp correlation) | system/infrastructure | database | network traces

**Metrics**: performance/resource utilization | error rates/response time trends | user behavior/transaction patterns | infrastructure health/capacity

**Configuration**: system/deployment settings | code changes/VCS history (git log, blame) | env vars/dependencies | security/access controls

### Evidence Validation
1. **Cross-reference**: verify from multiple independent sources
2. **Timestamp validation**: confirm event sequence accuracy
3. **Completeness check**: identify data gaps/corruption
4. **Correlation vs causation**: distinguish co-occurrence from causation

## Analysis Techniques

### Quantitative
- **Trend**: time series of metrics, error pattern frequency
- **Distribution**: response time percentiles, error rate across components
- **Pattern recognition**: log anomalies, behavior patterns, error clustering

### Qualitative
- **Timeline reconstruction**: detailed incident timeline, correlate changes with symptoms
- **Process analysis**: workflow disruptions, communication flow, decision chains
- **Environmental**: recent changes, system load, external factors, related incidents

## Solution Design Patterns

### Immediate Mitigations (restore service)
Quick fixes | workarounds to minimize impact | emergency procedures | monitoring enhancements

### Permanent Fixes (prevent recurrence)
Architecture modifications | code quality/defensive programming | config management/environment consistency | testing/validation improvements

### Early Detection (catch faster)
Leading indicators | anomaly detection/predictive alerting | automated quality gates | threshold tuning from learnings

### Solution Prioritization Matrix

| Priority | Criteria | Action |
|----------|----------|--------|
| P0 | Active incident, users impacted | Immediate mitigation, hours |
| P1 | Root cause fix for recurring issue | Permanent fix, current sprint |
| P2 | Prevention for potential issues | Next sprint |
| P3 | Systemic improvement | Backlog with evidence |