infrastructure-overview

# Infrastructure Layer

The `infrastructure/` package provides generic, reusable functionality for research projects. All business logic resides here (Layer 1) or in project-specific `src/` (Layer 2). Scripts are thin orchestrators that coordinate these modules.

## Finding skills (Cursor and other agents)

- **Hub:** This file lists every subpackage skill and a module map.
- **Manifest:** `.cursor/skill_manifest.json` lists every skill (`name`, `description`, `path`); regenerate with `uv run python -m infrastructure.skills write`.
- **Search:** `infrastructure/**/SKILL.md` (hub `infrastructure/SKILL.md`, top-level subpackage skills, and nested skills such as `infrastructure/core/telemetry/SKILL.md`, `infrastructure/reference/citation/SKILL.md`, and `infrastructure/search/literature/SKILL.md`).
- **In Cursor:** `@infrastructure/SKILL.md` or `@infrastructure/<module>/SKILL.md` to load context; pair with the matching `AGENTS.md` for API tables.
- **Frontmatter:** Each `SKILL.md` has YAML `name` and `description` for routing (see also the table in `infrastructure/README.md`).

## Module Map

| Module | Purpose | Key Imports |
|--------|---------|-------------|
| `autoresearch/` | Deterministic research plans, stage-gate readiness, artifact/evidence reports | `build_autoresearch_plan`, `validate_autoresearch_plan` |
| `benchmark/` | Deterministic benchmark manifests and public exemplar readiness scoring | `run_benchmark_manifest`, `score_project_against_manifest` |
| `config/` | Repo defaults (`.env.template`, `secure_config.yaml`) | (YAML/dotenv; use `core.config_loader`) |
| `core/` | Logging, config, exceptions, pipeline, progress, retry, security | `get_logger`, `load_config`, `PipelineExecutor` |
| `docker/` | `Dockerfile`, `docker-compose.yml` for container runs | (CLI / `docs/CLOUD_DEPLOY.md`) |
| `validation/` | PDF/Markdown/output validation, link checking, audits | `validate_pdf_rendering`, `validate_markdown`, `verify_output_integrity` |
| `rendering/` | Multi-format output (PDF, HTML, slides) | `RenderManager`, `RenderingConfig` |
| `documentation/` | Figure/image management, markdown integration, glossary | `FigureManager`, `ImageManager`, `MarkdownIntegration` |
| `llm/` | Local LLM integration via Ollama | `LLMClient`, `OllamaClientConfig`, `detect_repetition`, `check_format_compliance` |
| `methods/` | Methods orchestration plans over DAG, manuscript methods, artifacts, and evidence | `build_methods_orchestration_plan`, `validate_methods_orchestration_plan` |
| `orchestration/` | Pipeline CLI, menu, project discovery, stage logging, secure-run wrapper (backs `run.sh`/`secure_run.sh`) | `build_parser`, `PipelineRunner`, `validate_project_slug`, `run_secure_pipeline` |
| `publishing/` | Academic publishing (DOI, citations, Zenodo, arXiv) | `publish_to_zenodo`, `generate_citation_bibtex` |
| `reference/` | BibTeX I/O (read / write / convert) matching `references.bib` syntax | `parse_bibfile`, `render_database`, `paper_to_bibentry` |
| `search/` | Paperclip-style multi-source literature search (arXiv, Crossref, local, Paperclip) | `LiteratureClient`, `SearchQuery`, `ArxivBackend`, `CrossrefBackend` |
| `prose/` | Prose analysis (readability, structure, editorial quality, manuscript reports) | `analyze_manuscript`, `compute_metrics`, `analyze_structure`, `analyze_quality` |
| `reporting/` | Pipeline reports, error aggregation, dashboards | `generate_pipeline_report`, `get_error_aggregator` |
| `scientific/` | Benchmarking, numerical stability, scientific templates | `benchmark_function`, `check_numerical_stability` |
| `project/` | Multi-project discovery and validation | `discover_projects`, `validate_project_structure` |
| `skills/` | Enumerate and parse `SKILL.md`; maintain Cursor manifest | `discover_skills`, `write_skill_manifest` |
| `steganography/` | Cryptographic PDF watermarking and verification | `SteganographyProcessor`, `SteganographyConfig`, `embed_steganography` |
| `core/telemetry/` | Unified pipeline telemetry (stage resources + diagnostics, JSON/text reports) | `TelemetryCollector`, `TelemetryConfig` |

## Import Patterns

```python
# Convenience imports (exported on infrastructure package root)
from infrastructure import get_logger, load_config, TemplateError

# Direct submodule imports (recommended for clarity)
from infrastructure.core import CheckpointManager
from infrastructure.core.pipeline import PipelineExecutor
from infrastructure.rendering import RenderManager
from infrastructure.llm.core.client import LLMClient
from infrastructure.methods import build_methods_orchestration_plan
from infrastructure.validation import validate_markdown, validate_pdf_rendering
from infrastructure.skills import discover_skills

# In project scripts (thin orchestrator pattern)
from infrastructure.core.logging.utils import get_logger
from infrastructure.validation import validate_pdf_rendering
```

## Architecture Principles

1. **Thin Orchestrator Pattern** — Scripts import from infrastructure; never implement business logic
2. **No Mocks Policy** — All tests use real data and computations
3. **Test Coverage** — Infrastructure requires 60%+ coverage
4. **Exception Hierarchy** — All errors extend `TemplateError` from `infrastructure.core.exceptions`
5. **Unified Logging** — Always use `get_logger(__name__)` for consistent output

## Common Workflows

### Running Infrastructure Tests

```bash
uv run pytest tests/infra_tests/ --cov=infrastructure --cov-fail-under=60
```

### Using the Pipeline

```bash
uv run python scripts/execute_pipeline.py --project template_code_project --core-only
```

### Validating Outputs

```bash
uv run python -m infrastructure.validation.cli.main markdown projects/template_code_project/manuscript/
uv run python -m infrastructure.validation.cli.main pdf output/template_code_project/pdf/
```

## Submodule Skills

Each subpackage has a `SKILL.md` (YAML frontmatter + body) for agent discovery—search the repo for `infrastructure/**/SKILL.md` or open paths below:

- `infrastructure/config/SKILL.md` — Repo configuration templates and secure defaults
- `infrastructure/autoresearch/SKILL.md` — Deterministic AutoResearch readiness planning
- `infrastructure/benchmark/SKILL.md` — Deterministic public exemplar benchmark harnesses
- `infrastructure/core/SKILL.md` — Foundation utilities
- `infrastructure/docker/SKILL.md` — Container build and compose
- `infrastructure/doctor/SKILL.md` — Repository diagnostics and audited repairs
- `infrastructure/documentation/SKILL.md` — Figure and doc management
- `infrastructure/llm/SKILL.md` — LLM integration
- `infrastructure/methods/SKILL.md` — Methods orchestration plans
- `infrastructure/orchestration/SKILL.md` — Pipeline CLI, menu, and runner orchestration
- `infrastructure/prose/SKILL.md` — Manuscript prose quality analysis
- `infrastructure/project/SKILL.md` — Project discovery and validation
- `infrastructure/publishing/SKILL.md` — Academic publishing
- `infrastructure/reference/SKILL.md` — Bibliographic reference workflows
- `infrastructure/rendering/SKILL.md` — Output generation
- `infrastructure/reporting/SKILL.md` — Pipeline reporting
- `infrastructure/scientific/SKILL.md` — Scientific computing
- `infrastructure/search/SKILL.md` — Literature discovery and search workflows
- `infrastructure/skills/SKILL.md` — Programmatic skill discovery and manifest I/O
- `infrastructure/steganography/SKILL.md` — Secure PDF post-processing
- `infrastructure/core/telemetry/SKILL.md` — Unified pipeline telemetry (nested under `core/`)
- `infrastructure/validation/SKILL.md` — Quality assurance