nlweb-retrieval-backends

# NLWeb Retrieval Backends

## Before writing code

**Fetch live docs**:
1. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/nlweb-retrieval.md for the architectural overview.
2. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/config/config_retrieval.yaml for the **canonical list of endpoint names** and their defaults — config keys move release to release.
3. Pick the per-backend setup page from `docs/setup-*.md` (Qdrant, Azure AI Search, Elasticsearch, OpenSearch, Postgres, Snowflake, Cloudflare AutoRAG).
4. Inspect `AskAgent/python/retrieval_providers/<backend>.py` for the exact client signature and required env vars.
5. Verify the embedding dimension and metric (cosine/dot/L2) the backend expects — must match the embedding provider.

## Conceptual Architecture

### The Read-Fanout, Single-Write Pattern

NLWeb does something unusual: it **reads from every enabled retrieval endpoint in parallel and deduplicates by URL**, but writes go to **exactly one** `write_endpoint`. This means:
- You can run a hybrid index (e.g., local Qdrant for site content + Bing for fresh news) without code changes
- You migrate between backends by re-running `db_load` against the new `write_endpoint`
- "Result quality" is the union of all enabled stores — a noisy backend pollutes the top-k

### All Supported Backends

| Endpoint key (config_retrieval.yaml) | Backend | Notes |
|--------------------------------------|---------|-------|
| `qdrant_local` | Qdrant file-backed | Default-enabled; data in `../data/db` |
| `qdrant_url` | Qdrant remote | Set URL + API key in env |
| `nlweb_west` | Azure AI Search | Default-enabled MS-hosted demo instance — usually disable |
| `azure_ai_search` | Azure AI Search (your own) | Bring your own index name |
| `milvus` | Milvus | Flagged "under development" in YAML |
| `elasticsearch` | Elasticsearch | dense_vector + int8_hnsw |
| `opensearch_knn` | OpenSearch + k-NN plugin | The recommended OpenSearch path |
| `opensearch_script` | OpenSearch no plugin | script_score fallback, slower |
| `postgres` | Postgres + pgvector | Good if you already run Postgres |
| `snowflake_cortex_search_1` | Snowflake Cortex Search | Data lives in Snowflake tables |
| `cloudflare_autorag` | Cloudflare AutoRAG | Indexing managed by CF; ingest via R2 |
| `shopify_mcp` | Shopify's MCP endpoint | Default-enabled; live proxy, no ingest |
| `bing_search` | Bing Web Search API | Live web fallback; not a vector store |

### Read vs Write

Every endpoint declares:
- `enabled: true/false` — whether `/ask` queries it
- `read: true/false` — finer-grained: enable for reads
- (only one endpoint should be the `write_endpoint`)

The default config has `qdrant_local`, `nlweb_west`, and `shopify_mcp` enabled — for local dev disable the last two.

### Choosing a Backend

| If you need... | Use |
|----------------|-----|
| Local dev with no cloud deps | `qdrant_local` |
| Largest scale + Microsoft-stack | `azure_ai_search` |
| Already on AWS | `opensearch_knn` |
| Already on Postgres | `postgres` (pgvector) |
| Live e-commerce catalog | `shopify_mcp` |
| Snowflake-resident data | `snowflake_cortex_search_1` |
| Edge deployment | `cloudflare_autorag` |
| Live news/freshness | `bing_search` (combine with a vector backend) |

### Embedding Dimension Compatibility

Each backend stores fixed-dimension vectors. The embedding provider must emit the same dimension:

| Embedding provider | Default model | Dim |
|--------------------|---------------|-----|
| OpenAI `text-embedding-3-small` | default | 1536 |
| OpenAI `text-embedding-3-large` | — | 3072 |
| Azure OpenAI `text-embedding-3-small` | default | 1536 |
| Gemini `text-embedding-004` | — | 768 |
| Snowflake `arctic-embed-m-v1.5` | — | 768 |
| Elasticsearch `multilingual-e5-small` | — | 384 |

Pick the embedding provider FIRST, configure the backend's index to match THAT dimension, then ingest.

### Metric Compatibility

Most NLWeb providers use **cosine similarity**. When creating a new index manually (Azure AI Search, OpenSearch, Postgres) make sure the metric matches what the retrieval provider class expects. Look in `retrieval_providers/<backend>.py` for the metric the SDK call passes.

### The `nlweb_west` Trap

`nlweb_west` is a Microsoft-hosted demo Azure AI Search instance that's **enabled by default**. For most users this:
- Pollutes results with MS demo data
- Requires the demo's Azure credentials to even connect
- Costs nothing but adds latency

Disable it in local dev unless you specifically want the demo content.

## Implementation Guidance

### Switching the Write Endpoint

Edit `config/config_retrieval.yaml`:

```yaml
write_endpoint: azure_ai_search

endpoints:
  qdrant_local:
    enabled: false
  azure_ai_search:
    enabled: true
    api_key_env: AZURE_SEARCH_API_KEY
    endpoint_env: AZURE_SEARCH_ENDPOINT
    index_name: nlweb-main
```

Then re-ingest:
```bash
python -m data_loading.db_load --only-delete delete-site <site>
python -m data_loading.db_load <source> <site> --database azure_ai_search
```

### Running Multiple Backends in Parallel

Leave several `enabled: true` simultaneously — `/ask` will fan out reads, dedup by URL. Useful for:
- Hybrid: Postgres (cheap) for site content + Bing for live web facts
- A/B: Qdrant + Azure AI Search to compare retrieval quality

### Adding a New Backend Provider

If NLWeb doesn't ship the backend you need:
1. Subclass the base in `retrieval_providers/` — look at any existing one for the contract (search, upsert, delete-by-site).
2. Add an endpoint entry in `config_retrieval.yaml`.
3. Register the class in the provider factory (verify location in current code).
4. Ingest a test site.

### Backend-Specific Notes

**Qdrant local**: zero setup; collection lives at `../data/db`. To reset, delete the directory.

**Azure AI Search**: create the index manually (or via the setup doc's ARM template). Vector field must be `vector` (or whatever the provider class names it — verify).

**Postgres + pgvector**: `CREATE EXTENSION vector;` then ensure the column is `vector(1536)` or whichever dim matches your embedding. NLWeb uses cosine distance by default.

**Snowflake Cortex Search**: data is in a Snowflake table; you create a `CORTEX SEARCH SERVICE` over it. NLWeb queries via the Cortex API. No `db_load.py` involvement.

**Cloudflare AutoRAG**: upload files to R2, point AutoRAG at the bucket, wire NLWeb to the AutoRAG endpoint. CF manages indexing.

**Shopify MCP**: zero ingest. NLWeb proxies queries to a Shopify store's MCP endpoint. Configure the shop domain per-site. Disable for non-commerce deployments.

**Bing**: API key required; only useful combined with at least one vector backend (Bing returns web pages, not your indexed content).

### Debugging Empty Results

Diagnostic ladder:
1. `curl http://localhost:8000/sites` — site is registered?
2. `curl 'http://localhost:8000/ask?query=test&site=X&mode=list&streaming=false'` — any results at all?
3. Disable all backends except the one you wrote to — does the write_endpoint return results?
4. Check embedding dimension: `python -c "from embedding_providers import get_default; print(get_default().dim)"` (verify exact API) and compare to your index schema.
5. Inspect the raw store: `qdrant` CLI / Azure Search Studio / `SELECT count(*) FROM index` for Postgres.
6. Re-ingest with the embedding provider that matches the index.

Always re-fetch `config_retrieval.yaml` from the live repo before generating config — keys change.