indexing
When the user wants to fix indexing issues from Search Console, use noindex, or implement Google Indexing API. Also use when the user mentions "fix indexing," "not indexed," "Crawled - currently not indexed," "discovered - currently not indexed," "index coverage," "noindex," "noindex tag," "pages not indexed," "why not indexed," "request indexing," or "Google Indexing API." For sitemap, use xml-sitemap.
Best use case
indexing is best used when you need a repeatable AI agent workflow instead of a one-off prompt.
When the user wants to fix indexing issues from Search Console, use noindex, or implement Google Indexing API. Also use when the user mentions "fix indexing," "not indexed," "Crawled - currently not indexed," "discovered - currently not indexed," "index coverage," "noindex," "noindex tag," "pages not indexed," "why not indexed," "request indexing," or "Google Indexing API." For sitemap, use xml-sitemap.
Teams using indexing should expect a more consistent output, faster repeated execution, less prompt rewriting.
When to use this skill
- You want a reusable workflow that can be run more than once with consistent structure.
When not to use this skill
- You only need a quick one-off answer and do not need a reusable workflow.
- You cannot install or maintain the underlying files, dependencies, or repository context.
Installation
Claude Code / Cursor / Codex
Manual Installation
- Download SKILL.md from GitHub
- Place it in
.claude/skills/indexing/SKILL.mdinside your project - Restart your AI agent — it will auto-discover the skill
How indexing Compares
| Feature / Agent | indexing | Standard Approach |
|---|---|---|
| Platform Support | Not specified | Limited / Varies |
| Context Awareness | High | Baseline |
| Installation Complexity | Unknown | N/A |
Frequently Asked Questions
What does this skill do?
When the user wants to fix indexing issues from Search Console, use noindex, or implement Google Indexing API. Also use when the user mentions "fix indexing," "not indexed," "Crawled - currently not indexed," "discovered - currently not indexed," "index coverage," "noindex," "noindex tag," "pages not indexed," "why not indexed," "request indexing," or "Google Indexing API." For sitemap, use xml-sitemap.
Where can I find the source code?
You can find the source code on GitHub using the link provided at the top of the page.
SKILL.md Source
# SEO Technical: Indexing
Guides indexing troubleshooting and fix actions. For how to find and diagnose issues in GSC, see **google-search-console**.
**When invoking**: On **first use**, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On **subsequent use** or when the user asks to skip, go directly to the main output.
## Scope (Technical SEO)
- **Fix actions**: noindex, canonical, content quality, URL Inspection; verify robots.txt does not block (see **robots-txt**)
- **Noindex**: Page-level index control; which pages to exclude and how. Complements **robots-txt** (path-level crawl control) and **google-search-console** (Coverage diagnosis)
## Initial Assessment
**Check for project context first:** If `.claude/project-context.md` or `.cursor/project-context.md` exists, read it for site URL and indexing goals.
Identify issue from GSC (see **google-search-console** for Coverage report, issue types, diagnosis workflow). Then apply fix below.
## Crawled - Currently Not Indexed
| Cause | Action |
|-------|--------|
| Low quality, duplicate, off-topic | Improve content, fix duplicates, set correct canonical |
| Static assets (CSS/JS) | See below |
| Feed, share URLs with params | Usually OK to ignore; or noindex, canonical to main URL |
| Important content pages | Use URL Inspection, verify canonical/internal links/sitemap, Request indexing |
### Static Assets (Next.js / Vercel)
Vercel adds unique `dpl=` params to static assets per deploy, creating many "Crawled - currently not indexed" URLs.
| Do | Don't |
|----|-------|
| Keep robots.txt allowing `/_next/` | Do not block `/_next/` (breaks CSS/JS loading). See **robots-txt** |
| Accept static assets in GSC as expected | Do not block `/_next/static/css/` or `?dpl=` |
| Use X-Robots-Tag for static assets | CSS/JS should not be indexed; no SEO impact |
Static assets in "Crawled - currently not indexed" is **normal and expected**.
## Other Issue Types (from GSC Coverage)
| Issue | Fix |
|-------|-----|
| Excluded by «noindex» tag | Remove noindex if accidental; keep if intentional |
| Blocked by robots.txt | See **robots-txt**; remove Disallow for important paths |
| Redirect / 404 | Fix URL or add redirect |
| Duplicate / Canonical | Set correct canonical; usually OK |
| **Soft-404** | Page returns 200 but content says "not found" or empty—Google may treat as 404. Fix: return 404 status for truly missing pages; or add real content for 200 pages |
### Soft-404
A soft-404 occurs when a page returns HTTP 200 but the content indicates the page doesn't exist (e.g. "Page not found" message, empty state). Google may treat it as 404 and exclude from index.
| Fix | When |
|-----|------|
| **Return 404** | Page truly doesn't exist; use proper 404 status |
| **Add content** | Page is intentional (e.g. empty search results); ensure substantive content or use noindex |
| **Redirect** | If URL moved, use 301 to correct destination |
## Noindex Usage
- **How**: `metadata.robots = { index: false }` or `<meta name="robots" content="noindex">` or X-Robots-Tag
- **Rationale**: Not all site content should be indexed; noindex is a valid choice for many pages
- **Caution**: Avoid noindex on important content pages
- **With robots.txt**: robots.txt = path-level crawl control; noindex = page-level index control. Do **not** block noindex pages in robots.txt—crawlers must access the page to read the directive. Use both: robots for /admin/, /api/; noindex for /login/, /thank-you/, etc. See **robots-txt** for when to use which.
- **nofollow ≠ noindex**: nofollow controls link equity only; it does **not** prevent indexing. To exclude from search, use noindex. See **page-metadata** for meta robots implementation.
### Page Types That Typically Need Noindex
| Category | Page Types | Typical Meta | Reason |
|----------|------------|--------------|--------|
| **Auth & Account** | Login, Signup, Password reset, Account dashboard | Login: `noindex,nofollow`; Signup: `noindex,follow` | No search value; login indexed = security risk; signup follow allows crawl of Privacy/Terms links |
| **Admin & Private** | Admin, Staging, Test pages, Internal tools | `noindex,nofollow` | Not for public; avoid discovery |
| **Conversion Endpoints** | Thank-you, Confirmation, Checkout success, Download gate | `noindex,follow` | Post-conversion; no SERP value; allow link equity |
| **System & Utility** | 404, Internal search results, Faceted/filter URLs | `noindex,follow` or `noindex,nofollow` | Thin/duplicate; 404 = error state |
| **Legal** | Privacy, Terms, Cookie Policy (optional) | Often `noindex,follow` | Low-value indexed; reduces clutter |
| **Duplicate & Thin** | Printer-friendly, Parameter URLs, Near-duplicate | `noindex,follow` or canonical | Duplicate content; canonical preferred when possible |
| **Low-Value** | Media kit, Feedback board (external), Thin press | `noindex` or index for brand queries | Case-by-case |
**noindex,follow vs noindex,nofollow**: Use `noindex,follow` for most cases—excludes from SERP but allows link equity. Use `noindex,nofollow` only for login (security), staging, or temporary test pages.
## Google Indexing API
| Type | Typical use |
|------|-------------|
| JobPosting | Job boards |
| BroadcastEvent | Live platforms |
**Requirements**: Enable Indexing API, create service account, add owner in Search Console, request quota (default 200 URLs/day).
## Output Format
- **Action items**: Prioritized fixes
- **References**: [Page indexing report](https://support.google.com/webmasters/answer/7440203)
## Related Skills
- **google-search-console**: Find and diagnose indexing issues in GSC
- **robots-txt**: Path-level crawl control; when to use robots.txt vs noindex; do not block /_next/ or noindex pages
- **page-metadata**: Meta robots implementation; noindex vs nofollow
- **xml-sitemap**: Submit and maintain sitemap
- **indexnow**: Faster indexing for Bing
- **canonical-tag**: Resolve duplicate contentRelated Skills
website-structure
When the user wants to plan website structure, decide which pages to build, or prioritize pages for a new or existing site. Also use when the user mentions "website structure," "site structure," "which pages do I need," "page planning," "sitemap planning," "Must Have pages," "website architecture," or "site hierarchy." For a specific page template (e.g. homepage), use homepage-generator or landing-page-generator as appropriate. Not for organic SEO roadmap alone; use seo-strategy.
seo-strategy
When the user wants to plan SEO strategy, prioritize SEO work, or understand the SEO workflow. Also use when the user mentions "SEO strategy," "SEO plan," "SEO roadmap," "SEO priority," "SEO audit," "SEO workflow," "where to start SEO," "SEO approach," "organic growth strategy," "why SEO," "SEO value," or "search strategy." For technical/crawl audit execution, use seo-audit. For keyword research, use keyword-research. For AI search visibility, use generative-engine-optimization.
seo-audit
When the user wants to run an SEO audit, technical SEO audit, or site health check. Also use when the user mentions "SEO audit," "technical audit," "site audit," "crawl audit," "indexing audit," "SEO health," or "fix SEO issues." For prioritization and organic strategy, use seo-strategy. For GSC data analysis, use google-search-console.
retention-strategy
When the user wants to reduce churn, improve customer retention, or plan lifecycle marketing. Also use when the user mentions "retention," "churn," "customer lifecycle," "churn prevention," "at-risk customers," or "loyalty program." For lifecycle, use growth-funnel.
research-sources
When the user wants to find information sources for content ideation, competitor monitoring, or industry tracking. Also use when the user mentions "research sources," "information sources," "content ideation," "industry monitoring," "competitor monitoring," "market intelligence," "content research," or "topic research." For keywords, use keyword-research.
product-launch
When the user wants to plan a product launch, execute launch channels, or create a launch checklist. Also use when the user mentions "product launch," "launch strategy," "product announcement," "launch channels," or "market launch." For GTM motion and positioning, use gtm-strategy. For cold start and first users, use cold-start-strategy. For Product Hunt day-of, use product-hunt-launch.
pmf-strategy
When the user wants to validate product-market fit, measure PMF, or plan before scaling. Also use when the user mentions "PMF," "product-market fit," "product market fit," "Sean Ellis test," "very disappointed," "vitamin vs painkiller," "PMF validation," "premature scaling," or "validate before scale." For GTM after validation, use gtm-strategy.
indie-hacker-strategy
When the user wants indie hacker or bootstrapping founder strategy—growth, channels, Build in Public, or solo founder tactics. Also use when the user mentions "indie hacker," "indie developer," "bootstrapping," "bootstrapped founder," "solo founder," "Build in Public," "scratch your own itch," "Micro-SaaS," "first 100 users," or "solo company." For cold start, use cold-start-strategy.
gtm-strategy
When the user wants to plan go-to-market strategy, GTM framework, or market entry. Also use when the user mentions "GTM," "go-to-market," "market entry," "new market," "repositioning," "PLG," "sales-led," "product-led," "marketing-led," "ICP," "buyer persona," "GTM motion," or "market expansion." For launch checklist, use product-launch.
growth-funnel
When the user wants to plan growth using the AARRR framework, diagnose growth bottlenecks, or map actions across the customer lifecycle. Also use when the user mentions "growth funnel," "AARRR," "pirate metrics," "acquisition activation retention," "customer lifecycle metrics," or "growth framework." For retention tactics, use retention-strategy.
conversion-optimization
When the user wants to improve conversion rates, run A/B tests, optimize funnels, or reduce friction. Also use when the user mentions "CRO," "conversion rate optimization," "A/B test," "split test," "funnel optimization," "checkout optimization," "form optimization," or "conversion funnel." For pricing psychology, use pricing-strategy.
cold-start-strategy
When the user wants to plan cold start, get first users, or launch a new product with zero traction. Also use when the user mentions "cold start," "cold start problem," "first users," "seed users," "finding users," "finding early users," "Fiverr Upwork," "comment outreach," "Twitter search users," "product launch strategy," "0 to 1 growth," "early-stage acquisition," "launch channels," "get first customers," "Product Hunt launch," "AppSumo," "LTD," "indie hacker," "bootstrapping," or "solo founder." For directory listing copy and submissions, use directory-submission. For Product Hunt day-of execution, use product-hunt-launch. For GTM motion design, use gtm-strategy.