SEO & Indexing
Robots Meta Tag Debugging Guide
Check robots meta tags, robots.txt rules and crawl directives before assuming Google ignored a new developer page. This reference is written as a practical operating note for developers, DBAs and platform teams who need to make the next troubleshooting step visible.
Use it with the live Formalint tools, then keep the final finding in a ticket, release note or runbook so the next person can repeat the same checks.
Good fit for
- A page is live but excluded by noindex.
- A static site moved from private testing to public launch.
- A crawler can reach assets but not page HTML.
Practical workflow
| Step | What to verify |
|---|---|
| Read the page source | Confirm the final deployed HTML has index, follow or no robots meta tag blocking indexing. |
| Check robots.txt | Confirm robots.txt allows the page path and points to the sitemap. |
| Look for templates | If one template is wrong, many generated pages may inherit the same directive. |
| Retest after deploy | Use URL Inspection only after the public page is updated. |
Command or evidence sample
<meta name="robots" content="index, follow, max-image-preview:large">Quality checklist
- Confirm the public URL, browser, terminal, deployment or Search Console context.
- Capture the first useful error, header, status, sitemap entry or metric before editing.
- Keep secrets, tokens, cookies and private user data out of copied examples.
- Make one meaningful fix, publish it, then verify the live page rather than the local preview.
- Link the page to related Formalint references so visitors and crawlers can understand the topic cluster.
Common mistake
The common mistake is checking local files while Google sees an older deployed page or a different host.
Formalint's rule for this topic: improve the page because it helps a real developer, then let Search Console observe the improvement naturally.
Related Formalint references
Continue with Robots Txt Sitemap Launch Guide, ads.txt Debugging, Google Search Console Indexing.
Frequently asked questions
Should this be checked after every release?
Check it whenever the related page, deployment path, crawler signal, browser behavior or user-facing workflow changes in a meaningful way.
Can I use sensitive production data in examples?
No. Redact secrets and personal data before copying commands, URLs, payloads, logs or screenshots into a public tool or support note.