noindex vs robots.txt: keep a page out of Google
Read the conclusion first
Robots.txt controls crawling; noindex controls indexing when Google can read the instruction. First decide whether the URL should remain public and searchable, remain public but excluded, or require authentication. Apply the control for that purpose and inspect the delivered response, not only the source template.
Applicable conditions
Applies to accidentally blocked public pages, test sites, and pages that need to be removed from the index. Private data cannot be protected by these rules alone.
Before you open the report
The exact host’s robots.txt, the page HTML and HTTP headers, and whether the goal is privacy or search exclusion.
Use one concrete URL or report first. Keep account credentials and private visitor data out of shared notes.
Steps and verification
Inspect all delivered restriction layers
Fetch the final page and inspect its robots meta and X-Robots-Tag headers, then the robots.txt for the exact protocol, host and port. Check redirects and rendering so a proxy or alternate route is not adding a restriction you missed.
Save the actual directive values and intended state. A CMS checkbox is not deployment evidence. For a PDF or other non-HTML response, inspect the supported HTTP header rather than searching for an HTML meta tag.
Repair an accidentally excluded public page
For content intended to be searchable, remove the unintended noindex at its source and ensure the applicable robots rule permits fetching the page and necessary resources. Retest representative URLs if a shared template or CDN rule caused the problem.
Check that the final response contains real content and a coherent canonical. The completed live repair establishes today's accessibility and directives; later URL Inspection must establish whether the indexed version has been updated.
Exclude public content without hiding the instruction
For a public page that should leave the index, use a supported noindex instruction and allow crawling so it can be read. A robots Disallow by itself can leave a known URL appearing without page content. Urgent search removal and a durable site change are separate tasks.
For account data or private environments, use authentication and authorization. Noindex, nofollow and publicly readable robots paths do not prevent people from accessing a URL. A staging environment's exclusion policy must not leak accidentally into production.
Retest both intended and protected routes
After release, test one public indexable page, one deliberately excluded page and a private route using an unauthenticated request. Check actual content, headers and robots matching for each. Retain the before/after configuration and rollback reference.
Do not remove every restriction merely to improve indexing totals. Completion means the controls match each URL's real purpose; search removal or renewed indexing still needs its own later evidence.
Common mistake: noindex written in robots.txt
Google does not support setting page noindex this way; use the supported HTML or HTTP header instead.
How to decide when you encounter different situations
Match your actual observation to a row before changing the page. If it does not match, collect the missing evidence first.
| What you see | Next step | How to confirm |
|---|---|---|
| Important public page mistakenly has noindex | Check HTML and response header sources | Official page is readable and the directive has been removed |
| Index-removed page is blocked by robots | Plan a crawlable noindex according to the goal | Google can read the directive |
| Private account pages | Use authentication and access control | Do not rely on crawler voluntary compliance |
A blocked crawl cannot reliably read your noindex
Teaching scenario. Names, URLs and numbers illustrate the method; they are not customer or measured site results.
- What you observe
- Example: a public page is blocked by robots.txt and also contains noindex. The team expects the noindex to be read immediately.
- What it establishes
- If crawling is blocked, Google may not fetch the page to see its noindex. A URL can still be known from links.
- What to do next
- For a public page intended to leave the index, plan a crawlable noindex response. For private information, enforce authentication or remove the source; do not temporarily expose secrets to achieve indexing behavior.
What counts as a checked result
Check the actual URL’s policy in both HTML and headers, then verify search exclusion separately. Robots directives are not an access-control test.
If a required check fails, keep the error and return to the matching decision branch. A blank record means unverified, not passed.
Leave with a checked result, not another vague instruction
- Your deliverable
- A per-URL control decision and tests for public, excluded and authenticated paths.
- Stop and resolve this first
- Do not treat noindex or robots as protection for private records.
Record missing evidence in the worksheet below. An unresolved test is useful information; a blank field is not a pass.
Checklist for this page
Fill in your own check results, then decide the next step. Blank means not yet verified, not normal. Content is saved only in the current local session and is not submitted to the server or AI.
Frequently asked questions
Can nofollow protect account data?
No. It is not authentication or permission control.
Where can a noindex come from besides the HTML?
Check the X-Robots-Tag response header and the template or server rule that creates it. Fixing one visible meta tag does not remove a conflicting header.
Official materials and usage boundaries
This page provides operational judgment and checking methods, and does not show customer success cases. Calculation examples and request samples are teaching materials, not actual website metrics. Google's reports, conditions, and interfaces may change; please refer to the current official documentation.
Read the corresponding Google official documentationContinue with the relevant tool
Save the HTML, header observation and intended exclusion in a workspace record. Do not treat an inaccessible page as evidence that Google read noindex.
See how it works →