GSC knowledge base
GSC crawling and discovery tutorials and troubleshooting guide
Check sitemap, robots, and host responses so important pages have a stable discovery path. Choose tutorials by specific issue and verify operations, limitations, and results item by item.
Check what Google can actually fetch
What to prepare
Prepare the exact sitemap and robots.txt URLs, response headers, recent server errors and links leading to the affected page. A browser opening one URL is only one observation.
| Your situation | First action | Read next |
|---|---|---|
| Sitemap submission succeeded but pages are missing | Check processing and last-read state, then verify the listed canonical URLs and discovery links. | Sitemap: submission, processing, and indexing are three states |
| Google cannot fetch the sitemap | Test the exact response, XML content and redirect chain; rule out login pages or challenge responses. | Sitemap cannot be crawled: first see what Google gets |
| Crawl volume changed or the host has errors | Review host availability and response trends alongside server logs, without treating fewer requests as a penalty. | Crawl stats: host failures take priority over crawl volume |
| Robots rules may block a path | Read the actual rules for the relevant user agent and test the specific path. Robots blocking is not noindex. | robots.txt report: verify host and actual rules |
When to stop and verify
The fetch returns the expected content and important pages have stable crawlable links. Sitemap acceptance alone does not prove a crawl.
Choose another Search Console problem · Check the official requirements
Crawling and discovery
Check sitemap, robots, and host responses so important pages have a stable discovery path.
Sitemap: submission, processing, and indexing are three states
A sitemap supplies discovery hints, not an indexing command. Build a truthful list of intended canonical pages, verify the public file and its URLs, then track submission, Google processing and page indexing separately. Lastmod should describe a real significant change, not a daily freshness script.
Steps · error branches · checklistSitemap cannot be crawled: first see what Google gets
For ‘Couldn't fetch’, first identify the exact file Google tried to read and what was delivered. Separate network/access failure from XML parsing failure. A browser's 200 response can still contain a challenge or HTML error; repeated submission does not repair either problem.
Steps · error branches · checklistCrawl stats: host failures take priority over crawl volume
Start with host availability and failing responses, not the number of crawl requests. A higher count can come from repeated resources or duplicate URLs. Combine the native Crawl Stats report with dated release and server evidence to determine whether important content was actually reachable.
Steps · error branches · checklistrobots.txt report: verify host and actual rules
Check the robots file for the exact protocol and host, then distinguish fetch success, parsing and rule matching. The native report shows Google's fetched version; your live file may be newer. Robots restrictions are crawl instructions, not authentication or a dependable removal method.
Steps · error branches · checklistCheck guide coverage and product capabilities separately
SEO operations adds bounded technical checks, saved index-version reads, laboratory records and human content review. Native Google security, Discover, News and enhancement reports still require their own tools. A guide explains a method; it does not establish every report is connected.
View real product capabilities