Shivuknet EN HE +972 54 238 3789
SEO service · robots.txt

robots.txt audit and fix, checked in Search Console

robots.txt can block important pages and resources by accident, or leave cart pages, internal search and tracking parameters open for Google to waste time on. I go through the file line by line, fix the wrong rules, and check the result in Search Console's robots.txt report. ₪290 one-time.

Get a quote See pricing
robots.txt audit and fix, checked in Search Console - Shivuknet
Why this works

Why a clean robots.txt matters

robots.txt is a text file at the root of the site that tells search engine crawlers which URLs they may access. One overly broad Disallow line can block crawling of a whole section, so it is worth checking every time the site structure changes.

Important: robots.txt does not keep a page out of Google. Google states that a blocked URL can still be indexed if other pages link to it, just without a description. To keep a page out of the index you need a noindex rule or password protection, and in that case the page must not be blocked in robots.txt, or Google will never see the noindex.

The audit starts with the existing file compared against the Page indexing report in Search Console, then a check in the robots.txt report, which replaced the old robots.txt Tester in late 2023. I then write a corrected file, deploy it, request a recrawl of the file from the report, and confirm with URL Inspection that the important pages are open to crawling.

Find wrong blocks

Every Disallow line is compared against the Page indexing report, and important pages or resources blocked by accident get surfaced.

Point crawling at the right pages

Cart pages, internal search and tracking parameters that do not need crawling get blocked, so Google focuses on pages that bring you customers.

Blocking versus removal

Pages that need to leave the index get noindex, not Disallow, so they do not linger in results without a description.

robots.txt report check

After deployment I confirm in Search Console's robots.txt report that Google fetched the new file without errors, and check key pages with URL Inspection.

Process

How we work

  1. 1

    Scan the existing file

    I pull the current robots.txt, review every line, and compare it against the Page indexing report to find wrong blocks.

  2. 2

    Map pages that do not need crawling

    I identify URLs like /cart/, internal search and UTM parameters, and separate what needs a crawl block from what needs noindex.

  3. 3

    Write the corrected file

    A new file with CSS and JS left open, a Sitemap line, and only focused blocks.

  4. 4

    Deploy and verify

    Push to production, request a recrawl of the file in the robots.txt report, confirm key pages are open, and deliver a change report.

Pricing

robots.txt audit pricing

₪290 one-time

Includes the file scan, fixes, a Sitemap line, and verification in Search Console's robots.txt report.

Get a custom quote

Or see all SEO packages →

Common questions

FAQ

Does robots.txt prevent indexing?

No. robots.txt blocks crawling, so Google does not read the content, but a blocked URL can still appear in results if links point to it, just without a description. To keep a page out of Google, use a noindex rule on the page or password protection. Note that if the page is blocked in robots.txt, Google cannot see the noindex.

What happened to the robots.txt Tester in Search Console?

Google retired it at the end of 2023 and replaced it with the robots.txt report, found under Settings in Search Console. The report shows which robots.txt files Google found for your site, when they were last crawled and any warnings or errors, and lets you request a recrawl of the file after an urgent fix.

Should I block UTM parameter URLs?

Usually not as the first fix. UTM parameters create duplicate URLs for the same page, but if those URLs are blocked in robots.txt, Google cannot read the canonical tag on them. The cleaner fix is a canonical tag on every page pointing to the clean version, and no UTM links in your internal linking. I add a robots.txt block for parameters only when they create crawling waste at scale.

Ready to begin

Let us help

Send a message describing what you need and we will reply with a quote and recommended next step within an hour during business hours.

WhatsApp +972-54-238-3789 support@shivuknet.co.il
Other SEO services

Related