Shivuknet EN HE +972 54 238 3789
SEO service · robots.txt

robots.txt audit and fix — based on real GSC reports

robots.txt is accidentally blocking important pages, or failing to block /wp-admin/ and /cart/ that waste crawl budget. I scan the file, find the wrong blocks, and write a version that points Googlebot at the pages that actually earn money. ₪290 one-time.

Get a quote See pricing
robots.txt audit and fix — based on real GSC reports - Shivuknet
Why this works

Why a clean robots.txt matters

robots.txt is the first file Googlebot reads on the site. One bad line — a Disallow too broad, a missing Allow for CSS, a blocked sitemap — can wipe hundreds of pages out of the index inside two weeks.

The audit starts with a scan of the existing file against the Coverage report in GSC, then a check in robots.txt Tester. Three categories of error get flagged: wrong blocks (pages that should be indexed but are not), missing blocks (URLs like ?utm_*, /cart/, /search/ that drain crawl budget), and blocks that stop Googlebot from rendering CSS and JS.

The new file gets written from that audit: precise User-agents, focused Disallows, Allow for resources, sitemap directive, Crawl-delay only when there is a real reason. Tested again before going live, deployed to production, and verified in URL Inspection that Googlebot can reach the right pages — all inside 24-48 hours.

Scan for wrong blocks

Every Disallow line is compared against the Coverage report in GSC — important pages blocked by accident get surfaced.

Crawl budget optimization

Blocks added for /cart/, /?utm_*, /search/, /wp-admin/ — Googlebot redirected toward pages that rank.

Sitemap directive

A Sitemap: line added to robots.txt so Google discovers the sitemap even without manual submission.

Tester verification

Every change tested in GSC's robots.txt Tester, with key pages fetch-verified before going live.

Process

How we work

  1. 1

    Scan the existing file

    I pull the current robots.txt, review every line, and compare against the Coverage report to find wrong blocks.

  2. 2

    Map crawl budget

    Identify waste pages — /cart/, /?utm_*, /search/ — and prepare a focused Disallow list.

  3. 3

    Build and test

    Write the new version with Allow for CSS/JS and the sitemap directive, then test in GSC's robots.txt Tester.

  4. 4

    Deploy and verify

    Push to production, verify key pages are accessible via URL Inspection, deliver a change report within 24-48 hours.

Pricing

robots.txt audit pricing

₪290 one-time

Includes scan, crawl-budget optimization, sitemap directive, and full Tester verification.

Get a custom quote

Or see all SEO packages →

Common questions

FAQ

Does robots.txt prevent indexing?

Not directly. robots.txt blocks crawling — Google does not read the content — but a blocked URL can still appear in the index if external links point to it, just without a description. To prevent indexing entirely you need a noindex tag on the page itself, not a Disallow in robots.txt.

What is the difference between Disallow and noindex?

Disallow in robots.txt tells Googlebot "do not crawl this URL" — it protects crawl budget but does not guarantee removal from the index. noindex is a meta tag on the page itself saying "do not include this page in the index". For pages that must stay out of the index, use noindex, not Disallow.

Should I block /?utm_*?

Usually yes. UTM parameters generate endless duplicate URLs that burn crawl budget with no SEO upside. I add Disallow: /*?utm_ to robots.txt combined with canonical tags pointing to the clean version — Google focuses on the main page instead of dozens of variants.

Ready to begin

Let us help

Send a message describing what you need and we will reply with a quote and recommended next step within an hour during business hours.

WhatsApp +972-54-238-3789 support@shivuknet.co.il
Other SEO services

Related