Skip to main content
Real Time Marketing
Real Time Marketing

Blog

Should You Block AI Crawlers? Probably Not

You can tell AI crawlers to stay off your site. Publishers do it to defend their product. For a local trade, blocking removes you from the answer for nothing.

4.98 ★ · 180 Google reviews

You can tell AI crawlers to stay off your website. Publishers have done it in numbers, and some of the same advice has been passed to contractors as though the situations were comparable. For a local trade, blocking is almost always the wrong call.

What the decision actually is

Automated crawlers read websites for different purposes: indexing pages for search, gathering material to train models, and fetching pages to answer a question somebody is asking right now. They can be allowed or refused separately, through instructions in a file on your site.

Those three are not the same decision, which is where most of the confusion starts. Refusing to be used for training is a different question from refusing to be read when a customer is asking about you.

Why publishers block and you should not

A publisher’s writing is the asset. If a system absorbs it and answers the question, the visit that paid for producing it never happens. Blocking is a defence of the product itself.

Your pages are not the product. They are how somebody finds a business that will come to their house. Being read, summarised and named costs you nothing you were selling, and being absent means the answer describes a competitor instead.

A trade that blocks these crawlers is not protecting revenue. It is removing itself from a place customers increasingly ask questions, in exchange for nothing.

The asymmetry that settles it

The downside of being read is theoretical: somebody gets an answer without visiting your site, which was never how you made money anyway.

The downside of being invisible is concrete: a homeowner asks who does this locally, and you are not among the names. There is no mechanism by which blocking produces a call.

Check you have not blocked them by accident

More common than a deliberate decision, and worth checking rather than assuming.

It happens when a site is built behind restrictive defaults, when a security plugin is configured aggressively, when a staging configuration reaches production, or when somebody applied a template of blocking rules found online without reading it. The result is a contractor who wonders why they are never mentioned, while their own site is refusing the request.

The same class of mistake as the staging setting that hides an entire new website after a redesign: invisible, easy to make, and it costs you everything until somebody looks.

The cases where blocking is reasonable

Two, and neither is common for a trade.

If you publish something genuinely proprietary: original research, a substantial reference work that is itself the offer, the publisher’s argument applies to that material and you may want it excluded.

If you have a strong objection to your content being used for training, that is a legitimate position to hold on principle. Just take it deliberately, understanding it may reduce how often you are mentioned, rather than absorbing it from advice written for a different industry.

What actually gets you mentioned

Not access permissions. Being clearly and consistently described across the places that establish a local business, and answering real questions plainly on your own pages.

Allowing the crawler is the precondition, not the strategy. It puts you in the room; what gets you named is being the clearest answer available.

Frequently asked questions

Should contractors block AI crawlers?

Almost never. Your pages are a route to a phone call rather than the product, so being read costs nothing you were selling, and being absent means the answer names a competitor.

Why do publishers block them?

Because their writing is the asset. If a summary answers the question, the visit that funded producing it never happens. That reasoning does not transfer to a trade.

Could I be blocking them without knowing?

Easily: through a security plugin, a restrictive build default, a staging configuration reaching production, or a copied set of rules. It is worth checking rather than assuming.

Does allowing them get me mentioned?

It is the precondition, not the strategy. Being mentioned comes from consistent local information and plain answers to real questions.

Where to start

Check whether your site is refusing these crawlers before deciding anything about them. A contractor blocking them by accident and wondering why they are never mentioned is a more common situation than one who chose to.

Real Time Marketing has worked with home service contractors since 2016: plumbing, HVAC, roofing, electrical and other trades, nationwide, from our office in Bradenton, Florida. Agreements are month-to-month. See our answer engine optimisation or book a strategy call.

Related reading

What Gold Star's +874% Actually Shows

webdev

Gold Star Plumbing and Drain's organic leads rose 874% over two years while organic traffic rose 246%. The gap between those two figures is the most useful part of the case study.

You Cannot Be Cited If You Do Not Rank

Courtlyn Saxby

Over 99% of AI Overview sources came from the top ten results and nearly 80% from the top three. The summary is not an alternative to ranking; it is a reward for it.

Asking Questions Versus Giving Commands

Courtlyn Saxby

Two things are happening to search and they pull opposite ways: longer questions, and commands. One rewards writing, the other rewards your listing data.

  • Since 2016
  • No long-term contracts
  • Every marketing dollar is tied directly to a closed invoice

Connect with us

Follow the work, not just the ads

Algorithm shifts, badge changes, and what they mean for a home-service business, posted where you already are.