PROFESSIONALLY OPTIMIZED WEBSITES STARTING AT $995
Our team of SEO Web Design gurus are standing by to assist you achieve your online marketing goals.

+1-971-599-3330

REQUEST QUOTE
SEO Web Design, LLC aims to improve business by delivering effective solutions based on innovative technologies and professional designs. Discover the variety of services we offer and convince yourself on the basis of the latest works that we've done. We love building fresh, unique and usable websites optimized specifically for your niche.

Responsive Web Design

SEO / SEM / Social Media

Conversion Rate Optimization

Email Marketing

Online Presence Analysis

Web Hosting
Top
SEO Web Design, LLC / SEO  / Google: Do Not Block GoogleBot From Crawling 404s

Google: Do Not Block GoogleBot From Crawling 404s

John Mueller of Google said it would be “a really bad idea which will cause all sorts of problems” if you block Google or other search engines from crawling pages that return a 404 server status code. He said “billions of 404 pages are crawled every day” by Google and it is normal.

One webmaster wrote that his “website automatically blocks user agents that get more than 10 404 errors, including Googlebot, so that’s a problem.” John responded to that that this is a really bad idea, he said “That sounds like a really bad idea which will cause all sorts of problems.. You can’t avoid that Googlebot & all other search engines will run into 404s. Crawling always includes URLs that were previously seen to be 404.”

That sounds like a really bad idea which will cause all sorts of problems.. You can’t avoid that Googlebot & all other search engines will run into 404s. Crawling always includes URLs that were previously seen to be 404.

— 🍌 John 🍌 (@JohnMu) July 15, 2020[1]

He said in a different tweet, the same day, “Billions of 404 pages are crawled every day – it’s a normal part of the web, it’s the proper way to signal that a URL doesn’t exist. That’s not something you need to, or can, suppress.”

Billions of 404 pages are crawled every day – it’s a normal part of the web, it’s the proper way to signal that a URL doesn’t exist. That’s not something you need to, or can, suppress.

— 🍌 John 🍌 (@JohnMu) July 15, 2020[2]

So while you can fix your 404 pages through other means, automatically blocking Google from accessing 404 pages without knowing how Google is accessing those pages can be a really bad idea.

Forum discussion at Twitter[3].

References

  1. ^ July 15, 2020 (twitter.com)
  2. ^ July 15, 2020 (twitter.com)
  3. ^ Twitter (twitter.com)

Powered by WPeMatico

[email protected]

Search Engine Roundtable is a well-rounded view on search engines and search engine marketing from five segments of the Web population represented by senior members of the major SEO/SEM forums on the Internet.

No Comments

Sorry, the comment form is closed at this time.