Google updated their documentation with new information about a mysterious crawler used by Google that was not widely known until today
Google updated the list of their official crawlers by adding the name and information for a relatively unknown crawler that publishers have been seeing now and then but no documentation for it previously existed. Second Hand Excavator Machine
Although Google added official documentation for this crawler the information provided seems to encourage more clarification.
Google has several kinds of crawlers (also known as bots and spiders).
The different forms of crawlers are:
The new crawler documentation is for the Google-Safety user agent. The crawler is not new but the documentation is new.
The documentation in the Special-case Crawlers Google-Safety crawler is one that’s used by Google’s processes for finding malware.
Unique among Special-case crawlers, the Google-Safety Crawler completely ignores all robots.txt directives.
The new documentation for the Google-Safety Crawler:
“The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.
This user agent ignores robots.txt rules.”
The full agent string for the crawler:
Read the new documentation for the Google-Safety user agent on the Google Search Central page for crawlers in the section devoted to Special-case crawlers.
Overview of Google crawlers and fetchers (user agents) – Special-case crawlers
In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses -- and careers.
Used Light Duty Dump Trucks Copyright © 2023 Search Engine Journal. All rights reserved. Published by Alpha Brand Media.