Crawler

The GO BEST SEO crawler and link checker

If you found GoBestSEO-Audit or GoBestSEO-Links in your server logs or your firewall, here is who was visiting, and why.

What it is

GoBestSEO-Audit is the crawler of GO BEST SEO. It reads the public pages of a website so that the site’s owner gets an SEO audit. It visits a site only after its owner has registered it with us and connected the site’s own Google Search Console property, and only for that owner’s audits.

The crawler only reads. Changes an owner has agreed to, such as a better page title, are made separately, through the WordPress connection that owner set up, never by the crawler.

The link checker is a separate crawler with a different job, described below.

How to recognize it

Every request carries a User-Agent that starts with the product token GoBestSEO-Audit/1.0, followed in parentheses by a plus sign and the address of this page. To allow it, match on the name GoBestSEO-Audit.

Requests come from Google Cloud without a fixed IP address, so match on the User-Agent, not on an IP range.

What it reads, and how often

The public pages of the registered site, starting from the home page and the sitemap, up to the page limit of the owner’s plan (25 pages at most).
A status check of other addresses listed in the sitemap, without reading their content.
Your robots.txt and sitemap, to find pages and to report what they tell search engines.
It doesn’t log in or submit forms, so it can’t see anything behind a login.
It scans a site every day or every two days, depending on the owner’s plan, normally at night, Israel time.

If your firewall or security plugin blocks it

If you own the site and use GO BEST SEO, allow requests whose User-Agent contains GoBestSEO-Audit in your firewall, CDN or WordPress security plugin. Otherwise the app will tell you that your site is blocking our crawler, and your findings will be incomplete.

If it visits a site you own but never registered with us, or it causes a problem, report the visit to our team with the address it visited and the time.

The link checker

To show customers on Pro and Extra which sites link to them (backlink monitoring), GO BEST SEO also reads single public pages on other websites that may contain a link to a registered site. This is GoBestSEO-Links, the link checker: a separate crawler from GoBestSEO-Audit, with its own User-Agent and robots.txt token.

Every request carries a User-Agent that starts with the product token GoBestSEO-Links/1.0, followed in parentheses by a plus sign and the address of this page. To recognize it, match on the name GoBestSEO-Links.
Its robots.txt token is gobestseo-links. It follows the rules for that token or, when there are none, the rules for all crawlers, including Crawl-delay.
It used to be called SEOOptimizerBacklinkBot. Disallow rules you wrote for the old token seooptimizerbacklinkbot still apply, so a site that blocked the old name stays blocked.
It reads one page at a time from each site and waits between requests to the same site.
It never logs in or submits forms, and it keeps only the links it finds to tracked sites, not the content of your pages.
To keep it out of your site, add a Disallow rule for gobestseo-links in robots.txt.