Crawler

The seonumi crawler and link checker

If you found seonumi-audit or seonumi-links (or their earlier names GoBestSEO-Audit and GoBestSEO-Links) in your server logs or your firewall, here is who was visiting, and why.

In short

  • seonumi-audit reads the public pages of a site only after its owner registered it with us.
  • seonumi-links reads single public pages that may link to a registered site.
  • Neither logs in or submits forms.

By seonumiUpdated

What it is

seonumi-audit is the crawler of seonumi. It reads the public pages of a website so that the site’s owner gets an SEO audit. It visits a site only after its owner has registered it with us and connected the site’s own Google Search Console property, and only for that owner’s audits.

The crawler only reads. Changes an owner has agreed to, such as a better page title, are made separately, through the WordPress connection that owner set up, never by the crawler.

The link checker is a separate crawler with a different job, described in the last section of this page.

How to recognize it

Every request carries a User-Agent that starts with the product token seonumi-audit/1.0, followed in parentheses by a plus sign and the address of this page, and ends with GoBestSEO-Audit/1.0, the product token of the crawler’s earlier name. To allow it, match on the name seonumi-audit. A rule you wrote for the earlier name GoBestSEO-Audit keeps working, because every request still carries that name too.

Requests come from Google Cloud without a fixed IP address, so match on the User-Agent, not on an IP range.

What it reads, and how often

  • The public pages of the registered site, starting from the home page and the sitemap, up to the page limit of the owner’s plan (50 pages at most).
  • A status check of other addresses listed in the sitemap, without reading their content.
  • Your robots.txt and sitemap, to find pages and to report what they tell search engines.
  • It doesn’t log in or submit forms, so it can’t see anything behind a login.
  • It scans a site every day or every two days, depending on the owner’s plan, normally at night, Israel time.

If your firewall or security plugin blocks it

If you own the site and use seonumi, allow requests whose User-Agent contains seonumi-audit in your firewall, CDN or WordPress security plugin (a rule you already have for GoBestSEO-Audit keeps working). Otherwise the app will tell you that your site is blocking our crawler, and your findings will be incomplete.

If it visits a site you own but never registered with us, or it causes a problem, report the visit to our team with the address it visited and the time.

The link checker

To show customers on SEO BASIC and SEO PRO which sites link to them (backlink monitoring), seonumi also reads single public pages on other websites that may contain a link to a registered site and, for customers who use link-building outreach, to suggest pages that link to their competitors as sites they may contact, and to check daily, for up to two months, whether a link appeared after the customer contacted a site. This is seonumi-links, the link checker: a separate crawler from seonumi-audit, with its own User-Agent and robots.txt token.

Pages that link to a customer’s competitors may be suggested to that customer as sites to contact. We never collect contact details, and we never read a third-party page only to rate it or to find contact details on it. To ask that your site is never suggested to seonumi customers, contact us with the address; we act within 7 days.

  • Every request carries a User-Agent that starts with the product token seonumi-links/1.0, followed in parentheses by a plus sign and the address of this page, and ends with GoBestSEO-Links/1.0, the product token of its earlier name. To recognize it, match on the name seonumi-links.
  • Its robots.txt token is seonumi-links. It follows the rules for that token or, when there are none, the rules for all crawlers, including Crawl-delay.
  • It used to be called GoBestSEO-Links and, before that, SEOOptimizerBacklinkBot. Disallow rules you wrote for the old tokens gobestseo-links and seooptimizerbacklinkbot still apply, so a site that blocked an old name stays blocked.
  • It reads one page at a time from each site and waits between requests to the same site.
  • It never logs in or submits forms, and it keeps only the links it finds to tracked sites, not the content of your pages.
  • To keep it out of your site, add a Disallow rule for seonumi-links in robots.txt.

Next step

Want this for your own site?

The same crawler scans your WordPress site, and you decide what changes.