SEOmise

Region and currency

Crawler information

SEOmiseBot

SEOmiseBot is the crawler of SEOmise, a tool that helps businesses improve how their websites appear in search. It reads public web pages the way a search engine does, says who it is in every request and obeys robots.txt.

User agents

Every request SEOmise makes to other websites carries one of these user agents. Each contains the address of this page.

Mozilla/5.0 (compatible; SEOmiseBot/2.81; +https://seomise.com/bot)

Every page SEOmise reads: site scans, audits, uptime checks, link checks, sitemap checks, speed checks, placement checks, app store pages, the AI content check, the JavaScript renderer and the research crawl.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Only on a customer's own website that they verified, when MiseGuard checks it for cloaked spam pages (pages shown only to search engines). Never on anyone else's site. Sent with a From header giving our contact address.

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36

Only on a customer's own website that they verified, when MiseGuard checks it for cloaked spam pages (pages shown only to search engines). Never on anyone else's site. The same page as a visitor's browser sees it, to compare.

The SEOmise cloaking check (MiseGuard). Hacked sites often show search engines other pages than their visitors ("cloaking"). To catch it, MiseGuard reads a page as Google (the Googlebot user agent above, with a From header giving our contact address) and as an ordinary browser, and compares the two.

It does this only on a customer's own website that they verified (Domain verification), never on anyone else's site. On any other site the check reads the page as SEOmiseBot only, and skips the read as Google. It cannot borrow Google's addresses and solves no challenges.

Why it visits

  • Audits for customers. Most visits happen because someone added a website to a project on SEOmise: usually its owner, their agency, or a business comparing itself with a competitor.
  • A small research crawl (pilot). SEOmise is testing a crawl of public pages in Ireland and the United Kingdom, to build its own view of how sites link to each other. It visits domains our team chooses and the competitor sites of customers who have switched this on. It is small: at most 300 addresses a day across all sites.

What it collects

  • Facts about each page: its status code, title, description, headings, language, word, image and link counts, the types of structured data it uses, and the addresses of links to other sites.
  • It does not keep the page's HTML or its text, does not submit forms, log in or run checkout flows, and does not try to reach private network addresses.
  • What the research crawl finds is used inside SEOmise. It is never published, and it is not shown to anyone who is not a customer.

Crawl rate

1request at a time to one site
5 sat least between two requests to one site (research crawl)
4 MBlargest page it reads

Customer audits make at most five requests at a time to one site, fewer when a site responds slowly. The research crawl makes one request at a time and honours Crawl-delay up to 60 seconds (customer audits up to 10 seconds). When your server answers 429 Too Many Requests or 503 Service Unavailable, the research crawl stops visiting your site for the time given in Retry-After, or for 15 minutes when there is none.

Addresses it uses

The addresses SEOmiseBot uses are not published yet. Until they are, allow it by its user agent together with the per-site secret header X-SEOmise-Verify (see below), not by address.

The same list, for scripts and firewalls: /bot/ips.json (JSON, with separate IPv4 and IPv6 lists).

How to verify it is really us

  • The user agent contains SEOmiseBot and the address of this page. Anyone can copy a user agent, so check one of the points below as well.
  • The addresses are not published yet, so a request cannot be checked by its address for now.
  • A reverse DNS check is not offered. We do not publish a host name for our addresses, so a host name lookup cannot verify SEOmiseBot; use the points above and below instead.
  • The per-site secret header. A SEOmise customer can turn on a secret value for their own site in the site's Site health card. SEOmiseBot then sends it in the X-SEOmise-Verify header, only to that site, and your firewall rule can match it. The value is shown once to the site's owner and can be changed at any time.
  • It never pretends to be someone else to get in. SEOmiseBot never claims to be Googlebot or a browser to get past a website's protection, and never solves CAPTCHAs or bot challenges. The one exception to the name is the MiseGuard cloaking check described above, on a customer's own verified site only.

Signed requests (Web Bot Auth)

Signed requests are not switched on yet. When they are, every request SEOmiseBot sends will carry the Signature-Agent, Signature-Input and Signature headers, and this section will give the key directory to verify them.

Allow it through your firewall

SEOmise reads your pages as SEOmiseBot and says so in every request. It never pretends to be a browser or a search engine, and never solves a challenge, so your firewall has to let it in. Each rule below matches SEOmiseBot only, so the rest of your protection stays on.

Cloudflare

  1. Sign in to Cloudflare and open the website's zone.
  2. Go to Security → WAF → Custom rules and choose "Create rule".
  3. Name it "Allow SEOmiseBot" and choose "Edit expression".
  4. Paste the secret header expression below, with the secret value from your Site health card.
  5. Set the action to "Skip" and tick: All remaining custom rules, All managed rules, All Super Bot Fight Mode rules, All rate limiting rules, and the product Browser Integrity Check (it answers "error code: 1010"). If you use Bot Fight Mode (free plans), it cannot be skipped by a rule: turn it off under Security → Bots, or use the secret header with a "Skip" rule on a paid plan.
  6. Place the rule first in the list and choose "Deploy".
Expression: user agent and your secret header
(http.user_agent contains "SEOmiseBot" and any(http.request.headers["x-seomise-verify"][*] eq "<your secret value>"))

Replace <your secret value> with the value from your Site health card.

Sucuri

  1. Sign in to the Sucuri dashboard and open the website's firewall.
  2. Go to Settings → Access Control → Allowlist IP addresses (the addresses are not published yet; ask Sucuri support to allowlist the user agent "SEOmiseBot" instead).
  3. Under Access Control → Allowlist user agents (or ask Sucuri support), allow the user agent containing "SEOmiseBot".
  4. Save, wait a minute.

Other firewalls and security plugins

  1. Open the firewall or security settings that protect the website: your hosting control panel, CDN or security plugin.
  2. Allow requests whose user agent contains "SEOmiseBot", or that carry the header X-SEOmise-Verify with your secret value.
  3. If the setting is a "bad bot" or "block unknown crawlers" switch, add SEOmiseBot to its exceptions instead of turning it off.
  4. Save.

The reader's outbound addresses are not published yet, so match the user agent together with the secret header below.

Controlling SEOmiseBot with robots.txt

SEOmiseBot obeys rules for any of these user-agent names, and for User-agent: * when none of them has a group:

  • SEOmiseBot
  • SEOmise
  • Visualised
  • VisualSEO

To block it completely:

User-agent: SEOmiseBot
Disallow: /

To keep it out of one folder and slow it down:

User-agent: SEOmiseBot
Disallow: /private/
Crawl-delay: 5

The research crawl, customer scans and link checks follow the robots.txt standard (RFC 9309): when your file has a group for SEOmiseBot, only that group applies and the * group is ignored; the longest matching rule wins, and Allow wins a tie. If your robots.txt cannot be read because of a server error, the research crawl reads nothing on your site until it can; if the file does not exist, it may crawl. It keeps a copy of your robots.txt for up to 24 hours. Customer scans read robots.txt at the start of each scan.

Single reads a person asks for that skip robots.txt

robots.txt is for crawlers that wander a site. A few reads are not crawling: a person asks for one address, usually on their own website, and gets an answer straight away. These read the address they asked for (redirects only within the same site) without checking robots.txt, still as SEOmiseBot (apart from the cloaking check described above):

  • Adding a website: its home page, once, to show its title and which software it runs on.
  • Uptime and website watch: the home page of a site its owner asked us to watch, every few minutes.
  • The AI content check: the one page of their own site a customer picks, when the scan has not read it; and the free public AI content check, the one address a visitor types.
  • The instant website check (free SEO audit): the one address typed, with its robots.txt (shown in the result), its sitemap and the status of up to 10 of its links.
  • The free tools (redirect checker, sitemap checker, meta tags checker, robots.txt tester): the one address a visitor types.
  • Speed checks: the home page and a few of the most visited pages of a customer's own site, with one stylesheet, script and picture.
  • Placement checks: the partners page of a publisher in the SEOmise network, which the publisher agreed to show.
  • The MiseGuard cloaking check: pages of a customer's own verified site, as described above.

To stop these too, block SEOmiseBot in your firewall, or ask us to exclude your domain with the form below.

Contact

If SEOmiseBot causes a problem on your site, e-mail [email protected] with your domain and the approximate time. We will look into it and can exclude your domain.

Opt out or ask us to remove data

Use this form to keep the research crawl away from your domain (an opt-out), or to have what it stored about your domain deleted (a takedown, which also opts you out). Both cover the domain's subdomains. If you can show that you control the domain, the request takes effect at once; otherwise our team reviews it.

How you will show that the domain is yours