Let SEOmise reach a site behind Cloudflare

SEOmise reads your pages as SEOmiseBot and says so in every request. It never pretends to be a browser or a search engine, and never solves a "Verify you are human" challenge. If your site's firewall or bot protection does not know SEOmiseBot, it may answer with a challenge page ("Just a moment…") or a refusal instead of your page. Your firewall has to let SEOmise in.

How you know

SEOmise recognises the answers of Cloudflare, Sucuri, Akamai, Imperva (Incapsula), DataDome, HUMAN (PerimeterX), DDoS-Guard and pages titled as a bot check. When one of them answers:

• Site health: the page is listed as Not checked: Cloudflare blocked our crawler (HTTP 403) (with the firewall's name and code). It is not a broken page, not in the errors and not in the health score, and the challenge page is not checked for SEO.
• Uptime (Website watch, MiseGuard → Uptime): the check is Reachable, but blocking SEOmise. The site counts as available, and no problem or alert is raised. A real error from your own server behind the firewall (such as a 503 with your site's own error page) is still an outage.
• A card "Cloudflare is blocking SEOmise" (or Sucuri, or the firewall found) appears at the top of Site health, on the uptime screen and on the first scan step of set-up. It shows while the last block is recent (14 days by default) and no read has got through since.

What the card shows

1. Why: one line on how SEOmise reads your site.
2. The steps for your firewall, numbered:
   • Cloudflare: Security → WAF → Custom rules → Create rule, with the expression to paste and the action Skip (all remaining custom rules, managed rules, Super Bot Fight Mode and rate limiting rules), placed first, then Deploy. Bot Fight Mode on free plans cannot be skipped by a rule: turn it off under Security → Bots, or use the secret header with a Skip rule on a paid plan.
   • Sucuri: Settings → Access Control: allowlist SEOmise's addresses (when published) and the user agent containing "SEOmiseBot".
   • Akamai, Imperva, DataDome, HUMAN and DDoS-Guard: an allow rule (bot management exception) for the user agent and the addresses or the secret header, for these requests only.
   • Any other firewall, CDN, host or security plugin: allow the user agent "SEOmiseBot", or add it to a "block unknown crawlers" switch's exceptions instead of turning the switch off.
3. The expression to copy (Cloudflare), in a box with a Copy button: one matching the user agent and SEOmise's published addresses, and one matching the user agent and your secret header.
4. SEOmise's addresses: the published outbound addresses, or a line saying they are not published yet (then match the user agent together with the secret header). The addresses are also on the SEOmiseBot page.

None of the steps weakens your protection for anyone else: each rule matches SEOmise's requests only.

The optional secret header

Matching a secret is the most precise rule, especially while SEOmise's addresses are not published.

• Press Create secret. SEOmise makes a random value and shows it once, with a Copy button and a warning that it will not be shown again. The Cloudflare expression on the card is filled in with it, ready to paste.
• From then on SEOmise sends the header X-SEOmise-Verify with that value, but only to this website and only for your workspace. It is stored encrypted and is never shown again: the card shows •••• set on the date.
• Rotate makes a new value at once (the old one stops working: update your firewall rule straight away). Remove stops sending the header.
• Only people who manage the project can create, rotate or remove it.

Check again

After you change your firewall, press Check again on the card. SEOmise reads your start page once, the way it always does, and tells you the result:

• Still blocking: the firewall answered again; check the rule (wait a minute after deploying it).
• Reached the site: the card goes away, and the next scan reads your pages. Scan again in Site health to read them now.
• Another answer: the site answered with a code that is not a firewall SEOmise recognises (a security plugin or your host may block unknown visitors; allow SEOmiseBot there).

Check again follows redirects only on your own site and can be pressed a few times a minute.

Let SEOmise create the rule in Cloudflare

If your site is behind Cloudflare and you connected it to CDN and edge with a separate write token that has Zone → Zone WAF → Edit, the card offers Create the allow rule in Cloudflare for me. SEOmise never asks for your Global API key.

1. The card first shows exactly the rule it would create: its name (Allow SEOmiseBot (created by SEOmise)), the action Skip, its place (first in your custom rules), what it matches, what it skips, and the request sent to Cloudflare, with your secret shown as a placeholder.
2. It matches requests whose user agent contains SEOmiseBot and that come from SEOmise's published addresses or carry your secret header. It is never made on the user agent alone, which anyone could copy: while the addresses are not published, create the secret header first.
3. It skips all remaining custom rules, all rate limiting rules, all Super Bot Fight Mode rules and all managed rules, for SEOmise's requests only. Everyone else meets your protection as before.
4. Press the button, then confirm in the window that opens. Nothing changes in Cloudflare before that. If you may approve your own CDN changes (the workspace owner), your confirmation is the approval; otherwise the request waits for the owner under CDN and edge → Changes.
5. SEOmise adds the rule, reads your rules back to check it, and the card shows the zone, the ruleset and rule ids, who asked for it and when. Remove the allow rule deletes it again and puts your zone back as it was.

Good to know:

• Your secret header value is part of the rule's expression, so it is sent to Cloudflare and kept in your zone. SEOmise does not show it again and never writes it to its own logs. If you rotate or remove the secret, the card tells you to remove the rule and create it again.
• Bot Fight Mode (Cloudflare's free plan) cannot be skipped by any rule. If it is on, turn it off under Security → Bots, or use Super Bot Fight Mode on a paid plan, which the rule skips.
• If Cloudflare refuses the rule (for example the zone's plan does not offer one of the protections it skips, or the token lacks the permission), the card shows Cloudflare's refusal in plain words. Nothing was changed and the manual steps above still work.
• Without CDN and edge, or with a write token that cannot change firewall rules, the card links to CDN and edge to connect one, and the manual steps stay.
• Staff choose under Settings → Policies → Crawl control whether the card offers this (on by default), what the rule skips (the remaining custom rules and the three protections above by default; older products such as Browser Integrity Check only when named), its name in Cloudflare, its risk score and how many creations and removals a person may make an hour.

Sites with the SEOmise connector

When your website has the SEOmise connector (the WordPress plugin 0.52.0, the Laravel package 0.42.0 or the PHP connector 0.36.0, or later) and it may read your site, you do not have to wait for the firewall rule: during a scan, the pages your firewall blocks are read through the connector automatically. The connector asks your own server for the page and sends it to SEOmise, signed, and the page is checked like every other page: its title, description, headings, pictures and links, and the pages it links to are scanned too.

• In Site health such a page is marked Read through the site connector, and the scan panel counts them.
• Only pages of your own website (with or without www), only HTML pages, up to 2 MB each and at most 60 a minute. A redirect to another website is never followed, and SEOmise checks the answer again before it reads it.
• Up to 200 pages a scan; any others stay Not checked: blocked by the firewall.
• The card still shows, because the firewall still blocks SEOmise's crawler. Allowing SEOmiseBot is still the way for every page and every uptime check to reach your site.
• Shopify, Wix and Webflow do not let an app read pages from inside the site: there, allow SEOmiseBot as above.

Settings (staff)

Staff can change these under Settings → Policies → Crawl control ("Crawler identity and firewalls"): whether firewall blocks are recognised, the header's name, whether workspaces may set a secret header, the published addresses, how long the card shows, how many Check again presses a minute and secret changes an hour are allowed, whether uptime counts a plain Cloudflare 503 as a block (off by default: only a 503 that carries Cloudflare's challenge does), whether blocked pages are read through the site's connector (on by default) and how many pages a scan may read that way (200 by default).