Crawler information
SEOmiseBot
SEOmiseBot is the crawler of SEOmise, a tool that helps businesses improve how their websites appear in search. It reads public web pages the way a search engine does and obeys robots.txt.
User agent
SEOmiseBot/1.0 (+https://seomise.com/bot)
Why it visits
- Audits for customers. Most visits happen because someone added a website to a project on SEOmise: usually its owner, their agency, or a business comparing itself with a competitor.
- A small research crawl (pilot). SEOmise is testing a crawl of public pages in Ireland and the United Kingdom, to build its own view of how sites link to each other. It visits domains our team chooses and the competitor sites of customers who have switched this on. It is small: at most 300 addresses a day across all sites.
What it collects
- Facts about each page: its status code, title, description, headings, language, word, image and link counts, the types of structured data it uses, and the addresses of links to other sites.
- It does not keep the page's HTML or its text, does not submit forms, log in or run checkout flows, and does not try to reach private network addresses.
- What the research crawl finds is used inside SEOmise. It is never published, and it is not shown to anyone who is not a customer.
Crawl rate
Customer audits make at most five requests at a time to one site, fewer when a site responds slowly. The research crawl makes one request at a time and honours Crawl-delay up to 60 seconds (customer audits up to 10 seconds). When your server answers 429 Too Many Requests or 503 Service Unavailable, the research crawl stops visiting your site for the time given in Retry-After, or for 15 minutes when there is none.
Controlling SEOmiseBot with robots.txt
SEOmiseBot obeys rules for User-agent: SEOmiseBot and for User-agent: *. To block it completely:
User-agent: SEOmiseBot Disallow: /
To keep it out of one folder and slow it down:
User-agent: SEOmiseBot Disallow: /private/ Crawl-delay: 5
The research crawl follows the robots.txt standard (RFC 9309): when your file has a group for SEOmiseBot, only that group applies and the * group is ignored; the longest matching rule wins, and Allow wins a tie. If your robots.txt cannot be read because of a server error, it crawls nothing on your site until it can; if the file does not exist, it may crawl. It keeps a copy of your robots.txt for up to 24 hours. Customer audits read robots.txt at the start of each scan.
Contact
If SEOmiseBot causes a problem on your site, e-mail [email protected] with your domain and the approximate time. We will look into it and can exclude your domain.
Opt out or ask us to remove data
Use this form to keep the research crawl away from your domain (an opt-out), or to have what it stored about your domain deleted (a takedown, which also opts you out). Both cover the domain's subdomains. If you can show that you control the domain, the request takes effect at once; otherwise our team reviews it.
