CoostiBot Information
Information about Coosti's web crawler
About CoostiBot
CoostiBot is Coosti's official web crawler that collects product information and prices from online retailers to provide consumers with the best price comparison experience.
We respect robots.txt and only crawl publicly accessible product pages.
Technical Details
User Agent
Mozilla/5.0 (compatible; CoostiBot/1.0; +https://coosti.dk/bot-info) Crawl Frequency
- Typically once daily per product
- Respects Crawl-delay in robots.txt
- Maximum 1 request per 2 seconds per domain
What do we crawl?
- Product pages (prices, availability)
What do we NOT crawl?
- Login pages or protected areas
- Personal user data
- Checkout process
- Pages marked with noindex or disallow in robots.txt
Verified IP Addresses
CoostiBot operates from the following verified IP addresses:
- 140.99.60.199
- 168.93.216.108
IP-liste (JSON):coostibot-ips.json
Company Information
Contact & Opt-out
Problems with our crawler?
If you experience problems with CoostiBot or have questions, please feel free to contact us:
Email: [email protected]
Don't want to be crawled?
We fully respect robots.txt. To block CoostiBot, add the following to your robots.txt:
User-agent: CoostiBot
Disallow: / Alternatively, you can contact us at [email protected] for manual removal.
Merchant Partnership
Interested in becoming an official partner and gaining better visibility on Coosti? Contact us at [email protected]
Compliance
CoostiBot complies with all relevant laws and standards:
- Robots Exclusion Protocol (robots.txt)
- GDPR (we do not collect personal data)
- Respects rate limits and Crawl-delay
- Clearly identifies itself with user agent