CarHindsightBot

What it is for

CarHindsight shows a used car's adverts, prices and mileage over time, by registration, with a link to each dealer's own advert. To do that, the crawler reads the public pages of used cars for sale on dealers' websites: the price, the mileage, the registration or VIN, the car's details and the links to its photos. It keeps small copies of the first two photos so a car's history still has a picture after the advert has gone.

It reads nothing else. It doesn't open customer areas, fill in forms, read pages about people, or collect contact details, and nothing it reads is used to train artificial intelligence.

How to tell it is ours

• It identifies itself on every request with the user agent CarHindsightBot/0.1 (+https://carhindsight.co.uk/bot). In robots.txt its name is CarHindsightBot.
• It crawls only from the addresses listed at /bot/ips.json. A request with this user agent from any other address isn't ours.

How it behaves

• It reads a site's robots.txt before anything else and does what it says, including Crawl-delay. It reads nothing from a site whose robots.txt can't be read, and reads robots.txt again every day.
• It waits at least 3 seconds between requests to the same site, longer if robots.txt asks, and reads one page at a time from a site.
• It slows down and comes back later when a site answers "too many requests" or is busy (429, 502, 503, 504). It never tries to get past a bot check or a captcha: a site that turns it away is left alone.
• It reads a dealer's list of cars for sale about every 6 hours and each car's page about every 12 hours, more often for a few days after a change.

How to limit or stop it

Add these lines to your robots.txt to keep it out altogether. It notices within a day.

User-agent: CarHindsightBot
Disallow: /

Or keep it out of part of the site, or slow it down:

User-agent: CarHindsightBot
Disallow: /search
Crawl-delay: 10

You can also just tell us through our contact page, and we'll switch the crawler off for your site. If you would rather your cars weren't shown on CarHindsight at all, say so and we'll take them down.

Would you rather send us your stock?

If your website or stock system can send your cars to advertising sites, it can send them to us too, which spares your site the visits and keeps your prices up to date within the hour. Get in touch through our contact page and we'll set it up with your provider.

Who runs it

CarHindsight. The crawler's code is the packages/crawler part of CarHindsight. Our privacy notice says what we hold and why.