Caute's fetcher
When someone saves a web page to Caute, follows a feed, or keeps the sources an assistant's answer cited, Caute's server fetches that page once, on their behalf, to keep a copy in their private library. This page says what it fetches, how to recognise it, and how to stop it.
What it fetches, and when
- A page someone saved. One request when they save it, and again only if they ask Caute to read it again.
- A feed someone follows. About once an hour, with
If-None-MatchandIf-Modified-Since, so an unchanged feed costs one short request. A feed that names a WebSub hub is subscribed to it, and a notification from the hub brings the next check forward; the hourly check never stops. - The pages an answer cited, when the person keeps them.
- A page someone watches. About once a week, with
If-None-MatchandIf-Modified-Since, so an unchanged page costs one short request. A person sets a watch on a page they already saved, one page at a time, and a change is kept as a new version of their copy. Nothing is watched that nobody asked for.
Caute does not crawl. It never follows links from a page it fetched, never fetches a page nobody asked for, and never uses what it fetches to train a model. What it keeps is for the person who saved it.
How to recognise it
Every request carries the User-Agent CauteBot/0.1 (+https://getcaute.com/bot) and is signed with
Web Bot Auth (HTTP Message Signatures for automated traffic):
Signature-Agent: sig1="https://getcaute.com"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=…;expires=…;keyid="…";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:…:
The public key is at https://getcaute.com/.well-known/http-message-signatures-directory, a JWK set served as
application/http-message-signatures-directory+json, itself signed. A request that carries this signature and
verifies against that key came from Caute. Webhook deliveries and assistant event deliveries are signed the same
way, beside their own Standard Webhooks signature.
Payment
Caute never pays a crawl toll. A site that answers 402 Payment Required is left alone: the person's card keeps
the address and title and says the site asks to be paid for automated reading, and the person can catch the page
from their own browser instead, which is them reading it, not a crawler.
Stopping it
Refuse the request (any 4xx) and the card says the site wouldn't give Caute the page; the person can still save
it from their browser. A 410 Gone on a feed tells Caute the feed is gone. Questions or a problem:
support@getcaute.com.