Heartbeat
Cron and background job monitoring
The other way round: your job calls us.
What it is for
Everything else here goes and looks. This one waits. Your backup, your nightly import, your queue worker calls a URL when it finishes, and we alert when a call does not arrive.
It is the only way to monitor something with no address to check. A backup job that stopped running three weeks ago looks exactly like one that ran perfectly, from outside, until the day somebody needs the backup.
Like every check here, it runs from Amsterdam, New York and Singapore, and a failure is confirmed from a second continent before anybody is woken. How that works.
What you can set
Everything below is on the form when you add one, on every plan.
- How often it runs
- Not how often we look: how often your job should call. A weekly backup is an ordinary thing to own, so this one goes out to a month where the others stop at a day.
- Allowed lateness
- How many minutes late the call may be. A job that runs hourly and takes up to twenty minutes can be allowed thirty. Blank is a tenth of the period.
- Ping URL
- Generated when you create the monitor, and shown once. Call it from the end of your job. It can be regenerated, which retires the old one.
Questions
Is this the same as a dead man's switch?
It is exactly that, applied to cron job monitoring. Everything else on this site goes and looks at something. This one waits to be told, and alerts on the silence, which is the only way to watch a job that has no address to check and no page to read.
How late can my job be before you alert?
As late as you allow, in minutes. A job that runs every hour and takes anywhere from one to twenty minutes can be allowed thirty, so a call is expected between sixty and ninety minutes after the last one. Left blank it is a tenth of the period it runs on, and never less than a minute: two minutes late means nothing to a nightly backup and everything to a job that runs every five. The floor is there because a cron job firing a few seconds either side of the minute is normal and must never open an incident.
How do I call it from a cron job or a shell script?
Fetch the URL at the end of the job, with curl or wget or anything else that speaks HTTP. GET, POST and HEAD are all accepted, because half the world reaches for one and half for another, and the answer is an empty 204 so that a nightly job does not mail a page of HTML to the operator.
Should I call it at the start or at the end?
At the end, after the work succeeded and not before it starts. A call at the top of the script says the machine woke up, which is rarely the thing that goes wrong. Put it on the last line, behind whatever exit code check the script already has, and the silence then means the job failed rather than that the host is down.
What if the URL leaks?
The URL is the credential, which is why it is 256 bits of randomness and why it is shown to you once. Anybody who has it can report your job as healthy, which is the failure worth worrying about rather than anybody reading anything. Regenerate it and the old one stops working immediately.
Can a job that runs weekly use this?
Yes, and it is one of the better reasons to. This interval goes out to a month where the other check types stop at a day, because a weekly backup is an ordinary thing to own and a backup that quietly stopped running three weeks ago looks identical to one that ran perfectly, right up to the day somebody needs it.