
For two weeks a small share of report jobs disappeared without an error.
The cause
The worker timeout was shorter than retry_after, so a slow job was picked up twice and the second copy failed silently.
The fix
timeout below retry_after
Idempotent job keys
An alert on failed_jobs growth