Sending a notification is not always a single, reliable operation. Your application may lose its network connection, a provider may return a temporary error, or a worker may crash immediately after submitting a request. Retrying is essential for delivery reliability, but an unsafe retry can send the same notification more than once.
The solution is to separate two related problems: deciding whether a request should be tried again, and ensuring that a retry represents the same logical send rather than a new notification. This article presents a practical design for server-side notification retries using stable job identity, idempotency, controlled backoff, and clear operational state.
Model a notification as a durable job
Do not retry directly inside the HTTP request that triggered the notification. A request handler may time out, be restarted, or be invoked again by a client. Instead, record a notification job in durable storage and let a worker process it.
A useful job normally contains:
- A unique application-level job ID
- The notification title, body, and target
- The current state, such as pending, sending, succeeded, failed, or retrying
- The number of attempts
- The next time at which the job may be attempted
- A stable idempotency key
- The latest provider response or error classification
- Timestamps for creation, attempts, success, and final failure
The job ID and idempotency key should be generated when the logical notification is created, not each time a worker picks it up. If a worker crashes and another worker claims the job, both workers must use the same identity for the same logical send.
This design also gives product teams a way to define what “sent” means. A successfully accepted request is not necessarily the same as a notification being displayed on a device. Your records should distinguish submission to the notification service from later client-side handling where that distinction matters.
Use a stable idempotency key
For HoneyNotify server-side sends, the request is made with POST /v1/notifications, a Bearer API key, and an Idempotency-Key. A notification requires a title, body, and target. Targets can address a device, user, tag, segment, or all enabled devices.
The idempotency key should remain unchanged for every retry of one logical notification. It should not be based on the attempt number, current timestamp, or a freshly generated random value. Changing it tells the receiving service that the retry is a different operation, which removes the protection you intended to create.
A practical relationship is:
- One business event creates one notification job
- One notification job has one stable idempotency key
- Every HTTP attempt for that job sends the same idempotency key
- A genuinely new business event receives a new job and a new key
Use an identifier that is unique within the scope required by the provider. If your system can accidentally create two jobs for the same event, use a database uniqueness constraint or an equivalent deduplication rule before placing work on the queue.
You should also confirm the provider’s current documentation for how long idempotency keys are retained, which responses are replayed, and what happens if the same key is reused with a different request body. Your application should treat changing the payload while reusing a key as a data error, not as a recovery strategy.
Classify failures before retrying
Not every failure is recoverable. A retry policy should classify the result rather than retrying every non-success response.
Usually retryable failures
These often indicate that a later attempt may succeed:
- A connection could not be established
- The request timed out before a definitive response was received
- A temporary DNS or network failure occurred
- The service returned a transient server error
- Rate limiting was reported, especially when a retry delay was provided
The exact status codes and response fields must come from the current HoneyNotify documentation and your HTTP client’s behaviour. A timeout deserves particular care: the request may have reached the service even though your worker did not receive the response. Retry it with the same idempotency key, not a new one.
Usually non-retryable failures
These normally require correction rather than repetition:
- Invalid authentication or an expired API key
- A malformed request
- Missing title, body, or target
- A target that your application knows is invalid
- A policy or permission failure
Repeatedly retrying these errors increases load and delays diagnosis. Mark the job as failed, record enough context to investigate, and alert on the underlying configuration or data problem.
Ambiguous outcomes
The most important case is neither clear success nor clear failure. For example, a worker may submit a request and then lose its connection before reading the response. The service may have accepted the notification.
Do not create a second idempotency key in this situation. Keep the original key and retry according to the provider’s idempotency rules. If the provider confirms the original operation, your worker can record success. If it returns a definitive failure, apply the appropriate failure policy.
Build a bounded retry schedule
Retries should be delayed and limited. Immediate repeated attempts can amplify an outage, exhaust worker capacity, and turn a temporary provider problem into a longer incident.
A common schedule uses exponential backoff with jitter:
- Start with a short delay after the first transient failure
- Increase the delay after each subsequent failure
- Add random jitter so many workers do not retry simultaneously
- Stop after a maximum number of attempts or a maximum elapsed time
- Move exhausted jobs to a reviewable failure state or dead-letter queue
The values should reflect your product’s urgency and the provider’s documented limits. A time-sensitive security alert may have a shorter overall window than a weekly digest, but both still need a cap. Honour any retry-after guidance returned by the service where your integration supports it.
Do not let every retry extend forever. An unbounded notification job can remain active after the underlying user action is no longer relevant. Define what the product should do when the delivery window expires: discard it, mark it as missed, or expose it for manual investigation.
Make workers safe under concurrency
A queue may deliver the same job to two workers, particularly after a visibility timeout or worker crash. Idempotency protects the external operation, but your own state transitions must also be safe.
Use a lease, lock, or atomic claim so only one worker normally processes a job at a time. Set the lease duration longer than the expected request time, and renew it where your queue design requires. If a worker loses its lease, it must not continue making attempts merely because it still has the job in memory.
Update state with conditional writes. For example, a worker should only change a pending or retrying job to sending if the stored version is still the version it claimed. Record the attempt before or alongside the outbound request according to your recovery model, and make sure a crash cannot cause the scheduler to create a new identity.
This does not eliminate every race. It makes races visible and limits their effect. The provider-side idempotency key remains the final protection when two workers submit the same logical operation.
Keep client behaviour idempotent too
Server-side deduplication does not prevent your application from showing duplicate results on a device. A client may receive a notification more than once through a lifecycle event, a delayed delivery, or application logic that processes the same payload repeatedly.
The client SDK responsibilities include device registration, identity, token lifecycle, payload handling, and lifecycle events across iOS, Android, and Web Push. Build application handling so a repeated event does not repeat an irreversible action.
Useful techniques include:
- Put a stable event or notification identifier in application data where your payload design permits it
- Store recently processed identifiers for actions that must happen once
- Make navigation, database writes, and reward claims idempotent
- Separate displaying a notification from executing its associated business action
- Test foreground, background, offline, and app-restart behaviour on each supported platform
A notification is a message, not a transaction boundary. If tapping it confirms a payment, grants access, or changes an order, the server must validate that action independently and safely handle repeated requests.
Handle device and provider identity changes
Retries can expose identity problems that look like delivery failures. A device token may rotate, an application environment may change, or a user may sign out and another user may sign in on the same device.
Keep registration and identity updates separate from notification retry logic. Ensure that your client integration updates token lifecycle changes and that your server does not treat an old device identifier as permanently valid.
If you import OneSignal subscription IDs, they can remain as device IDs when matching provider tokens later register. However, token rotation or another provider environment can prevent matching. Treat an unmatched imported identifier as a migration and registration issue to investigate, not as a reason to send the same notification repeatedly to broader targets.
Measure the retry system
Observability should answer three questions: what happened, why was it retried, and whether the user could see more than one notification?
Record structured fields such as the job ID, idempotency key, target type, attempt number, outcome category, provider response classification, delay, and final state. Avoid logging API keys or sensitive payload content.
Useful alerts include:
- A rising rate of ambiguous timeouts
- Jobs approaching their retry limit
- A growing queue age
- Authentication or validation failures
- Duplicate application job creation
- Unexpected changes in success and failure distributions
Do not count every HTTP attempt as a separate product notification. Report logical jobs separately from attempts. This distinction prevents a temporary outage from making delivery statistics appear better or worse than they are.
Common mistakes
- Generating a new idempotency key for every retry
- Retrying authentication, validation, or permission errors indefinitely
- Treating a timeout as proof that no notification was accepted
- Sending from a web request without a durable job record
- Allowing multiple workers to update one job without conditional state changes
- Reusing a key for a changed title, body, or target
- Ignoring rate limits and retry-after guidance
- Assuming server-side deduplication makes client actions safe to repeat
- Keeping stale device registrations forever
- Omitting a final failure state and leaving jobs stuck in sending
Conclusion
Reliable notification retries require more than a loop around an HTTP request. Create a durable job, assign one stable idempotency key to the logical send, classify failures, use bounded backoff, and make workers and client actions safe under repetition.
For HoneyNotify integrations, keep the original Idempotency-Key when recovering from timeouts or transient failures against POST /v1/notifications. Verify the provider’s current idempotency and response rules, test worker crashes and token changes, and measure logical notifications separately from transport attempts. With those controls in place, retries can improve reliability without turning uncertainty into duplicate messages.