Delivery Failures

A webhook can fail to deliver messages if it receives a non-2xx status code for 10 consecutive delivery attempts. Each delivery attempt has an exponential backoff with a maximum period of one minute.

Failure Handling: When delivery fails, the webhook's status is changed to "failed." No further messages are delivered until the verification process is completed. However, messages are still added to the target's topic to prevent data loss when reconnecting.
Reconnection: Failed or active targets periodically receive a CRC request every 10 minutes. Upon successful verification, the target's status changes to "connected." A system message and backlogged messages will be sent to the target, followed by new messages as shown below.

{
    "protocol": "system",
    "event_type": "system.connection"
    "data": {
        "target_id": "ddfc2e56-f82c-4a24-975e-427230242476",
        "disconnect_time": "2023-07-29T17:37:38.242292Z",
        "connection_time": "2023-07-29T17:39:18.725464Z",
        "previous_status": "failed",
        "new_status": "connected"
    }
}

📘
Info:
The number of delivery attempts may be more than 10 in the event of a rolling upgrade of our services.

Message Drops

Each target’s topic has a maximum size that if exceeded because of slow processing or the target being unable to accept messages will result in older messages being dropped.

If messages are dropped, a notification is sent to the user as a "dropped message" event.

{
  "protocol": "system",
  "event_type": "system.dropped.messages",
  "data": {
    "dropped_msg_count": 1,
    "pending_msg_count": 100,
    "target_id": "85366adb-7550-400a-889f-3ae46f4b7668",
    "time": "2023-07-29T16:09:31.045872283Z"
  }
}

The current maximum number of messages is 10,000. The number of buffered messages per target can be controlled using the max_buffer_count parameter.