RabbitMQ Inaccurate Delayed Message Processing

Hi all,

In our current production setup, we are managing over 100,000 delayed messages with varying delay intervals ranging from 2 weeks to 6 or 7 months. Unfortunately, we have observed that certain messages, which should be processed in the future (e.g., 6 months from now), are being executed prematurely. This behavior is unexpected and inconsistent with the intended delay interval.

We would like to inquire whether the tool has any known limitations regarding the handling of a higher number of delayed messages. We came across a related issue in RabbitMQ (Delay interval predictability · Issue #72 · rabbitmq/rabbitmq-delayed-message-exchange · GitHub), which raised concerns about performance and scalability under similar circumstances.

It is worth noting that we started encountering this issue a couple of months after transitioning our infrastructure and migrating to a RabbitMQ Cluster with three nodes from a single RabbitMQ server without cluster configuration. Additionally, the number of delayed messages we are now managing has significantly increased since the migration.

Our primary question is whether this issue is known or if other users have encountered a similar problem. We would greatly appreciate any insights, information, or guidance you can provide to help us address this challenge.

Here are some additional details about our environment:

  • NServiceBus version: 6.1.1
  • RabbitMQ version: 3.8.14
  • Erlang version: 22.2.7

This is crazy. Same feeling you have when using DTC that works, until it doesn’t work :frowning:

Hi @Parkaran,

The version of NServiceBus and the corresponding RabbitMQ transport packages you’re using have been out of support since 2017. RabbitMQ 3.8 is also out of support for about a year. In the meantime, there we plenty of bug fixes to RabbitMQ and our RabbitMQ transport packages, some of them directly related to delayed messages, so I’d strongly recommend to start with upgrading your dependencies to the latest supported versions. Here are some links to our upgrade guides that should help with that process:

NServiceBus delayed messages don’t use the RabbitMQ Delayed Message Plugin. You can read more about the implementation of delayed delivery with NServiceBus here: RabbitMQ Delayed Delivery • RabbitMQ Transport • Particular Docs. Please note, that when running on RabbitMQ clusters, it’s strongly recommended to switch to quorum queues to prevent accidental message loss due to RabbitMQ cluster node failures. You can also read more about this here: Migrating RabbitMQ from classic to quorum queues • RabbitMQ Transport • Particular Docs

If you continue observing inaccurate delayed message processing after the upgrade, opening a support case might be best.