Azure Service Bus, Large Number of Commands, Outbox

I am working on an issue similar to this (which is my post):

The summary is that I receive a single command which contains 10 or 50 or 500 or 5000 or 1,000,000 account numbers and I want to send a new command for each of those account numbers.

Because of limitations in Azure Service Bus batches (max 100 commands per batch, max 1 MB batch size), we need to chunk the original message. This presents its own issues because you can end up with too many chunks or a batch that is too large. 1,000,000 account numbers turns into 10,000 chunks (commands) which is too many commands to send in a single batch. Try to send 99 chunks (commands) with the remaining account numbers in the 100th chunk and the batch size is too large.

My solution is to have two endpoints, endpoint A and endpoint B. Endpoint A has the outbox turned on, outbox B does not use the outbox.

In endpoint A, all in the same message handler:

  • The message handler receives a command, c1, containing 50,000 account numbers.
  • The message handler sends a command c2 to itself for housekeeping purposes.
  • The message handler sends 500 commands, c3, each containing 100 account numbers. These commands will be handled by endpoint B.
  • The message handler exits.

In endpoint B:

  • A message handler receives the 500 c3 commands (all at different times of course), each of which contains 100 account numbers. Endpoint b sends a command, c4, for every account number for a total of 50,000 commands. The message handler exits.
  • A different message handler receives the 50,000 c4 commands and processes them successfully.

Now here is my confusion/problem: when I examine the outbox table in SQL Server, I see command c1 and c2 (both dispatched) but I do not see any of the 500 c3 commands. I know this is working as intended because if I turn off the outbox in endpoint A, I get an exception about too many messages in a batch. So why aren’t the 500 c3 commands in the outbox table in SQL Server?

Regards,

Mark

Hi Mark

The outbox will create one row in the outbox per incoming message, so in your case:

  • C1 will create a row in the outbox with 501 messages (C2 + the 500 C3s)
  • C2 is processed by endpoint A and will generate a row in the outbox
  • The 500 C3 commands are processed by Endpoint B, which doesn’t have the outbox enabled and therefore will not generate any outbox rows in the database

In short: Only C1 and C2 are visible as rows in the database since they are the messages processed by the endpoint with outbox enabled

Does this make sense?

/Andreas

Andreas,

That does make sense, but I do have a follow-up question: How are the 500 c3 commands dispatched from endpoint A? If I run this scenario with the outbox OFF in endpoint A, I get an exception that there are too many messages in the Azure Service Bus batch. However, when I run this scenario with the outbox ON in endpoint A, the 500 messages get delivered successfully.

Regards,

Mark

Hi!

When the outbox is enabled, NServiceBus will use what we call immediate dispatch to ensure that the outgoing messages are not enlisted in the receive transaction of the message being processed, since that could cause message loss should the queuing system rollback after we have marked the outbox record as processed.

When immediate dispatch is used, transports must send messages immediately, preventing batching. This is the behavior you are seeing.

Hope this helps.

Andreas