I am working on an issue similar to this (which is my post):
The summary is that I receive a single command which contains 10 or 50 or 500 or 5000 or 1,000,000 account numbers and I want to send a new command for each of those account numbers.
Because of limitations in Azure Service Bus batches (max 100 commands per batch, max 1 MB batch size), we need to chunk the original message. This presents its own issues because you can end up with too many chunks or a batch that is too large. 1,000,000 account numbers turns into 10,000 chunks (commands) which is too many commands to send in a single batch. Try to send 99 chunks (commands) with the remaining account numbers in the 100th chunk and the batch size is too large.
My solution is to have two endpoints, endpoint A and endpoint B. Endpoint A has the outbox turned on, outbox B does not use the outbox.
In endpoint A, all in the same message handler:
The message handler receives a command, c1, containing 50,000 account numbers.
The message handler sends a command c2 to itself for housekeeping purposes.
The message handler sends 500 commands, c3, each containing 100 account numbers. These commands will be handled by endpoint B.
The message handler exits.
In endpoint B:
A message handler receives the 500 c3 commands (all at different times of course), each of which contains 100 account numbers. Endpoint b sends a command, c4, for every account number for a total of 50,000 commands. The message handler exits.
A different message handler receives the 50,000 c4 commands and processes them successfully.
Now here is my confusion/problem: when I examine the outbox table in SQL Server, I see command c1 and c2 (both dispatched) but I do not see any of the 500 c3 commands. I know this is working as intended because if I turn off the outbox in endpoint A, I get an exception about too many messages in a batch. So why aren’t the 500 c3 commands in the outbox table in SQL Server?
That does make sense, but I do have a follow-up question: How are the 500 c3 commands dispatched from endpoint A? If I run this scenario with the outbox OFF in endpoint A, I get an exception that there are too many messages in the Azure Service Bus batch. However, when I run this scenario with the outbox ON in endpoint A, the 500 messages get delivered successfully.
When the outbox is enabled, NServiceBus will use what we call immediate dispatch to ensure that the outgoing messages are not enlisted in the receive transaction of the message being processed, since that could cause message loss should the queuing system rollback after we have marked the outbox record as processed.
When immediate dispatch is used, transports must send messages immediately, preventing batching. This is the behavior you are seeing.
We’ve been discussing this internally and have concluded that an isolated dispatch is unnecessary, since the outbox already forces the transport to run in receive-only mode, which prevents outgoing messages from enlisting in the current receive transaction. We have raised a bug to change this and thereby allow transports capable of batching (ASB and SQS at the time of writing) to benefit from it Outbox prevents transports from batching outgoing messages · Issue #7509 · Particular/NServiceBus · GitHub