I am working on an issue similar to this (which is my post):
The summary is that I receive a single command which contains 10 or 50 or 500 or 5000 or 1,000,000 account numbers and I want to send a new command for each of those account numbers.
Because of limitations in Azure Service Bus batches (max 100 commands per batch, max 1 MB batch size), we need to chunk the original message. This presents its own issues because you can end up with too many chunks or a batch that is too large. 1,000,000 account numbers turns into 10,000 chunks (commands) which is too many commands to send in a single batch. Try to send 99 chunks (commands) with the remaining account numbers in the 100th chunk and the batch size is too large.
My solution is to have two endpoints, endpoint A and endpoint B. Endpoint A has the outbox turned on, outbox B does not use the outbox.
In endpoint A, all in the same message handler:
- The message handler receives a command, c1, containing 50,000 account numbers.
- The message handler sends a command c2 to itself for housekeeping purposes.
- The message handler sends 500 commands, c3, each containing 100 account numbers. These commands will be handled by endpoint B.
- The message handler exits.
In endpoint B:
- A message handler receives the 500 c3 commands (all at different times of course), each of which contains 100 account numbers. Endpoint b sends a command, c4, for every account number for a total of 50,000 commands. The message handler exits.
- A different message handler receives the 50,000 c4 commands and processes them successfully.
Now here is my confusion/problem: when I examine the outbox table in SQL Server, I see command c1 and c2 (both dispatched) but I do not see any of the 500 c3 commands. I know this is working as intended because if I turn off the outbox in endpoint A, I get an exception about too many messages in a batch. So why aren’t the 500 c3 commands in the outbox table in SQL Server?
Regards,
Mark