Azure 100 Message Send Limit With Transactions

Architecture & Design
,
1

Given that Azure Service Bus has a 100 message limit when sending messages with transactions enabled (ASB Quotas) and (Particular Discussion), is the Outbox a viable alternative?

If I enable Outbox (and configure transactions from sends-with-atomic-receives to receive-only), then I don’t have any trouble sending more than 100 messages outbound from a message handler.

Are there any downsides to this solution?

Thanks,
Mark

2

Hi Mark,

Which persistence are you thinking about using? I’m asking because some persisters have restrictions on how many outgoing transport operations they can handle when using the outbox.

Regards,
Daniel

3

Daniel,

We are using SQL Server persistence.

-Mark

4

Hi Mark,

Thanks for clarifying. I’m not aware of any restrictions with SQL P other than the usual resource utilization caveats when dealing with binary blobs, which is what is going to happen when using the outbox. So you should be able to handle more than 100 messages by using SQL P and the outbox.

Given that you are sending so many messages, I wonder what you are doing there? Are you trying to send a bulk up updates to some downstream system? What requirements do you have in terms of duplicates there? Depending on those requirements, you could think about moving that handler into a dedicated endpoint that is configured to use ReceiveOnly. Another option would be to still use sends with atomic receive but split into batches of 100 messages by having a technical message that is sent locally that then triggers those batches of 100 messages.

Regards,
Daniel

5

Daniel,

We are notifying external customers by email, sms, or voice that a certain event that impacts them has started within our organization. So a single NServiceBus event comes into a saga that indicates that this business event has started in our organization. We then gather up a list of customers that need to be notified and send individual commands to have each of these customers notified by their preferred notification method.

I considered writing, and prototyped, a handler that would send 95 (just under the 100 ASB limit) individual messages and then put the remaining customers to be notified into a command and send it back to the same handler. This worked but then it struck me that I could use the Outbox and do away with the custom logic.

-Mark

6

Hi Mark

Yes the outbox would work and make it possible to remove your custom logic.

Another option would be to do all that on a dedicated endpoint that has ReceiveOnly as a transaction set on the transport. Then that limit doesn’t apply there. Of course this solution wouldn’t give you the same benefits as the outbox from a reliability standpoint.

Difficult for me to say what would be the best option in your case.

Regards,
Daniel

7

While yes, outbox removes that limitation sending a notification to an unbounded set or destinations can be for many reasons problematic.

For example, with outbox dispatching the dispatching is based on the success of ALL messages in that outbox record. The more messages are in there the higher the risk that one of those message dispatch operations fails and the whole dispatching is ran again. When that happens a subset of these messages is send again and thus received/processed again on the target.

As this could be 1, 10, 100, 1000, 10.000s or even more I would highly recommend using an alternative approach that scaled better to prevent any issues in the future.

There are various chunking-based approaches that can work on an unbounded set.

– Ramon

8

Are there examples that can be linked to?

9

I would also appreciate some examples.

In my situation there is a saga running for each user in an application. For certain events these sagas need to be updated. So I need to iterate over each user and send a command to update each saga. Since there are more than 100 users registered, I can’t do this in a single handler with SendsWithAtomicReceive enabled because of the transaction limit.

Also is it possible to disable transactions for a single handler or even a single send operation via SendOptions? Because the endpoint itself should have SendsWithAtomicReceive enabled, but in this particular use case the messages are idempotent, so I could disable transactions without worrying about zombie messages.
Edit: I guess the RequireImmediateDispatch property on SendOptions would make this possible.

Does the transaction limit apply to transports other than ASB, such as RabbitMQ?