Currently in a few handlers we run a handful of calculations, call some external asynchronous APIs, and then send off a large chunk of messages after we’ve finished all of our computations. We’re currently using Azure Service Bus, so when we are sending >100 messages we have to send them with the options of RequireImmediateDispatch. This all works but we’re seeing a few issues related to locks being lost and we’re assuming it’s just our handlers taking too long so we’ve been trying to make them more efficient.
When send operations occur inside a handler the default behavior is to add the operation to a batch, which will be executed after the handler function returns. Unless the send options request for immediate dispatch, in such a case the execution is immediate. Therefore using an await on the Send() operation has a negative impact when used on an immediate dispatch, but has no impact on the default settings. It is advised to always return the task instead of awaiting the task never to be impacted by this difference in behavior, which might lead to subtle issues over time. To learn more about batched dispatch refer to the Batched message dispatch article.`
I’m wondering how it would be recommended to send all of these messages? Right now the handler method is “async” to make use of await in few other places, (like external APIs).
Right now in our code we’ve created an extension method:
public static Task SendAsInternalCommands<T>(this List<T> commands, IMessageHandlerContext context)
{
var options = new SendOptions();
// When the large commands are broken down in to smaller ones, we exceed the 100 message batchs size limit.
options.RequireImmediateDispatch();
return Task.WhenAll(commands.Select(command => context.Send(command, options)));
}
and then when we create all the commands we need to send we do:
You are being hit by a limitation of ASB in your current transaction mode. I think that if you go to ReceiveOnly that you do not have this limitation. However, you do have potential duplicates.
Consider using the outbox + receiveonly
How many messages are we talking about? You state more than 100 but what can be the upper limit? Maybe you can define chunks where you have some orchestration on tracking the current chunk? Also, how large are your messages?
Maybe even a different proposal as I don’t know anything about your current design and the process you are creating. why do you need to send that many messages in the first place at the end as that seems very batch oriented solution to be forced into messaging. Can’t you define a logical event at the end of the calculations where now a receiver has the ability to discover all calculation results?
Another thing that you might be doing is that you are ‘routing’ results to many different recipients. If you have this requirement at least have this completely isolated in its own handler.
Your extension method looks fine. The only thing that I would change is to inline create the send options per command. SendOptions are not supposed to be reused between multiple sends.
I created a sample that shows the approach
Do you know in which scenarios you see the lock lost exceptions? Are you sure the lock losts are from the message handler that sends out those commands? In my example I can send out 1000 messages in a few seconds without problems.
The thing you have to be aware of like Ramon mentioned is that with immediate dispatch what can happen is that at any point in time when you loose the connection to the broker the message could roll back and the batch could start from scratch. So the component that handles those commands has to deal with duplicates
I’ll look into changing it so we initialize a new SendOptions per command.
And yeah that’s our problem, we don’t have a great solution in place to handle duplicate messages so I think that might be one of our next steps to handle that.
I haven’t looked at it yet but will look into it now as it’s not a feature I’m super familiar with. Will certainly be messing around with it to see if it’ll help. Thanks!