I encountered some odd behaviour with NSB when one endpoint listens for the same event with multiple internal handlers. When publishing the event, it would appear that if any handlers of that message encounter an error, all the other handlers of the same message fail to receive the event.
For example:
I have an assembly that takes the message “OrderReceived” and implements it in two handlers: GoodHandler and BadHandler. If I publish my event, both handlers fire correctly. However, if I force BadHandler to throw an exception, GoodHandler is never triggered. The message sits in the queue and is never delivered.
If I restart the endpoint, GoodHandler now runs and completes successfully. When BadHandler kicks in a few seconds later through the error retry mechanism in NSB, it throws another exception, which stops messages arriving again.
To replicate the issue, try downloading the Pub/Sub sample, copy the event handler and manually throw an exception in the second handler. Set a breakpoint or log a message in the first handler and notice it never fires.
My guess is that it might be unusual to have two handlers of the same event in the same assembly? In our real world case we have an event published that several aggregates in the domain are interested in. Once they receive the event being published, they kick off some internal tasks to bring their aggregates up to date.
Are we using the event handlers in a way that is not expected? If multiple handlers listen for the same event in one endpoint, is this expected behaviour?
That’s absolutely expected behavior. Multiple handlers within the same endpoint will all be executed within one transaction (what that means depends on your configuration) and if any exception is thrown, the message (there is only one, as the endpoint is the subscriber, not the message handler) is not successfully processed.
If you want to have independent subscribers, then you need them separated in multiple endpoints, so that each endpoint (endpoint is the subscriber) will receive its own copy of the event, so that both can live their own lifetime disconnected from each other.
That makes a lot of sense - I definitely forgot that it’s the endpoint subscribing to the publishers, not the handlers. Such an easy but important point to miss!
Another solution is to just after receiving the event to send several commands to split up the work into isolated messages. Then you have event handlers that spit out work items.
@DavidBoike , problem is if the first handler fails the retry policy on retries the first handler not the second one.
Example:
I have one sender and two subscribers to the senders. first subscriber throws an exception, NSB correctly retries the message, only the first subscriber gets the message and not the second one.
If you have 2 handlers in 1 endpoint, then there is only 1 subscription. Only 1 message will be delivered, which will invoke 2 handlers. If the first handler fails, the 2nd will not get executed.
If you have 2 handlers separated into 2 endpoints, then there are 2 subscriptions. 2 messages will be sent, one to each endpoint, and the processing of each message will be independent.
Handlers don’t subscribe to messages, endpoints do.