I have a variety of message handlers that I am porting to NServiceBus from the MSMQ Binding for WCF. To preserve the concurrency characteristics of the existing handlers, we want each handler to process its commands serially, but also to run at the same time as other handlers so some are not starved by slow-running tasks. The first thing we’ve thought of is to make them all different endpoints with a concurrency limit of 1. (That’s what I’m callling ‘sharding’ in the post title)
However, because they’ve always been part of the same logical service as each other, some of the same events are published by multiple handlers. A given event is only supposed to by published by a single logical endpoint, correct?
So, how should we accomplish this sharding? Is it necessary to have the handlers shovel the events they want to publish over to some central publishing endpoint?
Before trying to address the technical problem, I’d like to try to understand why handlers need to run serially and cannot be changed to run in parallel. It would be nice to understand what’s the underlying requirement.
Our existing WCF MSMQ handlers were never designed for reentrancy, and introducing concurrency without a thorough audit and redesign of a massive amount of code is likely to make bugs. We do design all new handlers to be re-entrant. This question concerns porting old handlers that cannot easily be made reentrant.
I see that it is possible to call RegisterPublisher with multiple endpoints for the same event. Maybe this is the best way of handling it, but it does create a lot of work for subscribers.
Also as I mention above, part of why we would have multiple endpoints is to avoid starving faster handlers. To address this, we are looking at this section from the scaling out docs: Scaling with NServiceBus • NServiceBus • Particular Docs
Splitting handlers in different logical endpoints is the best thing to do, but different logical endpoints have different names, this means that routing needs to be adjusted both for senders and subscribers.
If the endpoint originally was named sales, to split out a message handler to a different logical endpoint a new endpoint needs to be created, e.g. sales_order_checkout to handle the OrderCheckout message only. To do so who was previously sending the OrderCheckout message to sales needs to be updated to send to the new endpoint, and who was subscribed to the OrderCheckedout event from sales needs to be updated because the publisher is now sales_order_checkout.
Would that topology change be an issue for you?
see that it is possible to call RegisterPublisher with multiple endpoints for the same event.
This sounds like a bug to me , as it should not be possible to have different logical endpoints publishing the same event. What transport are you using?
We are using the MSMQ transport, but the ability to call registerpublisher with multiple endpoints on the same event is at the framework level, not the transport.
The issue with the topology you mention is that these split-out handlers may be publishing the same event as each other: As a fictional case, the SendReportToClient command handler may raise the WidgetUpdated event, and so may the WithdrawOffer command handler. These two handlers rightfully belong to the same logical service, but if they are on different endpoints, then you will have two endpoints raising the same event.
, as it should not be possible to have different logical endpoints publishing the same event. What transport are you using?