Let’s say I start with one domain entity.
Initially I have one endpoint, with a few handlers in, on a single “queue”. I realise soon that this is causing me two problems:
The endpoint simply can’t get through the messages quickly enough. Not only are any one user’s messages taking a long time to be handled, but messages resulting from the actions of different users are behind each other. So Andy’s messages are waiting in a queue behind Susan’s messages, which are waiting in the same queue behind Alex’ messages, and so on.
This is a particular problem when a device that has been offline for some time re-connects and all it’s messages arrive at once.
Some messages take a particularly long time to process (relatively speaking). This is causing potentially “fast” messages to be delayed behind a few “slow” ones.
I look at the documentation on scaling out and decide to implement competing consumer.
I move “slower” handlers to a new endpoint, and I deploy two more instances of the “fast” endpoint. (EDIT: Come to think of it, the built-in multi-threading support in NServiceBus would have achieved almost the same thing).
I now see a situation where handlers are competing on the same domain entities by ID, causing concurrency failures resulting in automatic retries.
In particular the “slow” messages are almost never processed, because for every entity they load, process, and attempt to save, another endpoint has already updated that entity for a different, “faster” message, and so the slow message fails to save and the whole pipeline has to start over. Immediate retries and Delayed retries are superb built-in functionality, but in this case the failed attempts account for a huge amount of waste.
Thank you for reading this far.
Usually in these situations the advice from the community is “Your boundaries are wrong.”. Does this mean my entity is wrong, and should be more entities? What if the actions being performed all require the same value to satisfy invariants? In that case I can’t split the entity up.
I know that for NServiceBus, topic-based routing is considered undesirable - so I can’t have an endpoint-per-user and ensure at least any one user’s messages aren’t waiting behind their colleagues’ messages.
I’m not sure what the options are. Can anyone help?