We’re using the Azure ServiceBus transport, but I think the underlying topic could apply to all transports.
When there’s certain events happening in the system, we sometimes need to send huge batches of messages, in the range of thousands or tens of thousands. Think notifying all users in a group with a push notification, for example.
The starting point of this is often a single action/transaction (adding something applicable to all users) that we then need to fan out into individual, potentially personalized, messages sending an email, push notification, whatever “fan-out” method is needed.
We currently use a bit of a magical DataBus property (add unlimited ids) and keep batching this up into small chunks to stay under the Azure Service Bus limits of 100 messages, until it finally is a single message and we can process it with business logic.
Though this works great, is actually super fast it feels like a hack for something that should have a better process (or, if not, some helpers built-in that assist with this fan-out principle over all transports in a reliable manner, for example not running into limits of 100 messages).
Any thoughts or suggestions you might have would be more than welcome.