Use DataBus if only a few messages are large?

Architecture & Design
,
1

In our system we receive xml and send xml files from and to external clients.
Most of the xml files (say 90%) are small, less than 25kb. We also receive messages with larger content, and I estimate about 1% that are a couple of MB’s (say 2-8 MB).
Every now and then (literally about 4/5 times a year) we receive files that are 10+ MB. The maximum size is 25MB. In our current set up, the xml content is sent in events as payload using a the NServiceBus Data Bus. We try to break down the xml in smaller bits that we can process seperately, but that is not always an option.

In the past the system used MSMQ as it’s transport, and with the message limit of 4MB we had to use the Data Bus. But recently we have switched to RabbitMQ which would allow larger message sizes, but the setup is still with the Data Bus.

Using the DataBus comes with advantages (smaller messages) but also disadvantages (more complexity, clean up of the data bus files).

It seems that we could now completely remove the Data Bus and hold the content of the xml files directly in the events. I am wondering if there are any reasons to stick with the Data Bus or things that I should consider.

  • Is it better to stop using the Data Bus and keep the data on events? (and accept that a few messages will be large)
  • Even though there will not be many large events, what can I expect in terms of performance in general? Will it take a hit (more memory usage for instance) or a boost (no need to retrieve files from the Data Bus)?
  • How will this affect monitoring/audit? Anything to consider from that perspective?
  • Other things I should take into account?
2

Its really smart to reevaluate the need for databus when switching transport. You are right that it might not be needed for some transports but there are also other alternatives.

This information is listed at:

  • Use a different transport or different tier (e.g. Azure Service Bus Premium instead of Standard).
  • Use message body compression, which works well on text-based payloads like XML and JSON or any payload (text or binary) that contains repetitive data.
  • Use stream-based properties.
  • Use a more efficient serializer, such as a binary serializer.
  • Use NServiceBus.Attachments for unbounded binary payloads. The package is similar to the Data Bus but has some differences:
    • Read on demand: Attachments are only retrieved when read by a consumer.
    • Async enumeration: The package supports processing all data items using an IAsyncEnumerable.
    • No serialization: The serializer is not used, which may result in a significant reduction in memory usage.
    • Direct stream access: This makes the package more suitable for binary large objects (BLOBs since stream contents do not necessarily have to be loaded into memory before storing them or when retrieving them.

Let us know if you have any questions!

– Ramon

2 Likes