We are experiencing intermittent timeouts with our OutboxRecord tables across several hosts. These timeouts occur outside of high-load or batch processing periods, which leads us to suspect that they might be related to either high volumes of records or potential database locking issues.
we have identified that the TransportOperations column in the OutboxRecord table can become extremely large, leading to significant performance issues.
Specifically, we are considering:
Compression: Implementing compression for the TransportOperations column to reduce size and improve performance.
Outbox Feature Configuration: Verifying and optimizing our configuration for the Outbox feature in NServiceBus V 7.X
Could anyone provide guidance or best practices on:
Efficiently handling large data in the TransportOperations column?
Best practices for diagnosing and resolving database locking or timeout issues related to NServiceBus?
Recommendations for using compression for large TransportOperations data using custom behaviour and how?
Any insights or experiences would be greatly appreciated.
When you have large messages, you should consider using the DataBus feature; this will allow you to store any more significant segments of data (like binary files) outside of the queue and still have them associated with the message.
When using the DataBus feature, any data in a message that is using the databus feature will not be stored in the message queue or outbox (TransportOperations) table, so the data in the table should be a lot smaller.
Another option is to store some kind of reference to your large data in your message that you can access later, so that it doesn’t end up in your message. (This is what the databus does for you also)
Here are a few samples showing how to use the DataBus feature
Thanks for the suggestion! I understand that the DataBus feature helps with handling large data, but it seems like it would significantly alter our current implementation. In this case, what we really need is a more specific solution for compressing data, particularly for Outbox messages when they are stored in the Outbox record table.
The DataBus approach could be useful in other scenarios, but here, we need to maintain the data within the Outbox and just reduce its size to optimize payload and storage without changing how the message is handled. If you have suggestions for compression within this specific context, I’d be keen to review them!!
Message signing (and verification) is similar to compression in that you want to do it on the physical bytes of the message, but you’d have to be careful to also include a header or something so that non-compressed messages can still be read correctly.
