Hi everyone,
We’ve just released NServiceBus.AmazonSQS 3.3.2 and 4.1.2.
Fixed bugs
- #252 Message body on S3 deleted before an event is received
How to know if you are affected
If endpoints are publishing large event messages, with a payload size greater than 256 KB, to more than one subscriber, then you are affected.
Symptoms
Large message bodies are stored on S3. When messages, containing a reference to a large body stored on S3, are successfully handled the body is deleted from S3. In case of events, when there are multiple logical subscribers interested in the body, this causes a message loss scenarios because the fastest subscriber will delete the body in the S3 bucket before other subscribers will have a chance to process it.
Should you upgrade immediately
Customers publishing events with large bodies should upgrade immediately.
Behavior change
The transport cannot delete bodies in S3 anymore. It is the responsibility of the S3 bucket aging policy to remove bodies stored in S3. This behavior is also aligned with how the NServiceBus DataBus cleanup works.
The deletion logic of the message that takes the S3 body into account is a best effort. The assumption behind the code is that if the body can’t be deleted, it will be cleaned up by the S3 aging policy. So an aging policy is mandatory and enforced by design. Customers using this features need to make the necessary tradeoffs between the window of receiving a message and the actual deletion of the bucket content vs. the costs involved. An additional angle that has to be taken into account is that since the message body is no longer a part of the message itself the aging policy needs to take auditing into account as well. When a message is audited, and the body of the message has to be investigated as part of the audited message, messages bodies cannot be deleted as the message is received.
For customers using ServiceControl, it is advised to align the maximum number of days that a message will be retained within SQS and S3 with the data retention policies of ServiceControl or vice versa.
Where to get it
You can download the updated version and see the full release notes here:
- 3.3.2 https://github.com/Particular/NServiceBus.AmazonSQS/releases/tag/3.3.2
- 4.1.2 https://github.com/Particular/NServiceBus.AmazonSQS/releases/tag/4.1.2
With thanks,
The team in Particular
Please read our release policy for more details.