We’ve wired up a few of our test envs and production deployments to prometheus/grafana and I’ve noticed a bunch of the following metric: nservicebus_messaging_failures_total → nservicebus_failure_type = “NServiceBus.TransactionalSession.ConsumeMessageException” in our grafana dashboards that I’ve been playing around with.
We do use TransactionalSession & Outbox with the following versions of software:
.NET 6
NSB 8.2.3
NServiceBus.TransactionalSession 2.0.0
NServiceBus.Transport.AzureServiceBus 2.0.2
Microsoft.EntityFrameworkCore 6.0.36
It doesn’t seem to be impacting anything we’re doing. There are no additional logs with this, no related failed messages and I didn’t notice anything on the console of the service in question even - I’m just wondering if there’s something we should be taking a look at in more detail?
Usually, NServiceBus.TransactionalSession.ConsumeMessageExceptions arise when it takes some time for the database transaction to complete. This doesn’t indicate that something went wrong or that you lost any data, but it does indicate that your configuration can be optimized.
If you see this exception pop up repeatedly, it may indicate that your transaction is taking longer to commit, and it may be useful to adjust CommitDelayIncrement property on the session. If you’re aware that some of these transactions take longer, you can also extend the maximum commit duration on the session.
To share some context on how this works, when you commit the transactional session, the following happens:
a control message is dispatched immediately. The control message serves as a hook to fetch the messages that were lined up in the session and dispatch them in a consistent manner
then your database transaction is committed.
In some cases, the control message arrives before the outgoing messages were stored in the Outbox table, so when attempting to dispatch the messages, none are found. To mitigate this issue, NServiceBus throws a ConsumeMessageException which is handled by the framework itself, by resending a new control message that’s delayed a bit further. This happens as part of Phase 2.
This conclusion is only valid if this is the only exception you see popping up in the system. It is worth validating that no exceptions occur when committing the transactional session. You may want to ensure you have telemetry set up to capture exceptions in the controllers.
In order to make this a bit clearer, we will be adding some log statements to our next release so that you would see warnings of this occurring in your logs as well.