Manipulating message header in a behavior when an invoke handler throws exception

Hi,

I’m currently looking into adding distributed tracing capabilities to my endpoints. My current route is to implement a custom IInvokeHandler-behavior which adds a trace whenever a message is received. Outgoing messages will also add a trace with another behavior.

In the case if an invoke handler throws an exception, I want to create a new trace with an “error”-tag.

However, when i’m adding the trace-id to the message header, that operation is rolled back (since an exception occurred). I suppose this is excepted behavior, but I just wanted to check if it’s actually possible to still commit the header manipulation even if an invoke handler throws an exception?

Br,
Alexander

adding distributed tracing capabilities to my endpoints

Could you explain what you mean by this? I’m asking, because we might already be doing this for you. I’m not sure how familiar you are with our entire platform.

We allow auditing of messages. This means every successfully processed message will end up in some queue. Usually we advise to have them end up in the same queue for all endpoints and capture those message.

We do this ourselves by using ServiceControl. It stores every single message inside a database, with a retention of several days.

After that you can use this data to even visualize it, like we do with our ServiceInsight tool. It accesses ServiceControl via an API.

We do the same for messages that cannot be processed. We send them to a central queue, ServiceControl processes them and we visualize this using ServicePulse. It can even provide much more information about monitoring your endpoints.

To get back to your questions, I’m not sure what it is you’re trying to achieve. We already put a fair amount of information in headers though, like MessageId and CorrelationId and ConversationId. More info can be found here. We even put the exception and stacktrace inside message headers on messages that fail. But we don’t do it with every retry, we just do it after it keeps failing. More information on how to extend NServiceBus to deal with this can be found here and here.

But I’m wondering if we’re already doing what you’re trying to achieve. But I’m not 100% sure what it is you’re trying to achieve, from a functional perspective?

When I say distributed tracing I mean add the capability to retrieve the entire messaging flow of particular messages, including execution time for HTTP-requests and database calls (if there are any). The usage of these messaging flows are primarily developers when errors occur, but it’s also to analyze the performance of individual services that is participating in certain messaging flows. Another complimentary usage would be to enable correlation of the tracing information (using the conversationId) with other logging that the endpoint may log (we currently use NLog for this).

And right now i’m analyzing and testing if OpenTracing can be used for these purposes.

I realize that there are overlapping functionality with the built in auditing and using OpenTracing, but is it possible to achieve this by extending the auditing-behavior instead?

Take a look at the way we integrate Metrics with third parties like NewRelic. Maybe that’s something you can use to achieve your goal: https://docs.particular.net/samples/logging/