Adding Custom Tags/Values to NSB Metrics

Related to Otel metric feature:

NServiceBus 9.1.0 – Minor release available - Announcements - Discussion | Particular Software

Is there a way we can add our own custom tag key/value when NSB is adding metrics for things like critical time or processing time? It could be useful to be able to filter these by other domain specific things other than the handler/message type.

@BBrandtTX currentlly there is no option to do that.

Critical time is more a property of the endpoint queue than that of the message unless a queue only contains one message type.

Processing time states something of the message, a Tenant or Partition like attribute would a maybe provide more value.

Would you be able to provide a sample?

What type of information would you like to add:

  • data from message headers
  • data form the serialized message body
  • data from the deserialized object
  • custom data from the processing context
  • tags copied from spans or baggage?

Also, would you want to add the same tags to all metrics?

What is possible

What is currently possible is to create your own behavior that will measure processing duration and constructs critical time (DateTIme.UtcNow - timesent header):

sealed class DomainMetricsBehavior
    : IBehavior<IIncomingPhysicalMessageContext, IIncomingPhysicalMessageContext>
{
    static readonly Meter Meter = new("MyApp.Messaging");
    static readonly Histogram<double> ProcessingTime =
        Meter.CreateHistogram<double>("myapp.messaging.processing_time", "ms");
    static readonly Histogram<double> CriticalTime =
        Meter.CreateHistogram<double>("myapp.messaging.critical_time", "ms");

    public async Task Invoke(IIncomingPhysicalMessageContext context,
        Func<IIncomingPhysicalMessageContext, Task> next)
    {
        var start = Stopwatch.GetTimestamp();
        try
        {
            await next(context);
        }
        finally
        {
            var headers = context.MessageHeaders;
            var tags = BuildTags(headers);

            ProcessingTime.Record(Stopwatch.GetElapsedTime(start).TotalMilliseconds, in tags);

            if (headers.TryGetValue(Headers.TimeSent, out var sentRaw))
            {
                var sent = DateTimeOffsetHelper.ToDateTimeOffset(sentRaw);
                CriticalTime.Record((DateTimeOffset.UtcNow - sent).TotalMilliseconds, in tags);
            }
        }
    }

    static TagList BuildTags(IReadOnlyDictionary<string, string> headers)
    {
        var tags = new TagList { { "message_type", headers.GetValueOrDefault(Headers.EnclosedMessageTypes) } };
        if (headers.TryGetValue("MyApp.TenantId", out var tenant)) tags.Add("tenant", tenant);
        if (headers.TryGetValue("MyApp.Region", out var region)) tags.Add("region", region);
        return tags;
    }
}

Cardinality

Keep in mind that metrics create time series and that adding tags can result in may time series to be created to be performant and that it is recommended to keep low cardinality as high cardinality results in an significant increase in observability costs.