Capture and visualize metrics using NewRelic


(Daniel Marbach) #1

Hi everyone,

We created a sample that shows how to integrate the new NServiceBus.Metrics package with NewRelic. You can find it over here https://docs.particular.net/samples/logging/new-relic/

A sample dashboard with Insights

With thanks,
The team in Particular


#2

@danielmarbach - this is great! Is there any way to get messages to show as transactions (without decorating each handler with a newrelic attribute)? Otherwise, with the above solution, the APM in New Relic is blank and any time a new message is added in code, you need to manually add it to the dashboard in new relic.

thanks!


(Daniel Marbach) #3

Hi @MSWCRB

Have you tried adding the custom metrics with a wild card?

https://docs.newrelic.com/docs/insights/use-insights-ui/explore-data/metric-explorer-search-chart-metrics-sent-new-relic-agents

If you use a wildcard in the diagram then every new metric that matches a wildcard will show up in the diagram automatically if I recall correctly.

If that doesn’t work you could extend the sample solution by using the transaction SDK API to set transaction names.

Regards
Daniel


#4

@danielmarbach As far as I can tell, adding the custom metrics with a wildcard doesn’t get them to show as Transactions in APM - it just pulls them all into the Dashboard in Insights as metrics.

I didn’t understand your second suggestion about setting transaction names - it looks to me like that’s for overriding the default name given a transaction. The issue here is that messages being handled aren’t being viewed as transactions at all in the first place. Adding the[Transaciton] Attribute to each handler does this, but we really don’t want to be decorating all of our code to support a Monitoring service (e.g. what happens if we want to change monitoring services).


(Daniel Marbach) #5

You are correct they don’t show up as Transactions in APM. I thought the new Metrics Explorer is the new way to go with NewRelic, that’s why we focused mostly on delivering a sample for that. Unfortunately I’m not a NewRelic expert.

We originally engaged with NewRelic support in this issue

as well as over private channels to see if we can get them to update the transaction support in the NewRelic agent. The custom spike I did can be found on

The problem was that we had to reference agent assemblies which we didn’t know if it is OK to reference from a licensing perspective. That in combination with the push from NewRelic towards the new metric explorer we felt it is better to use custom metrics instead.

If you are willing to go down the route of custom instrumentations you can use the approach outlined in

https://docs.newrelic.com/docs/agents/net-agent/custom-instrumentation/introduction-net-custom-instrumentation

or combine it with the code provided in my spike repo. I would encourage you to use the XML variant.

Regards
Daniel


#6

@danielmarbach With the code from the sample above & metrics 2.0, I am seeing this when I start my endpoint:

2018-03-26 01:02:16.956 INFO  NServiceBus.RecoverabilityExecutor Immediate Retry is going to retry message 'c2669320-7c90-48e7-a704-ea57b32a304c\404012667' because of an exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at MyNamespace.NewRelicFeature.Normalize(String name)
   at MyNamespace.NewRelicFeature.<>c__DisplayClass4_1.<RegisterProbes>b__0(DurationEvent& event)
   at NServiceBus.OnEvent`1.Invoke(TEventType& e)
   at CriticalTimeProbeBuilder.<>c__DisplayClass2_0.<WireUp>b__0(ReceivePipelineCompleted e) in C:\BuildAgent\work\62a1a7ae466302a5\src\NServiceBus.Metrics\ProbeBuilders\CriticalTimeProbeBuilder.cs:line 25
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Threading.Tasks.Task.WhenAll(IEnumerable`1 tasks)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\MainPipelineExecutor.cs:line 34

(Daniel Marbach) #7

@MSWCRB Interesting. Works on my machine. Would you be able to share a bit more details or a repro?

I used the Metric 2.0 sample and started the endpoint.

I can imagine something going on with the normalize method when a null or empty string is passed in would I would like to understand how. Maybe you can add a try catch to the normalize method and then share the details (stack trace, input param etc.) if possible*.

* I know your time is valuable and I'm asking for much but if we can solve this other people can benefit from it

Regards
Daniel


#8

It is working sporadically. It seems to trip up on messages with no content. I’m not sure what is causing these messages to be sent TBH - perhaps they are subscription messages or something like that. If I delete the messages with no body from the queue, it processes messages with content correctly.

I’m going to try this:
if (!string.IsNullOrEmpty(@event.MessageType))
{
nameMapping.TryGetValue(duration.Name, out var mappedName);
var newRelicName = string.Format(mappedName ?? FormatMetric(duration.Name, endpointName),
Normalize(@event.MessageType));
RecordResponseTimeMetric(newRelicName, Convert.ToInt64(@event.Duration.TotalMilliseconds));
}