ServicePulse Mail notifications

#1

Hi,

We have configured Mail Notifications in ServicePulse. We are getting mails when endpoints become unhealty but we don’t get mails when there are errormessages. Reading the documentation, errormessages should also be alerted. Do I misunderstand the documentation or is there something that is not working?

Thx

Kind Regards,
Michael

#2

Hi @MichaelSleeuwagen,

the email notifications are generated based on custom checks. These are responsible for monitoring the health (index staleness, remote instance availability, etc.) of ServiceControl instances and are not triggered if error messages have been ingested from the error queue.

One way to get notifications on failed messages is to use integration events. These events can be subscribed to by user endpoints and used to trigger email notifications (you would need to write this piece of code yourself). Here is a sample showing how this can be set up.

Cheers,
Tomek

#3

@tmasternak, we are planning to use this sample below as a reference and host the CustomEventsHandler in Azure Functions with ServiceBus as the transport.

“Monitor with ServiceControl events • ServiceControl Contracts Samples • Particular Docs”

as mentioned in the referenced article

" If adjusting this sample to use the Azure Service Bus transport, note that the subscribing endpoint must also use the same name shortening strategy as ServiceControl."

  1. Not sure what this means, can you elaborate on what specific changes do we need to do ?
  2. How can we verify if servicecontrol is publishing the MessageFailed events ? is there a way i can check them ?
  3. We are using a custom topic name (not the bundle-1), will this cause any issues ?

Thanks -Nen

#4

Hi @Nen_Zax,

I think that this part is indeed confusing:

If adjusting this sample to use the Azure Service Bus transport, note that the subscribing endpoint must also use the same name shortening strategy as ServiceControl.

Please have a look at the proposed tweaks that I’ve made to the doco page.

Cheers
Tomek

#5

@tmasternak thanks it makes sense now, can you help with questions 2 and 3 please ?

and one more question, do we need to do any extra configuration if we are using azure functions to host ? and subscribe to service control events ?

#6

Hi @Nen_Zax,

  1. How can we verify if servicecontrol is publishing the MessageFailed events? is there a way I can check them?

Sending a message to the error queue should trigger MessageFailed event to be delivered to the subscribed endpoint.

  1. We are using a custom topic name (not the bundle-1), will this cause any issues?

The integration events should use the bundle-1 topic as well. What is the transport that you’ve chosen for you ServiceControl instance?

Cheers,
Tomek

#7
  1. we are using azure service bus with azure functions.
  • we created an endpoint to subscribe to the MessageFailed event, the handler never gets triggered, the following is my configuration.
    public static ServiceBusTriggeredEndpointConfiguration GetAzBusConfiguration(Configuration configuration, string serviceBusEndpointName )
    {
        var azureServiceBusConfiguration = new ServiceBusTriggeredEndpointConfiguration(serviceBusEndpointName);
        azureServiceBusConfiguration.UseSerialization<NewtonsoftSerializer>();
        azureServiceBusConfiguration.AdvancedConfiguration.Conventions().DefiningEventsAs(
            type =>
            {
                return typeof(IEvent).IsAssignableFrom(type) ||
                       // include ServiceControl events
                       type.Namespace != null &&
                       type.Namespace.StartsWith("ServiceControl.Contracts");
            });
        azureServiceBusConfiguration.Transport.CustomTokenCredential(new DefaultAzureCredential());
        azureServiceBusConfiguration.Transport.ConnectionString(configuration.AzurSBdNamespace);

        return azureServiceBusConfiguration;
    }

and my handler is below

public class EndpointsMonitor :
    IHandleMessages<MessageFailed>
{
    private static readonly ILog Log = LogManager.GetLogger(typeof(EndpointsMonitor));


    public async Task Handle(MessageFailed message, IMessageHandlerContext context)
    {
        Log.Error(
            $"Received ServiceControl 'MessageFailed' event for a {message.MessageType} with ID {message.FailedMessageId} in {GetType()}.");

    }
}

THanks -Nen

#8

Have you set up the subscription for the event using Azure Functions with Azure Service Bus • NServiceBus.AzureFunctions.InProcess.ServiceBus • Particular Docs ?

The reason this is needed is that subscriptions are created at startup which happens once the first message arrives which would never happen if your function endpoint is only handling that specific event.

Cheers,

Andreas

#9

@andreasohlund i figured out it out, the problem was, we had our own custom topic, and i was creating service control subscriptions in the custom topic.

but then i realized that servicecontrol uses bundle-1 topic, once i moved my subscription into this topic, i am able to handle service control events.

questions

  1. can we change servicecontrol to use our own custom topics ?

  2. after implementing this, now i am confused on how are we going to monitor this integrationevents endpoint in ServicePulse using “ServicePlatformConnectionConfiguration” ?

  3. it cannot use the same error queue (as it will trigger an infinite loop), can service pulse/ service control monitor multiple error queues ?

  4. can it use the same heartbeats queue ?

  5. assuming it can use the same audit queue

Thanks -Nen