We have configured Mail Notifications in ServicePulse. We are getting mails when endpoints become unhealty but we don’t get mails when there are errormessages. Reading the documentation, errormessages should also be alerted. Do I misunderstand the documentation or is there something that is not working?
the email notifications are generated based on custom checks. These are responsible for monitoring the health (index staleness, remote instance availability, etc.) of ServiceControl instances and are not triggered if error messages have been ingested from the error queue.
One way to get notifications on failed messages is to use integration events. These events can be subscribed to by user endpoints and used to trigger email notifications (you would need to write this piece of code yourself). Here is a sample showing how this can be set up.
@tmasternak, we are planning to use this sample below as a reference and host the CustomEventsHandler in Azure Functions with ServiceBus as the transport.
" If adjusting this sample to use the Azure Service Bus transport, note that the subscribing endpoint must also use the same name shortening strategy as ServiceControl."
Not sure what this means, can you elaborate on what specific changes do we need to do ?
How can we verify if servicecontrol is publishing the MessageFailed events ? is there a way i can check them ?
We are using a custom topic name (not the bundle-1), will this cause any issues ?
If adjusting this sample to use the Azure Service Bus transport, note that the subscribing endpoint must also use the same name shortening strategy as ServiceControl.
we are using azure service bus with azure functions.
we created an endpoint to subscribe to the MessageFailed event, the handler never gets triggered, the following is my configuration.
public static ServiceBusTriggeredEndpointConfiguration GetAzBusConfiguration(Configuration configuration, string serviceBusEndpointName )
{
var azureServiceBusConfiguration = new ServiceBusTriggeredEndpointConfiguration(serviceBusEndpointName);
azureServiceBusConfiguration.UseSerialization<NewtonsoftSerializer>();
azureServiceBusConfiguration.AdvancedConfiguration.Conventions().DefiningEventsAs(
type =>
{
return typeof(IEvent).IsAssignableFrom(type) ||
// include ServiceControl events
type.Namespace != null &&
type.Namespace.StartsWith("ServiceControl.Contracts");
});
azureServiceBusConfiguration.Transport.CustomTokenCredential(new DefaultAzureCredential());
azureServiceBusConfiguration.Transport.ConnectionString(configuration.AzurSBdNamespace);
return azureServiceBusConfiguration;
}
and my handler is below
public class EndpointsMonitor :
IHandleMessages<MessageFailed>
{
private static readonly ILog Log = LogManager.GetLogger(typeof(EndpointsMonitor));
public async Task Handle(MessageFailed message, IMessageHandlerContext context)
{
Log.Error(
$"Received ServiceControl 'MessageFailed' event for a {message.MessageType} with ID {message.FailedMessageId} in {GetType()}.");
}
}
The reason this is needed is that subscriptions are created at startup which happens once the first message arrives which would never happen if your function endpoint is only handling that specific event.
@andreasohlund i figured out it out, the problem was, we had our own custom topic, and i was creating service control subscriptions in the custom topic.
but then i realized that servicecontrol uses bundle-1 topic, once i moved my subscription into this topic, i am able to handle service control events.
questions
can we change servicecontrol to use our own custom topics ?
after implementing this, now i am confused on how are we going to monitor this integrationevents endpoint in ServicePulse using “ServicePlatformConnectionConfiguration” ?
it cannot use the same error queue (as it will trigger an infinite loop), can service pulse/ service control monitor multiple error queues ?