Why am I not receiving HeartbeatRestored event from ServiceControl?

1

Current setup

  • Service Control v6.11
  • Service Control Monitoring v6.11
  • Service Pulse v2.5.0
  • Transport - Azure Storage Queue

We have a perfectly working setup for almost a year now, and numbers of our endpoints is growing as well. Service Platform is great, it’s giving a lot of value to the company I’m working on, and we want to extend the functionality to push something to our Slack channel to notify us if things are going south.

The perfect solution to our problem is written on Using ServiceControl Events • ServiceControl Contracts • Particular Docs, and we are particularly interested in the events below :backhand_index_pointing_down:

  • MessageFailed
  • HeartbeatStopped
  • HeartbeatRestored

The weird thing is, we can receive the first 2 events just fine; however, we are not receiving any for HeartbeatRestored.

To simulate the heartbeat events, I purposely stop our container apps, wait for a minute, and receive the HeartbeatStopped event. Then, I started them again and waited for about 5 to 10 minutes, but we didn’t receive any. What could be the issue here?

Another thing that might be worth mentioning is that I can see the endpoints move from Unhealthy to Healthy via Service Pulse.

2

Hi @christopherenriquez,

Can you please verify that the HeartbeatRestored event appears in the subscription routing table? If it is missing from here, it would explain the behaviour that you’re seeing.

3

Hi @PhilBastian, I think it is here. Please refer to the screenshot below

service-control-contracts
service-control-contracts1574×183 24.5 KB

4

Hi @christopherenriquez,

I just tested this with the same ServiceControl version, v6.11, with a very simple test app and I am receiving both the HeartbeatStopped and HearbeatRestored message. My setup for the message receiver is as follows:

var endpointConfiguration = new EndpointConfiguration("RetryFailedMessagesReceiver");
endpointConfiguration.UseSerialization<SystemJsonSerializer>();
var transport = new AzureStorageQueueTransport("<<myconnectionstring>>");
endpointConfiguration.UseTransport(transport);
endpointConfiguration.EnableInstallers();
endpointConfiguration.SendHeartbeatTo("Particular-ServiceControl");
var conventions = endpointConfiguration.Conventions();
conventions.DefiningEventsAs(
    type =>
    {
        return typeof(IEvent).IsAssignableFrom(type) ||
               // include ServiceControl events
               type.Namespace != null &&
               type.Namespace.StartsWith("ServiceControl.Contracts");
    });

public Task Handle(HeartbeatRestored message, IMessageHandlerContext context)
{
    logger.LogInformation("Received HeartbeatRestored message");
    return Task.CompletedTask;
}

public Task Handle(HeartbeatStopped message, IMessageHandlerContext context)
{
    logger.LogInformation("Received HeartbeatStopped message");
    return Task.CompletedTask;
}

Can you please setup a similar application, designed to contain the minimum amount of code necessary for receiving contract messages, and verify that you’re still seeing the issue?

5

The rest of the setup is the same except for this one. I’ll try this and get back to you soon.