Delete Inactive endpoints

Eirikur_Haraldsson · November 22, 2022, 1:56pm

What is the best way to delete Inactive endpoints automatically.
Both from Heartbeats in SerivcePulse and Endpoint list in ServiceInsight (is it the same source maybe?)

We have retention policy (24h) in our short retention environment. I would be nice if endpoints would be cleaned up as well. The messages are cleaned up automatically.

I was thinking about calling Delete on ServiceControl api is there any simpler way to achieve this.

mauroservienti · November 25, 2022, 10:44am

Hi @Eirikur_Haraldsson,

There isn’t an automatic way to purge inactive endpoints. Exercising the ServiceControl API is the only option.

Let me try to dig a bit deeper into your scenario. It sounds like a new ServicePulse/ServiceControl feature might be a good idea.

If you’re using heartbeats, even if the message retention is short, how come endpoints become inactive?

Eirikur_Haraldsson · November 25, 2022, 12:59pm

The reason they become inactive is because they are running in a container as part of a pull request, every container has a unique comptername and the container is automatically stopped after 4 hours.

Each branch consumes a isolated queue (using a postfix in the name)

mauroservienti · November 25, 2022, 1:23pm

I see. Thanks for the additional details.

One option would be to use ServiceControl integration events and use the HeartbeatStopped event to determine when to invoke the delete API in ServiceControl.

In the meantime, I raised User cannot set retention to cleanup inactive/deleted endpoint instances · Issue #3260 · Particular/ServiceControl · GitHub

SeanFeldman · November 25, 2022, 2:55pm

@mauroservienti wouldn’t deleting a perceived inactive endpoint based on a single HeartbeatStopped be risky? If heartbeats failed within heartbeat grace period, and that fine has exceeded but the endpoint is still alive, the endpoint would be removed while still running.

SeanFeldman · November 25, 2022, 2:58pm

Also,

invoke the delete API in ServiceControl

Is API documented?

mauroservienti · November 29, 2022, 7:59am

That’s correct, in a rush I didn’t provide more details. The endpoint that subscribes to ServiceControl events should have a saga with a timeout that allows to invoke the (not documented) HTTP API after a user-defined period of time.

Eirikur_Haraldsson · November 29, 2022, 2:01pm

We are stopping the endpoints (containers) with a script when they expire using script, so we could could invoke Delete API at the same time.

But we would prefer that NServiceBus environment could by itself handle the cleaning up process.

Something like this would be super:

endpointConfiguration.EnableEndpointTimeToLive(Timspan.FromHours(5));

(When the endpoint is created in RavenDb it would have some TTL, and would be removed after that)

Eirikur_Haraldsson · December 7, 2022, 1:36pm

What url can be used to delete an endpoint
DELETE …:33333/api/endpoints/myendpointname returns 405 Method Not Allowed

Eirikur_Haraldsson · December 9, 2022, 1:15pm

I only find a endpoint to disable the heartbeat.

PATCH http://{SERVICECONTROLURLANDPORT}/endpoints/{endpointId}
{ "monitor_heartbeat": false }

is delete not supported?

mauroservienti · December 12, 2022, 9:48am

Endpoints can be deleted, more information in the documentation.

ServicePulse issues a DELETE request to ServicerControl API endpoints/<endpoint-id>, see the source code here:

github.com

Particular/ServicePulse/blob/master/src/ServicePulse.Host/angular/app/modules/configuration/js/configuration.service.js#L70-L73


      
          deleteEndpoint(endpointId) {
              var url = this.uri.join(this.scu, 'endpoints', endpointId);
              return this.$http.delete(url);
          }

The endpoint ID is not it’s name, it’s the endpoint instance unique identifier and it’s available, for example, in the HeartbeatStopped ServiceControl integration event as HostId.

Eirikur_Haraldsson · December 12, 2022, 10:26am

I think we need to upgrade to version 4.4 of Servicecontrol to get delete support.

mauroservienti · December 12, 2022, 10:39am

That’s correct, @Eirikur_Haraldsson

ramonsmits · December 12, 2022, 12:38pm

FYI we have the following enhancement issues regarding this:

Housekeeping policy to automatically cleanup decommissioned/deleted endpoint instances · Issue #468 · Particular/ServicePulse · GitHub
Update monitoring for elastic scale environments · Issue #1257 · Particular/ServicePulse · GitHub

remyvd · December 12, 2022, 11:04pm

I have the same issue but not with endpoints, but with instances.

Every time it is auto-scaling from 1 to 10 instances and then back to 1, I need to do some cleaning of inactive endpoints and instances .

But I see in the post of @ramonsmits that there is already a github issue for that

A retention period to remove dead endpoint/instances for known scaling endpoints sounds good. But I think it is the wrong fix.

I think it would be better to be able to config some minimum instance count into a policy and only report when the number of instances goes below this. I only want to be alerted when this happens.

I don’t want to clean up dead instances in bulk or after a retention period.

mauroservienti · December 13, 2022, 7:12am

Thanks, @remyvd.

I raised:

In scaled out environments: define a policy to report stopped heartbeats only when instances count goes below a threshold · Issue #3317 · Particular/ServiceControl · GitHub

to keep track of your suggestion.