Does the team or community have any recommendations for a docker health/liveness check over tcp/http (or even bash/powershell) for an endpoint running in docker to gauge if its still alive? We are starting to host our endpoints in kubernetes and have no liveness checks at the moment, with the thought that we should just raise critical errors and kill the process instead, but we are worried this wouldn’t necessarily detect a process that is deadlocked and not processing messages properly, the way that the ServiceControl heartbeats feature would, for example
It depends on what level you want to monitor Endpoint health/liveness. One the Platform level you can use https://docs.particular.net/monitoring/heartbeats/ and https://docs.particular.net/monitoring/metrics/ with https://docs.particular.net/nservicebus/hosting/critical-errors. On the hosting level (kubernetes, …) you can use any monitoring feature provided by the host technology. Maybe the good idea is to have double guard and use both ways
If you mean database deadlocked during message processing then NServiceBus kicks off https://docs.particular.net/nservicebus/recoverability/ and should resolve this kind of transient problem. In the worst scenario the message will land in the error queue. After fix the issue you will be able to re-try failed message using https://docs.particular.net/servicepulse/intro-failed-message-retries
Hope this help.
@mikedevbo the deadlocks in our case was the container was not crashed but not processing any messages. It seems by switching away from TopShelf and running as a simple console app has resovled this issue