We are running on an old version of service control (v1.41.3) right now and from what i can tell all the messages that are filling up the queue are heartbeat notification messages.
I’m still working on trying to fix the endpoint that i believe is causing the issue. We have service control on a separate server than the offending endpoint and when i’m watching the outgoing queue (particular.servicecontrol) on the endpoint server i’m seeing huge spikes in messages that are being sent over (2-3K) in a 5 second window. We have about 90 endpoints currently on that server that is having huge message spikes and another 50 endpoints on a different server that are all setup to send the heartbeat messages. The other server seems to stay consistently around 60 in the outgoing queue which seems reasonable.
The offending endpoint (at least the one i’m pretty sure is causing the issue) was written for us by a vendor and they did some odd things with the container due to how their code was structured. They were using the autofac container and it appears they are creating 2 endpoint instances (a send only and a regular endpoint instance) and registering them in the container within the endpoint config. I have no idea why this was done but from what i can see the majority of the heartbeats are coming from these container endpoints as they gave them a specific name and that is what most of the heartbeat messages are from.
Currently that particular.servicecontrol queue is staying consistently at 68K message. They are getting processed but it can’t seem to catch up due to the number of messages that are getting sent into it.