I had a question on the MSMQ queue created for service control named particular.serviceControl. Are the only messages that get sent/processed by this queue either the heartbeat or the custom check messages?
We have a huge backlog of messages in that queue due to an issue with how an endpoint was setup and were hoping we could just purge that queue of messages to get everything back to normal but i wasn’t sure if there was anything else that gets sent to that queue that might cause an issue if we were to do this.
You should not purge the Particular.ServiceControl queue. ServiceControl uses that queue to send messages to itself.
Heartbeat messages should have a Time to be Received applied to them so they should clear themselves out fairly quickly.
Which version of ServiceControl are you running? Can you provide more details about the messages that are filling the queue? Any information you can provide about the misconfigured endpoint would be helpful as well.
We are running on an old version of service control (v1.41.3) right now and from what i can tell all the messages that are filling up the queue are heartbeat notification messages.
I’m still working on trying to fix the endpoint that i believe is causing the issue. We have service control on a separate server than the offending endpoint and when i’m watching the outgoing queue (particular.servicecontrol) on the endpoint server i’m seeing huge spikes in messages that are being sent over (2-3K) in a 5 second window. We have about 90 endpoints currently on that server that is having huge message spikes and another 50 endpoints on a different server that are all setup to send the heartbeat messages. The other server seems to stay consistently around 60 in the outgoing queue which seems reasonable.
The offending endpoint (at least the one i’m pretty sure is causing the issue) was written for us by a vendor and they did some odd things with the container due to how their code was structured. They were using the autofac container and it appears they are creating 2 endpoint instances (a send only and a regular endpoint instance) and registering them in the container within the endpoint config. I have no idea why this was done but from what i can see the majority of the heartbeats are coming from these container endpoints as they gave them a specific name and that is what most of the heartbeat messages are from.
Currently that particular.servicecontrol queue is staying consistently at 68K message. They are getting processed but it can’t seem to catch up due to the number of messages that are getting sent into it.