Unable to Configure ServiceControl and ServicePulse

Hi,

I am trying to setup ServiceControl and ServicePulse in local machine to monitor Error and Audit Queue from application, but service control is throwing “remote service return an error (404) not found” exception. I am using Azure Storage Queue for processing. Please check below configuration and let me know if i miss anything.

Note: message from application is reaching applications Audit or Error queue, Service control also able to monitor Error queue and it is trying to process something in “particular service” endpoint queue, but it keep on retrying.

Error log from Service Control:

2019-12-24 08:51:21.3963|67|Warn|NServiceBus.Transport.AzureStorageQueues.MessagePump|Azure Storage Queue transport failed pushing a message through pipeline
Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. —> System.Net.WebException: The remote server returned an error: (404) Not Found.
at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase1 cmd, Exception ex) at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.<DeleteMessageImpl>b__39(RESTCommand1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
— End of inner exception stack trace —
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.b__5(IAsyncResult ar)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessageRetrieved.d__3.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessagePump.d__7.MoveNext()
Request Information
RequestID:013a621e-4003-00c8-037a-ba7c35000000
RequestDate:Tue, 24 Dec 2019 16:51:20 GMT
StatusMessage:The specified message does not exist.
ErrorCode:MessageNotFound
ErrorMessage:The specified message does not exist.
RequestId:013a621e-4003-00c8-037a-ba7c35000000
Time:2019-12-24T16:51:21.3559960Z

Application Worker Endpoint Configuraiton:

 endpointConfiguration.DoNotCreateQueues();
            endpointConfiguration.Conventions().DefiningCommandsAs(type => type.GetCustomAttributes(true).Any(t => t.GetType().Name == "CommandAttribute"));
            endpointConfiguration.LimitMessageProcessingConcurrencyTo(GeneralConstant.NSBMaxProcessingConcurrency);
            endpointConfiguration.SendFailedMessagesTo("autoinv-imageupload-nsb-worker-error");
            endpointConfiguration.AuditProcessedMessagesTo("autoinv-imageupload-nsb-worker-audit");
            var recoverability = endpointConfiguration.Recoverability();
            recoverability.Delayed(delayed =>
            {
                delayed.NumberOfRetries(GeneralConstant.NSBMaxNumberOfRetries);
                delayed.TimeIncrease(TimeSpan.FromMinutes(GeneralConstant.NSBRetriesDelaytime));
            });
            recoverability.Immediate(imd =>
            {
                imd.NumberOfRetries(0);
            });
            recoverability.Failed(settings => settings.OnMessageSentToErrorQueue(failed =>
            {
                //TODO:SK send email notification
                return System.Threading.Tasks.Task.Run(() => { });
            }));
            endpointConfiguration.SendHeartbeatTo(serviceControlQueue: "particular-imageupload01-error",
                 frequency: TimeSpan.FromSeconds(5),
                 timeToLive: TimeSpan.FromSeconds(30));
            endpointConfiguration.ReportCustomChecksTo(serviceControlQueue: "particular-imageupload01-error");

Service Control Configuration:

Error Queue Name in config: autoinv-imageupload-nsb-worker-error
Error Forwarding: OFF

Application Error and Audit Queue:
It has record

Thanks

Do you have more than a single instance of SC consuming messages from that queue (particular-imageupload01-error)? Is that a shared Storage account?

No… only one instance of SC is consuming. Yes, it is shared storage account.

I’m out of ideas. If the SC can connect to the Storage account and is stack on re-processing a message, then it could be that the message is malformed or something is not right with it. I would suggest to raise a support case at https://particular.net/support providing the link to this post and additional information such as SC version, if possible the message that is failing processing by SC, and anything else to help to troubleshoot this.

@saravanakumar38 has this ever worked? I.e. is this issue happening in a single environment or all of them? If this is in a single environment only, have you tried to drop the queue and recreate it? I have a hunch this might be a queue issue (infrastructure).