I’m getting lots of warnings in the in the ServiceControl Monitoring log.
“NServiceBus.Transport.AzureStorageQueues.MessagePump|Dispatching the message took longer than a visibility timeout. The message will reappear in the queue and will be obtained again.”
What could cause this? Can I change the timeout value for the monitoring service? The system is not very busy.
2019-01-29 15:36:56.9244|4|Warn|NServiceBus.TransportExtensions|A connection string named ‘NServiceBus/Transport’ was found. Using named connection strings is discouraged. Instead, load the connection string in your code and pass the value to EndpointConfiguration.UseTransport().ConnectionString(connectionString).
2019-01-29 15:42:19.7073|18|Warn|NServiceBus.Features.FeatureStartupTaskController|Exception occurred during stopping of feature startup task ‘TimeoutPollerRunner’.
System.ObjectDisposedException: Cannot access a disposed object.
Object name: ‘ExpiredTimeoutsPoller’.
at NServiceBus.ExpiredTimeoutsPoller.ThrowIfDisposed()
at NServiceBus.Features.TimeoutPollerRunner.OnStop(IMessageSession session)
at NServiceBus.Features.FeatureStartupTaskController.d__5.MoveNext()
2019-01-29 15:42:19.7183|18|Warn|NServiceBus.ExpiredTimeoutsPoller|Failed to fetch timeouts from the timeout storage
System.ObjectDisposedException: Cannot access a disposed object.
Object name: ‘ExpiredTimeoutsPoller’.
at NServiceBus.ExpiredTimeoutsPoller.ThrowIfDisposed()
at NServiceBus.ExpiredTimeoutsPoller.SpinOnce(CancellationToken cancellationToken)
at NServiceBus.ExpiredTimeoutsPoller.d__10.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ExpiredTimeoutsPoller.d__9.MoveNext()
2019-01-29 15:42:19.7273|18|Warn|NServiceBus.RepeatedFailuresOverTimeCircuitBreaker|The circuit breaker for TimeoutStorageConnectivity is now in the armed state
2019-01-29 15:42:26.3233|1|Warn|NServiceBus.TransportExtensions|A connection string named ‘NServiceBus/Transport’ was found. Using named connection strings is discouraged. Instead, load the connection string in your code and pass the value to EndpointConfiguration.UseTransport().ConnectionString(connectionString).
2019-01-29 15:42:27.5683|4|Warn|NServiceBus.TransportExtensions|A connection string named ‘NServiceBus/Transport’ was found. Using named connection strings is discouraged. Instead, load the connection string in your code and pass the value to EndpointConfiguration.UseTransport().ConnectionString(connectionString).
2019-01-29 16:03:58.1030|5|Warn|NServiceBus.Transport.AzureStorageQueues.MessagePump|Dispatching the message took longer than a visibility timeout. The message will reappear in the queue and will be obtained again.
NServiceBus.Transport.AzureStorageQueues.LeaseTimeoutException: The pop receipt of the cloud queue message ‘446894f3-bcc3-4f60-94e2-c5bdf7ff4175’ is invalid as it exceeded the next visible time by ‘00:00:00.0980521’.
at NServiceBus.Transport.AzureStorageQueues.MessageRetrieved.AssertVisibilityTimeout()
at NServiceBus.Transport.AzureStorageQueues.AtLeastOnceReceiveStrategy.d__1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessagePump.d__8.MoveNext()
are there any network connectivity issues between the monitoring service and the storage account?
Could you try upgrading the instance the latest version? The excessive storage usage might be caused by a bug that was patched in the newest version of the monitoring service.
Our monitoring service is installed on an Azure VM and I don’t think there are any network connectivity issues. I have also noticed the same problem on our test environment also on Azure.
I already have the latest version installed: 2.1.2.
It looks that we will need to release a patch for ASQ transport. Can you send an email to support@particular.net so that I can share the patch with you to validate that it solves the problem?
please have a look at ServiceControl 3.6.5 - Patch release available. This patch fixes a bug in native queue length monitroing logic that caused the large number of requests to the storage.