NServiceBus endpoint was not processing message but Kubernet pod is still running

,
1

Hi,

I am using NServiceBus 8.0.3 and Azure Storage 12 for Handler component to process the azure queue message. Handler code is written in .Net 6 and deployed in Kubernet pod.
Environment: Production
Issue:
Pod is running fine and NServicebus handling code is also working fine after deployment for few week, after some time, messages were not processed, but kubernet pod status is still running. Messages are still in queue and there is no error message logged.

As soon we restarted the pod, messages are getting processed by the Handler.

What could be causing this issue?

Attached code for Reference:
builder.UseNServiceBus(ctx =>
{
var endpointConfiguration = new EndpointConfiguration(emailEndpoint);

    var transport = new AzureStorageQueueTransport(azureConnectionString)
    {
        PeekInterval = TimeSpan.FromSeconds(peekInterval),
        MaximumWaitTimeWhenIdle = TimeSpan.FromSeconds(maximumWaitTimeWhenIdle),
        MessageInvisibleTime = TimeSpan.FromSeconds(messageInvisibleTime),
        ReceiverBatchSize = batchSize
    };  
    
    endpointConfiguration.UseTransport(transport);        
    endpointConfiguration.UsePersistence<NonDurablePersistence>();
    endpointConfiguration.UseSerialization<NewtonsoftJsonSerializer>();
    endpointConfiguration.LimitMessageProcessingConcurrencyTo(maxConcurrency);
    endpointConfiguration.SendFailedMessagesTo(errorQueue);
    endpointConfiguration.AuditProcessedMessagesTo(auditQueue);

    endpointConfiguration.EnableOpenTelemetry();

    var recoverability = endpointConfiguration.Recoverability();
    recoverability.Delayed(delayed =>
    {
        delayed.NumberOfRetries(maxNumberOfRetries);
        delayed.TimeIncrease(TimeSpan.FromSeconds(retriesDelaytime));
    });
    recoverability.Immediate(imd =>
    {
        imd.NumberOfRetries(0);
    });
    return endpointConfiguration;
});

Thanks,
Saravanakumar

2

Hi Saravanakumar,

We have a critical error that should be raised when the transport looses connectivity, but the default behavior is that is just logs the exception and keeps retrying indefinitely.

If I understood you correctly, there is no exception in the logs at all. That is suspicious.

How often does it happen? Do you have a suspicion it correlates to some other event?

I see that you have open telemetry enabled. What type of monitoring solution do you use? Is the monitoring tool or the portal still showing requests on the storage account from the pod or is there nothing happening there anymore?

Would it be possible to do the following:

  • Please send us the log files of such an incident
  • If possible, you could take a dump of the running process and send us that too (see for example dotnet-dump)
  • Interesting would also be the stack trace of the managed threads, but when do you the dump it should cover that too.
  • Alternatively, depending on the load of your system, you could also enable debug log to get more levels of details at the time of the incident next time

Please send us this information via our support form in Support options • Particular Software.

Thanks,

Daniel

3

Hi Daniel,

This issue is happening inconsistently and after long time i got this error in production and able to get the error log from the K8s pod.

Pod was still in running state and its memory usage were 150 MB. Below are the error message received or logged in pod.

Will this outofmemory exception will throw critical error from NServiceBus?
Handling critical error will shut down and restart the current pod will solve probelm?

Logs from Pod:

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:38:29.4446495Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:38:29.4446500Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:29.4446504Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:29.4446504Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:29.4446504Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:38:39.4468036Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:38:39.4468043Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:39.4468048Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:39.4468048Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:39.4468048Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:38:49.4449924Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:38:49.4449933Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:49.4449938Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:49.4449938Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:49.4449938Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:38:59.4451492Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:38:59.4451498Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:59.4451502Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:59.4451502Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:38:59.4451502Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:39:09.4453252Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:39:09.4453257Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:09.4453262Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:09.4453262Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:09.4453262Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:39:19.4454946Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:39:19.4454951Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:19.4454955Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:19.4454955Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:19.4454955Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:39:29.4456503Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:39:29.4456512Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:29.4456517Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:29.4456517Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:29.4456517Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:39:39.4448349Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:39:39.4448356Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:39.4448364Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:39.4448364Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:39.4448364Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:39:49.4450125Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:39:49.4450132Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:49.4450136Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:49.4450136Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:49.4450136Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Export nservicebus.messaging.successes, Total number of messages processed successfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5483710Z, 2023-07-27T16:39:59.4451747Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955372

Export nservicebus.messaging.fetches, Total number of messages fetched from the queue by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509623Z, 2023-07-27T16:39:59.4451754Z] nservicebus.discriminator: nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 1955446

Export nservicebus.messaging.failures, Total number of messages processed unsuccessfully by the endpoint., Meter: NServiceBus.Core/0.1.0
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:59.4451758Z] nservicebus.discriminator: nservicebus.failure_type: System.OutOfMemoryException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 56
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:59.4451758Z] nservicebus.discriminator: nservicebus.failure_type: System.Net.Http.HttpRequestException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 3
(2023-06-19T20:01:52.5509913Z, 2023-07-27T16:39:59.4451758Z] nservicebus.discriminator: nservicebus.failure_type: System.Threading.Tasks.TaskCanceledException nservicebus.message_type: Nowcom.Services.EmailService.Common.SendGrid.Commands.OutboundEmail, Nowcom.Services.EmailService.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null nservicebus.queue: emailservice-dc-medium-outbound-worker LongSum
Value: 10

Thanks

4

Hi @saravanakumar38

It seems while I was on leave your reply was not picked up by our case management system and thus nobody followed through. I do apologize.

By looking at the information you provided, I’m uncertain if I can see anything that would let me conclude what is happening.

Would it be possible to do the following:

  • Please send us the log files of such an incident
  • If possible, you could take a dump of the running process and send us that too (see for example dotnet-dump)
  • Interesting would also be the stack trace of the managed threads, but when do you the dump it should cover that too.
  • Alternatively, depending on the load of your system, you could also enable debug log to get more levels of details at the time of the incident next time

Please send us this information via our support form in Support options • Particular Software .

Once we have processed that, we can also go through the code in a call.

Regards,
Daniel