Consumer disconnects from RabbitMQ and is not reconnecting

1

We have NserviceBus 7.4.4

When message sended to queue and service process time takes longer than “consumer_timeout” in rabbitmq, service loses queue and dont process any messages any more. Messages just hang in the status ready.

The only thing that helps is restarting the “dead” service after which it starts processing all messages from the queue

Can you provide some information that maybe can fix this issue via configuration of nservicebus or maybe this issue fixed in later versions of Nservicebus. Or maybe this issue cant be fixed.

2

Hi @Artem,

You might be experiencing a bug that was solved in the 6.1.1 release o the transport.

Could that be the case?

Cheers,
Tomek

3

Yeap, that it the problem gone! Thank you!

1 Like
4

I’ve been fighting this exact problem for months, only it is happening on the ServiceControl audit instance. Was this (or a similar) fix ever done on ServiceControl? We are currently running v5.0.5

5

Hi @RMarcum,

The 5.0.5 version of ServiceControl uses the already patched version of RabbitMQ transport.

Are there any errors in your log file for the audit instance?

6

Here are a few relevant line:

2024-07-16 00:25:53.1172|49|Warn|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’ channel shutdown: AMQP close-reason, initiated by Peer, code=406, text=‘PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more’, classId=0, methodId=0
2024-07-16 00:25:53.1172|49|Warn|NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker|The circuit breaker for ‘Particular.Nsb.Nova.Staging MessagePump’ is now in the armed state
2024-07-16 00:25:53.1172|75|Info|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’: Attempting to reconnect in 10 seconds.
2024-07-16 00:26:03.1584|229|Info|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’: Connection to the broker reestablished successfully.
2024-07-16 00:26:03.1584|75|Info|NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker|The circuit breaker for ‘Particular.Nsb.Nova.Staging MessagePump’ is now disarmed
2024-07-16 00:57:03.1664|173|Warn|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’ channel shutdown: AMQP close-reason, initiated by Peer, code=406, text=‘PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more’, classId=0, methodId=0
2024-07-16 00:57:03.1664|173|Warn|NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker|The circuit breaker for ‘Particular.Nsb.Nova.Staging MessagePump’ is now in the armed state
2024-07-16 00:57:03.1822|221|Info|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’: Attempting to reconnect in 10 seconds.
2024-07-16 00:57:13.2072|159|Info|NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker|The circuit breaker for ‘Particular.Nsb.Nova.Staging MessagePump’ is now disarmed
2024-07-16 00:57:13.2072|229|Info|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’: Connection to the broker reestablished successfully.
2024-07-16 01:28:13.2289|162|Warn|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’ channel shutdown: AMQP close-reason, initiated by Peer, code=406, text=‘PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more’, classId=0, methodId=0
2024-07-16 01:28:13.2289|162|Warn|NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker|The circuit breaker for ‘Particular.Nsb.Nova.Staging MessagePump’ is now in the armed state
2024-07-16 01:28:13.2289|131|Info|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’: Attempting to reconnect in 10 seconds.
2024-07-16 01:28:23.3184|220|Info|NServiceBus.Transport.RabbitMQ.MessagePump|‘Particular.Nsb.Nova.Staging MessagePump’: Connection to the broker reestablished successfully.