"A message was already published on this channel with the same confirmation number"

arezoomand · December 26, 2018, 10:10am

I use NServicebus 7.x, I implement a scenario with saga and in one step I should try the same step about 5 times . I send message with DelayDelivery to that step, but after a period of time when delayed messages reaches the main queue I receive the error “A message was already published on this channel with the same confirmation number”
I use RabbitMq as Transport.
The program should process 100000 saga instance per hour.
The configuration is :

var config = new EndpointConfiguration(setting.Name);
config.SendFailedMessagesTo(setting.ErrorQueue);

var recoverability = config.Recoverability();
recoverability.Delayed(
delayed =>
{
      delayed.NumberOfRetries(0);
});

var timeoutManager = config.TimeoutManager();
timeoutManager.LimitMessageProcessingConcurrencyTo(50);

config.UseSerialization<NewtonsoftSerializer>();
config.LimitMessageProcessingConcurrencyTo(50);

var persistence = config.UsePersistence<SqlPersistence>();
var transport = config.UseTransport<RabbitMQTransport>();
transport.UseDirectRoutingTopology();
transport.ConnectionString(Configuration.GetConnectionString("RabbitMqConnectionString"););
transport.Transactions(TransportTransactionMode.ReceiveOnly);

config.EnableInstallers();

ramonsmits · December 27, 2018, 9:46am

Do you have additional details like a stack trace containing all exception details?

You also have hidden a second question in there related to performance:

The program should process 100000 saga instance per hour."

Without knowing any system details this is hard to answer. Are you currently having performance issues?

You are specifically stating saga instances which means about 25 instances need to be updated every second continuously. To achieve this you must make sure that your storage and network performance are sufficient.

arezoomand · December 29, 2018, 9:53am

The stack trace is:

2018-12-29 13:07:02.897 INFO NServiceBus.RecoverabilityExecutor Immediate Retry is going to retry message ‘0fc0b607-b176-44df-b116-067fd320b95b’ because of an exception:
System.Exception: Cannot publish a message with sequence number ‘8126’ on this channel. A message was already published on this channel with the same confirmation number.
at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector3 forkConnector, TForkContext context) at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.Invoke(ITransportReceiveContext context, Func2 next)
at NServiceBus.MainPipelineExecutor.Invoke(MessageContext messageContext)
at NServiceBus.Transport.RabbitMQ.MessagePump.Process(BasicDeliverEventArgs message)

Also, there is no problem with storage and network. The thing that should be considered is that we run several consumer to handle saga instances.

bording · January 7, 2019, 7:40pm

If you look at your log files, are there any other errors logged before the first Cannot publish a message with sequence number entry?

ramonsmits · January 21, 2019, 8:08am

@arezoomand have you resolved your issue?

mzrks · March 4, 2019, 9:10pm

I have the same problem. Under a high load, there are a lot of such errors. Maybe there are some suggestions about how to get rid of this problem?

2019-03-04 20:57:23.479 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message '0a00ff5d-4f3d-4ecf-bd84-aa0601569b24' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '180' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.479 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '2032' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()
2019-03-04 20:57:23.494 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message '0a00ff5d-4f3d-4ecf-bd84-aa0601569b24' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '699' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.494 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '426' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()
2019-03-04 20:57:23.494 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message '0a00ff5d-4f3d-4ecf-bd84-aa0601569b24' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '3364' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.494 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '452' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()
2019-03-04 20:57:23.494 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message '0a00ff5d-4f3d-4ecf-bd84-aa0601569b24' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '165' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.526 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message 'bccd49d4-c441-4a73-bf3f-aa0601569b35' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '448' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.526 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '176' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()
2019-03-04 20:57:23.541 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message 'bccd49d4-c441-4a73-bf3f-aa0601569b35' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '3391' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.541 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '2071' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()
2019-03-04 20:57:23.541 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message 'a42d132c-9811-49bd-a6c8-aa0601527530' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '2055' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.541 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '478' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()
2019-03-04 20:57:23.557 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message 'a42d132c-9811-49bd-a6c8-aa0601527530' after a delay of 00:05:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '3340' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.ForkExtensions.Fork[TFromContext,TToContext,TForkContext](IForkConnector`3 forkConnector, TForkContext context)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-04 20:57:23.557 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '3392' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()

bording · March 5, 2019, 3:39pm

@mzrks Same question for you. If you look at your log files for the first “Cannot publish a message” entry, do you have a different error logged right before it? If so, please post the details.

mzrks · March 6, 2019, 8:53am

There is only this in the logs.

    2019-03-06 08:50:09.193 WARN  NServiceBus.RecoverabilityExecutor Delayed Retry will reschedule message '7a867b9c-f1e1-46fd-8f4e-aa0701398f24' after a delay of 00:10:00 because of an exception:
System.Exception: Cannot publish a message with sequence number '7013' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.PublishMessage(Type type, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.PublishMessage(MulticastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at (Closure`2 , IDispatchContext )
   at NServiceBus.RoutingToDispatchConnector.Invoke(IRoutingContext context, Func`2 stage)
   at Core.Logging.NServiceBus.LoggingFlowHeaderOutgoingBehavior.Invoke(IOutgoingPhysicalMessageContext context, Func`1 next)
   at NServiceBus.SerializeMessageConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.DataBusSendBehavior.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Feed.Domain.Services.ScanFeedDomainService.<ProcessFeedItems>d__8.MoveNext() in J:\Jenkins\jobs\PT_Worker_Build\workspace\Feed\Mikz.PT.Feed.Domain\Services\ScanFeedDomainService.cs:line 236
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Feed.Domain.Services.ScanFeedDomainService.<InternalScanFeed>d__7.MoveNext() in J:\Jenkins\jobs\PT_Worker_Build\workspace\Feed\Feed.Domain\Services\ScanFeedDomainService.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.LoadHandlersConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.DataBusReceiveBehavior.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.DeserializeLogicalMessagesConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.UnitOfWorkBehavior.<InvokeUnitsOfWork>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at NServiceBus.UnitOfWorkBehavior.<InvokeUnitsOfWork>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.ProcessingStatisticsBehavior.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.MainPipelineExecutor.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
2019-03-06 08:50:10.251 WARN  NServiceBus.Transport.RabbitMQ.MessagePump Failed to process message. Returning message to queue...
System.Exception: Cannot publish a message with sequence number '2191' on this channel. A message was already published on this channel with the same confirmation number.
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.GetConfirmationTask()
   at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel.SendMessage(String address, OutgoingMessage message, IBasicProperties properties)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.SendMessage(UnicastTransportOperation transportOperation, ConfirmsAwareChannel channel)
   at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context)
   at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__27.MoveNext()

bording · March 6, 2019, 2:49pm

That is still a " Cannot publish a message with sequence number" error. You don’t have any other errors in your log?

mjohn · April 5, 2019, 8:50am

We encounter this problem under load when RabbitMQ is down or a network related problem. Connection is closed but channel. It may also related with RabbitMQ .Net Client. Maybe ModelShutdown event isn’t fired properly by RabbitMQ .Net Client. There are also messages in ConcurrentDictionary<ulong, TaskCompletionSource> object. Thats why we got that error message repeatedly.

bording · April 5, 2019, 8:19pm

@mjohn I’m aware of what the bad state inside the transport is, but the thing I’m still trying to narrow down is the exact sequence of events that can lead to the bad state.

Do you have log files for the endpoint that you could share? That’s what I really need to help me figure it out.

bording · May 9, 2019, 8:17pm

The patches that were just released should resolve this issue. See NServiceBus.RabbitMQ 5.1.2, 5.0.5, 4.4.5 - Patch releases available for more details.