Message failed at endpointQueue.Timeouts queue

[Nirav]

Hello,

facing one issue , below i have mentioned scenario for same.

NSB version : 5.2.14 , SQL Transport

Scenario :

1. Message failed first 3 attempt to endpointQueue, message moved to endpointQueue.Timeouts for Second level retry.

2019-11-13 01:21:06.9957 Warn NServiceBus.Faults.Forwarder.FaultManager Message with ‘3842CAC4-B9A0-8223-0DCC-509A6F75849B’ id has failed FLR and will be handed over to SLR for retry attempt 1.

2. It failed in endpointQueue.Timeouts queue with below message & dint try for Second level retry

2019-11-13 01:21:07.1643 Info NServiceBus.Unicast.Transport.TransportReceiver Failed to process message
NHibernate.StaleStateException: Batch update returned unexpected row count from update; actual row count: -1; expected: 1
at NHibernate.AdoNet.Expectations.VerifyOutcomeBatched(Int32 expectedRowCount, Int32 rowCount)
at NHibernate.AdoNet.SqlClientBatchingBatcher.DoExecuteBatch(IDbCommand ps)
at NHibernate.AdoNet.AbstractBatcher.ExecuteBatchWithTiming(IDbCommand ps)
at NHibernate.AdoNet.AbstractBatcher.ExecuteBatch()
at NHibernate.Engine.ActionQueue.ExecuteActions()
at NHibernate.Event.Default.AbstractFlushingEventListener.PerformExecutions(IEventSource session)
at NHibernate.Event.Default.DefaultFlushEventListener.OnFlush(FlushEvent event)
at NHibernate.Impl.SessionImpl.Flush()
at NServiceBus.Persistence.NHibernate.AmbientTransactionAwareWrapper.Commit() in C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate\Internal\AmbientTransactionAwareWrapper.cs:line 49
at NServiceBus.TimeoutPersisters.NHibernate.TimeoutPersister.StoreTimeoutEntity(TimeoutData timeout, IDbConnection connection, Guid timeoutId) in C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate\TimeoutPersisters\TimeoutPersister.cs:line 128
at NServiceBus.TimeoutPersisters.NHibernate.TimeoutPersister.Add(TimeoutData timeout) in C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate\TimeoutPersisters\TimeoutPersister.cs:line 98
at NServiceBus.Timeout.Core.DefaultTimeoutManager.PushTimeout(TimeoutData timeout) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Timeout\Core\DefaultTimeoutManager.cs:line 26
at NServiceBus.Timeout.Hosting.Windows.TimeoutMessageProcessor.HandleInternal(TransportMessage message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Timeout\Hosting\Windows\TimeoutMessageProcessor.cs:line 134
at NServiceBus.Timeout.Hosting.Windows.TimeoutMessageProcessor.Handle(TransportMessage message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Timeout\Hosting\Windows\TimeoutMessageProcessor.cs:line 60
at NServiceBus.Satellites.SatelliteLauncher.HandleMessageReceived(Object sender, TransportMessageReceivedEventArgs e, ISatellite satellite) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Satellites\SatelliteLauncher.cs:line 89
at NServiceBus.Unicast.Transport.TransportReceiver.OnTransportMessageReceived(TransportMessage msg) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 413
at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage(TransportMessage message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 344
at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 230
at NServiceBus.Transports.SQLServer.AmbientTransactionReceiveStrategy.TryReceiveFrom(TableBasedQueue queue) in C:\BuildAgent\work\888dd942b44bad5a\src\NServiceBus.SqlServer\AmbientTransactionReceiveStrategy.cs:line 57

[Nirav]

can see post on similar error at NHibernate.StaleStateException: Batch update returned unexpected row count from update; actual row count: -1; expected: 1 - #6 by ramonsmits

NOCOUNT set to OFF , still getting this error

[ramonsmits]

It’s clear that NOCOUNT is still ON. Did you really set it to OFF at the Server?

Alternatively, you can use a custom connection factory as mentioned here:

NHibernate.StaleStateException: Batch update returned unexpected row count from update; actual row count: -1; expected: 1

Can you check if you have NOCOUNT on or off on the SQL server instance? NHibernate requires this to be OFF so that it gets the row count returned. If this is not possible then you can customize the connection factory used by NHibernate to disable this. The code for this can be found at the following issue: Make usage of NHibernate compatible with `SET NOCOUNT ON` DB-wide setting · Issue #112 · Particular/NServiceBus.NHibernate · GitHub Code: public class CustomDriverConnectionProvider : NHi…

[ramonsmits]

Also, you are running an older versions, I advise you to upgrade to at least:

• NuGet Gallery | NServiceBus 5.2.26
• NuGet Gallery | NServiceBus.SqlServer 2.2.6

I didn’t notice earlier that you are using SQL transport. You should be able to set a custom connection factory there for the SQL transport:

• Connection Settings • SQL Server Transport • Particular Docs

[Nirav]

Thank you, provided solution has resolved the problem.

Connection Settings • SQL Server Transport