Microsoft.WindowsAzure.Storage.StorageException: Element 0 in the batch returned an unexpected response code

nservicebus
azurepersistence
sagas

(Luke Perrin) #1

I get this error, fairly regularly, please see attached for full stack trace.

I am wondering if this is a bug with nservicebus to do with the azure storage?
My versions are

Nservicebus 6.4.3
Nservicebus Unity v7 (I tried updating this to 8.1.0 and got no end of issues with all the dependencies asp.net.unity, asp.net.mvc.unity threw errors etc… so I rolled back…)
Nservicebus.Azure.Transports.WindowsAzureServiceBus 7.2.11
Nservicebus.DataBus.AzureBlobStorage 1.1.0
Nservicebus.Newtonsoft.Json 1.1.0 1.2 requires Newtonsoft.Json (>= 11.0.1 && < 12.0.0
Newtonsoft.Json 10.0.3 tried installing 11.0.2 and I can’t because its not compatible with NserviceBus.Persistance.AzureStorage 1.4.1
Nservicebus.Persistance.AzureStoage 1.4.1

I have 2 separate azure namespaces test and local for the transport and they both use their own azure storage also called test and local.
I use test when I publish to azure otherwise I use local when I run it all on my machine.

I have 2 endpoints, website and itops, website is starting a saga which is handled by itops which is hosted in a webjob. It seems if I trigger an error on purpose to test the error handling then later I get a further error… and it seems to be a problem with actually moving a message to the message queue.

This is the stack trace…

2018-04-18 10:31:29.754 ERROR NServiceBus.RecoverabilityExecutor Moving message ‘0e09fb70-4001-4fb9-8b7b-a8c6009a310b’ to the error queue ‘error’ because processing failed due to an exception:
Microsoft.WindowsAzure.Storage.StorageException: Element 0 in the batch returned an unexpected response code.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Table.CloudTable.EndExecuteBatch(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Table\CloudTable.cs:line 208
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Persistence.AzureStorage.AzureSagaPersister.d__10.MoveNext() in C:\BuildAgent\work\ec133c968ad4248a\src\NServiceBus.Persistence.AzureStorage\SagaPersisters\AzureSagaPersister.cs:line 173
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at NServiceBus.SagaPersistenceBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Sagas\SagaPersistenceBehavior.cs:line 168
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.LoadHandlersConnector.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\Incoming\LoadHandlersConnector.cs:line 40
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.DataBusReceiveBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\DataBus\DataBusReceiveBehavior.cs:line 63
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.InvokeSagaNotFoundBehavior.d__0.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Sagas\InvokeSagaNotFoundBehavior.cs:line 16
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MutateIncomingMessageBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\MessageMutators\MutateInstanceMessage\MutateIncomingMessageBehavior.cs:line 44
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.DeserializeLogicalMessagesConnector.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\Incoming\DeserializeLogicalMessagesConnector.cs:line 33
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.UnitOfWorkBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\UnitOfWork\UnitOfWorkBehavior.cs:line 40
— End of stack trace from previous location where exception was thrown —
at NServiceBus.UnitOfWorkBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\UnitOfWork\UnitOfWorkBehavior.cs:line 62
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ReceivePerformanceDiagnosticsBehavior.d__2.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Performance\Statistics\ReceivePerformanceDiagnosticsBehavior.cs:line 40
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MutateIncomingTransportMessageBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\MessageMutators\MutateTransportMessage\MutateIncomingTransportMessageBehavior.cs:line 43
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ProcessingStatisticsBehavior.d__0.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Performance\Statistics\ProcessingStatisticsBehavior.cs:line 27
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.AzureServiceBus.TransactionScopeSuppressBehavior.d__0.MoveNext() in C:\BuildAgent\work\7eae8ccbde2584d7\src\Transport\Receiving\TransactionScopeSuppressBehavior.cs:line 16
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\Incoming\TransportReceiveToPhysicalMessageProcessingConnector.cs:line 37
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MainPipelineExecutor.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\MainPipelineExecutor.cs:line 32
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureServiceBus.MessagePump.<b__3_1>d.MoveNext() in C:\BuildAgent\work\7eae8ccbde2584d7\src\Transport\Seam\MessagePump.cs:line 60
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at NServiceBus.Transport.AzureServiceBus.MessageReceiverNotifier.d__13.MoveNext() in C:\BuildAgent\work\7eae8ccbde2584d7\src\Transport\Receiving\MessageReceiverNotifier.cs:line 227
Request Information
RequestID:2c3ae27b-e002-00b2-76f8-d6239c000000
RequestDate:Wed, 18 Apr 2018 09:31:29 GMT
StatusMessage:The update condition specified in the request was not satisfied.
ErrorCode:UpdateConditionNotSatisfied
ErrorMessage:The update condition specified in the request was not satisfied.
RequestId:2c3ae27b-e002-00b2-76f8-d6239c000000
Time:2018-04-18T09:31:29.8608998Z


Microsoft.WindowsAzure.Storage.StorageException when starting a saga
(Sean Feldman) #2

Tracking this via support case and direct communication.


#3

Hi,
Have you had any success tracking this down? We just starting using the Storage Tables for sagas and are getting lots of these.
Thanks,
Nate


#4

We are speculating that its a concurrency thing. We have a command that kicks off 100s of other smaller commands. Each of which update the saga. Could they be trying to update the saga data only to find out their copy is stale?


(Sean Feldman) #5

This is a slightly different scenario.
If you have a saga with 100s of concurrent commands and they all try to write to the state, it will lead to concurrency issues.