Microsoft.WindowsAzure.Storage.StorageException: Element 0 in the batch returned an unexpected response code

Luke_Perrin · April 18, 2018, 12:26pm

I get this error, fairly regularly, please see attached for full stack trace.

I am wondering if this is a bug with nservicebus to do with the azure storage?
My versions are

Nservicebus 6.4.3
Nservicebus Unity v7 (I tried updating this to 8.1.0 and got no end of issues with all the dependencies asp.net.unity, asp.net.mvc.unity threw errors etc… so I rolled back…)
Nservicebus.Azure.Transports.WindowsAzureServiceBus 7.2.11
Nservicebus.DataBus.AzureBlobStorage 1.1.0
Nservicebus.Newtonsoft.Json 1.1.0 1.2 requires Newtonsoft.Json (>= 11.0.1 && < 12.0.0
Newtonsoft.Json 10.0.3 tried installing 11.0.2 and I can’t because its not compatible with NserviceBus.Persistance.AzureStorage 1.4.1
Nservicebus.Persistance.AzureStoage 1.4.1

I have 2 separate azure namespaces test and local for the transport and they both use their own azure storage also called test and local.
I use test when I publish to azure otherwise I use local when I run it all on my machine.

I have 2 endpoints, website and itops, website is starting a saga which is handled by itops which is hosted in a webjob. It seems if I trigger an error on purpose to test the error handling then later I get a further error… and it seems to be a problem with actually moving a message to the message queue.

This is the stack trace…

2018-04-18 10:31:29.754 ERROR NServiceBus.RecoverabilityExecutor Moving message ‘0e09fb70-4001-4fb9-8b7b-a8c6009a310b’ to the error queue ‘error’ because processing failed due to an exception:
Microsoft.WindowsAzure.Storage.StorageException: Element 0 in the batch returned an unexpected response code.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Table.CloudTable.EndExecuteBatch(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Table\CloudTable.cs:line 208
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Persistence.AzureStorage.AzureSagaPersister.d__10.MoveNext() in C:\BuildAgent\work\ec133c968ad4248a\src\NServiceBus.Persistence.AzureStorage\SagaPersisters\AzureSagaPersister.cs:line 173
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at NServiceBus.SagaPersistenceBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Sagas\SagaPersistenceBehavior.cs:line 168
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.LoadHandlersConnector.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\Incoming\LoadHandlersConnector.cs:line 40
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.DataBusReceiveBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\DataBus\DataBusReceiveBehavior.cs:line 63
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.InvokeSagaNotFoundBehavior.d__0.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Sagas\InvokeSagaNotFoundBehavior.cs:line 16
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MutateIncomingMessageBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\MessageMutators\MutateInstanceMessage\MutateIncomingMessageBehavior.cs:line 44
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.DeserializeLogicalMessagesConnector.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\Incoming\DeserializeLogicalMessagesConnector.cs:line 33
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.UnitOfWorkBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\UnitOfWork\UnitOfWorkBehavior.cs:line 40
— End of stack trace from previous location where exception was thrown —
at NServiceBus.UnitOfWorkBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\UnitOfWork\UnitOfWorkBehavior.cs:line 62
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ReceivePerformanceDiagnosticsBehavior.d__2.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Performance\Statistics\ReceivePerformanceDiagnosticsBehavior.cs:line 40
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MutateIncomingTransportMessageBehavior.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\MessageMutators\MutateTransportMessage\MutateIncomingTransportMessageBehavior.cs:line 43
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ProcessingStatisticsBehavior.d__0.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Performance\Statistics\ProcessingStatisticsBehavior.cs:line 27
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.AzureServiceBus.TransactionScopeSuppressBehavior.d__0.MoveNext() in C:\BuildAgent\work\7eae8ccbde2584d7\src\Transport\Receiving\TransactionScopeSuppressBehavior.cs:line 16
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\Incoming\TransportReceiveToPhysicalMessageProcessingConnector.cs:line 37
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MainPipelineExecutor.d__1.MoveNext() in C:\BuildAgent\work\a93f853f0c1b9532\src\NServiceBus.Core\Pipeline\MainPipelineExecutor.cs:line 32
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureServiceBus.MessagePump.<b__3_1>d.MoveNext() in C:\BuildAgent\work\7eae8ccbde2584d7\src\Transport\Seam\MessagePump.cs:line 60
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at NServiceBus.Transport.AzureServiceBus.MessageReceiverNotifier.d__13.MoveNext() in C:\BuildAgent\work\7eae8ccbde2584d7\src\Transport\Receiving\MessageReceiverNotifier.cs:line 227
Request Information
RequestID:2c3ae27b-e002-00b2-76f8-d6239c000000
RequestDate:Wed, 18 Apr 2018 09:31:29 GMT
StatusMessage:The update condition specified in the request was not satisfied.
ErrorCode:UpdateConditionNotSatisfied
ErrorMessage:The update condition specified in the request was not satisfied.
RequestId:2c3ae27b-e002-00b2-76f8-d6239c000000
Time:2018-04-18T09:31:29.8608998Z

SeanFeldman · April 23, 2018, 5:57am

Tracking this via support case and direct communication.

Nate · May 21, 2018, 3:06pm

Hi,
Have you had any success tracking this down? We just starting using the Storage Tables for sagas and are getting lots of these.
Thanks,
Nate

Nate · May 21, 2018, 5:51pm

We are speculating that its a concurrency thing. We have a command that kicks off 100s of other smaller commands. Each of which update the saga. Could they be trying to update the saga data only to find out their copy is stale?

SeanFeldman · May 22, 2018, 5:43am

This is a slightly different scenario.
If you have a saga with 100s of concurrent commands and they all try to write to the state, it will lead to concurrency issues.

Avalaxy · June 13, 2020, 11:12pm

@SeanFeldman why is that?

I’m running into this same issue. I have a saga that spawns a bunch of commands (and keeps track of them in a List with their command Ids). Each command results in an event when it’s processed. When all commands are processed, the saga is marked as completed.

This randomly results in the same error: Element 0 in the batch returned an unexpected response code.

SeanFeldman · June 16, 2020, 4:17am

Due to the nature of the optimistic concurrency of this persister.

ramonsmits · May 26, 2021, 12:54pm

This very likely is the case and this is optimistic concurrency control validating if it can update the data server side if you are getting this exception with the following text at the end:

StatusMessage:The update condition specified in the request was not satisfied.
ErrorCode:UpdateConditionNotSatisfied
ErrorMessage:The update condition specified in the request was not satisfied.

Azure Storage does not allow transactions or locking thus uses optimistic concurrency control to guarantee consistency of saga state.

If multiple threads fetch the same data item and then try to update only a single thread wins. The other will fail with this error and will need to retry processing. This is as expected and by design to prevent corruption.

We have a full page dedicated to saga concurrency with many tips: