Microsoft.WindowsAzure.Storage.StorageException when starting a saga

nservicebus

(Bruno) #1

Hi.
Sorry if this seems like a duplicate thread but we’re seeing lots of storage exceptions in our backend and are looking for a solution or explanation.

This is the exception stack trace:

Microsoft.WindowsAzure.Storage.StorageException: Element 0 in the batch returned an unexpected response code.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 60
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass11.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at NServiceBus.Persistence.AzureStorage.AzureSagaPersister.d__10.MoveNext() in C:\BuildAgent\work\46ec1c2daf2d23b9\src\NServiceBus.Persistence.AzureStorage\SagaPersisters\AzureSagaPersister.cs:line 173
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.SagaPersistenceBehavior.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Sagas\SagaPersistenceBehavior.cs:line 170
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceControl.Plugin.SagaAudit.CaptureSagaStateBehavior.d__5.MoveNext() in C:\BuildAgent\work\83542c77b87f97e8\src\ServiceControl.Plugin.Nsb6.SagaAudit\CaptureSagaStateBehavior.cs:line 41
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.LoadHandlersConnector.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\Incoming\LoadHandlersConnector.cs:line 40
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.DataBusReceiveBehavior.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\DataBus\DataBusReceiveBehavior.cs:line 63
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.InvokeSagaNotFoundBehavior.d__0.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Sagas\InvokeSagaNotFoundBehavior.cs:line 16
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.DeserializeLogicalMessagesConnector.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\Incoming\DeserializeLogicalMessagesConnector.cs:line 33
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.InvokeAuditPipelineBehavior.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Audit\InvokeAuditPipelineBehavior.cs:line 18
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.SubscriptionReceiverBehavior.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Routing\MessageDrivenSubscriptions\SubscriptionReceiverBehavior.cs:line 29
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ReceivePerformanceDiagnosticsBehavior.d__2.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Performance\Statistics\ReceivePerformanceDiagnosticsBehavior.cs:line 44
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.ProcessingStatisticsBehavior.d__0.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Performance\Statistics\ProcessingStatisticsBehavior.cs:line 27
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\Incoming\TransportReceiveToPhysicalMessageProcessingConnector.cs:line 37
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.MainPipelineExecutor.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\MainPipelineExecutor.cs:line 32
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.AzureStorageQueues.AtLeastOnceReceiveStrategy.d__1.MoveNext() in C:\BuildAgent\work\4e5353dd260f0a07\src\Transport\AtLeastOnceReceiveStrategy.cs:line 27
Request Information
RequestID:2e2e898a-b002-0009-43b5-5819bc000000
RequestDate:Sun, 30 Sep 2018 12:00:01 GMT
StatusMessage:The update condition specified in the request was not satisfied.
ErrorCode:UpdateConditionNotSatisfied
ErrorMessage:The update condition specified in the request was not satisfied.
RequestId:2e2e898a-b002-0009-43b5-5819bc000000
Time:2018-09-30T12:00:01.8424052Z

Our current setup is:
Newtonsoft.Json 10.0.1
NServiceBus 6.3.4
NServiceBus.Azure.Transports.WindowsAzureServiceBus 7.2.6
NServiceBus.Azure.Transports.WindowsAzureStorageQueues 7.4.0
NServiceBus.DataBus.AzureBlobStorage 1.1.0
NServiceBus.Host 7.0.2
NServiceBus.Persistence.AzureStorage 1.4.0

Reproducing the issue:
The only way that I can reproduce this error in code is by sending multiple similar messages almost at the same time to start a given saga. In production, I imagine the same happens when multiple users work on the same record at the same time.

My questions are:

  • is it normal to see so many storage exceptions in our backends?
  • if this is normal, any way we can mitigate the issue?
  • since on the other thread it was mentioned that concurrent issues could happen, what is the recommendation here?

Thanks.


(Yves Goeleven) #2

This is indeed pointing at a concurrency problem, where 2 different messages handled result in changes on the same saga record. Then the last handler to commit will get this exception, rollback and try again. These exceptions are normal, and you should not really be worried about them, because of the retry the data will be consistent anyway. If you are overwhelmed by them, you should consider redesigning your process so that it becomes less concurrent.