Disclaimer: I’m an ops/support person who inherited this, and don’t quite understand how it all works and what is important data or isn’t.
Error and Audit ServiceControl Instances are v 4.26 for now due to Reasons.
Error instance RavenDB 3.5, Audit instance is v5.
Recently moved filesystem location of ServiceControl instance by the (Shutdown and check the DB state)->Export->Delete->Move->Re-Import method.
We noticed the next day that the SC error Queue (Azure ServiceBus Transport) is filling up with errors like:
Raven.Abstractions.Exceptions.ConcurrencyException:
PUT attempted on document 'SagaSnapshots/393027' using a non current etag
On the servicecontrol server, Windows Eventviewer shows them as well, I think the actual exception is in the SC instance?
Category: NServiceBus.RecoverabilityExecutor
EventId: 0
Moving message 'abcd123-f070-49c1-9afb-3361ba52efgh' to the error queue 'ACME.logging.production.Errors' because processing failed due to an exception:
Exception:
Raven.Abstractions.Exceptions.ConcurrencyException: PUT attempted on document 'SagaSnapshots/424411' using a non current etag
at Raven.Client.Connection.Async.AsyncServerClient.<>c__DisplayClass108_0.<<BatchAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Raven.Client.Connection.ReplicationInformerBase`1.<TryOperationAsync>d__34`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Our actual product messages are going through/getting processed just fine.
Our servicecontrol messages also seem to process fine (when the instance is running of course) but if we keep this going eventually the SC.errors queue will fill.
This is our Production SC instance, so we can’t just wipe and reset/re-create. (And it is rather large and probably configured incorrectly.)
Searching both the web at large, the RavenDB site and Particular.net hasn’t yielded much. Has anyone seen this before? Would upgrading the SC instances be likely to fix this? (Unfortunately this change was also made while deploying new code, I’m still awaiting confirmation that there was change made to the NSB hosts that might be the cause.)
If our SC instance (and queue) is named ACME.logging.production, then is there any harm in clearing out the messages in ACME.logging.production.errors?