We’re in the middle of getting Outbox+TransactionalSession up and running in our system, but have met a weird quirk I struggle to wrap my head around.
It seems that the first transactionalSession.Commit() that is done after app start consistently fails with Violation of UNIQUE KEY constraint ‘UQ__OutboxRe__C87C0C9DEBADFCE9’. Cannot insert duplicate key in object ‘nsb.OutboxRecord’. (Full error below). Any subsequent transactions seem to work fine in my current test setup, although we do have occasional failures in full test and production environments that are logged too.
Our code is quite straightforward, and essentially looks like this:
using (var scope = _serviceProvider.CreateScope())
{
var transactionalSession = scope.ServiceProvider.GetService<ITransactionalSession>();
transactionalSession.Open().Wait();
dynamic handler = FindHandlerForCommand(command);
handler.Handle(command as dynamic); // Magic happens inside
transactionalSession.Commit().Wait();
}
The full error is the following, and seems to indicate something about an internal message - not anything we’re sending/publishing:
info: NServiceBus.TransactionalSession.TransactionalSessionDelayControlMessageBehavior[0]
Consuming transaction commit control messages for messageId=8efdd736-6881-4d55-a94a-5ce5a4078398 to create the outbox tomb stone.
[10:44:33 INF] Consuming transaction commit control messages for messageId=8efdd736-6881-4d55-a94a-5ce5a4078398 to create the outbox tomb stone.
fail: MyService.Infrastructure.Commands.CommandExecutor[0]
Unexpected error when executing Subscribe
System.AggregateException: One or more errors occurred. ---> NHibernate.Exceptions.GenericADOException: could not insert: [NServiceBus.Outbox.NHibernate.OutboxRecord][SQL: INSERT INTO nsb.OutboxRecord (MessageId, Dispatched, DispatchedAt, TransportOperations) VALUES
(?, ?, ?, ?); select SCOPE_IDENTITY()] ---> System.Data.SqlClient.SqlException: Violation of UNIQUE KEY constraint 'UQ__OutboxRe__C87C0C9DEBADFCE9'. Cannot insert duplicate key in object 'nsb.OutboxRecord'. The duplicate key value is (MyService/8efdd736-
6881-4d55-a94a-5ce5a4078398).
The statement has been terminated.
Any ideas to what might be happening, or why?
Rewriting to async/await is not on the table for now across our services. It’s not optimal with .Wait(), but i don’t think that’s a cause here anyways.
We’re on NServiceBus 7.8.1, NServiceBus.NHibernate (and .TransactionalSession) 8.6.3, NServiceBus.Transport.Msmq 1.2.1
From the code you’ve provided, it’s not really clear what might be going on. Is your “magic” section even using the transactional session object? From the code snippet it doesn’t appear to be.
However, one thing you really should not be doing is making this code synchronous by calling Wait(). It’s not just that it’s not optimal, you can actually run into problems the code paths operating correctly by doing that.
Regardless, we’re going to need to see a more complete example of your code to be able to help you further.
The “magic” that happens inside the handlers is quite basic: Read objects from the transactional session, modify them, and send a message. The reason why I did not include it all, is because this is a generic handler for many types of commands, so some of them doesn’t do much, while others are more complex with business logic.
I suppose you’re right about the .Wait() - we’ll give rewriting to async/await a go for this service we currently have at hand and we’ll see if that helps. Will report back when we see the results for that!
EDIT: The problem does occur even if no messages are sent. Removed false statement above. Essentially, the code below is all that happens:
Rewrote this one to be async/await all the way - the problem still persists. First .Commit() on the transactional session fails. Full stack trace is added below (“Subscribe” is the name of the command):
Unexpected error when executing Subscribe
NHibernate.Exceptions.GenericADOException: could not insert: [NServiceBus.Outbox.NHibernate.OutboxRecord][SQL: INSERT INTO nsb.OutboxRecord (MessageId, Dispatched, DispatchedAt, TransportOperations) VALUES (?, ?, ?, ?); select SCOPE_IDENTITY()] —> System.Data.SqlClient.SqlException: Violation of UNIQUE KEY constraint ‘UQ__OutboxRe__C87C0C9DEBADFCE9’. Cannot insert duplicate key in object ‘nsb.OutboxRecord’. The duplicate key value is (ContentSubscriptionService/dee66b43-1e7a-46ca-9640-23bdf6c6d321).
The statement has been terminated.
at System.Data.SqlClient.SqlCommand.<>c.b__180_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.AdoNet.AbstractBatcher.d__72.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Id.IdentityGenerator.InsertSelectDelegate.d__1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Id.Insert.AbstractReturningDelegate.d__0.MoveNext()
— End of inner exception stack trace —
at NHibernate.Id.Insert.AbstractReturningDelegate.d__0.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Persister.Entity.AbstractEntityPersister.d__17.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Action.EntityIdentityInsertAction.d__17.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Engine.ActionQueue.d__3.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Engine.ActionQueue.d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NHibernate.Engine.ActionQueue.d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Event.Default.AbstractSaveEventListener.d__3.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Event.Default.AbstractSaveEventListener.d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Event.Default.AbstractSaveEventListener.d__1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.d__0.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NHibernate.Impl.SessionImpl.d__93.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NHibernate.Impl.SessionImpl.d__1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.TransactionalSession.OutboxTransactionalSession.d__1.MoveNext() in //src/NServiceBus.TransactionalSession/OutboxTransactionalSession.cs:line 57
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.TransactionalSession.TransactionalSessionBase.d__9.MoveNext() in //src/NServiceBus.TransactionalSession/TransactionalSessionBase.cs:line 64
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at .Infrastructure.Commands.CommandExecutor.d__3.MoveNext() in C:\Arbeid\etoto\code*\src**.Infrastructure\Commands\CommandExecutor.cs:line 45
At this point, I think we’re really going to need to see more of your code to understand what you’re trying to do and what problem you’re running into.
To make it easier for you to share more of your code and get into a more involved discussion of your scenario, please go to Support options • Particular Software and open a support case.
Yess, a support case will come. I’ll spend a few hours slimming down the code base and making a minimum reproducible case, and send one. Thanks for the help so far!
Was there any resolution to this? We are running into the same (or a very similar) issue using NSB 9.1. Our application was running smoothly, ran into the same exception, and now it fails on the first attempt to commit the transactional session on subsequent runs.
It is not clear to me yet how I would create a reproduction of this. Some information that will probably be helpful:
We are using NSB 9.1.
We are using the Outbox and Transactional Session.
We are using SQL persistence with EF Core 8.0 against a Postgresql database.
We are running in a local development environment, which uses LocalStack as an AWS emulator.
The issue is sporadic and rare, but once it occurs, we can’t seem to escape it without tearing down the full application. It just occurred to me that this is probably a clue, since this stops LocalStack, which would empty the queues.
Our internal issue number was 00082651 if anyone from Particular needs for context. Not really too much info there, since the main issue is described in the Github issue above.