The process was terminated due to an unhandled exception

1

We are using
NSB5,
NserviceBus.Host 6
Transport - Sql server
Persistence - Nhibernate with Sql

We cloned the servers with same AD group, also other endpoints on these clone server are running fine. Below is the issue got logged in the event viewer we got while starting the service.

Log Name:      Application
Source:        .NET Runtime
Date:          7/28/2019 2:45:58 AM
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xxx.xxx.com
Description:
Application: NServiceBus.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Data.SqlClient.SqlException
   at System.Data.SqlClient.SqlConnection.OnError(System.Data.SqlClient.SqlException, Boolean, System.Action`1<System.Action>)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(System.Data.SqlClient.TdsParserStateObject, Boolean, Boolean)
   at System.Data.SqlClient.TdsParser.TryRun(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject, Boolean ByRef)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.RunBehavior, System.String, Boolean, Boolean, Boolean)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean, Int32, System.Threading.Tasks.Task ByRef, Boolean, Boolean, System.Data.SqlClient.SqlDataReader, Boolean)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Threading.Tasks.TaskCompletionSource`1<System.Object>, Int32, System.Threading.Tasks.Task ByRef, Boolean ByRef, Boolean, Boolean)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(System.Threading.Tasks.TaskCompletionSource`1<System.Object>, System.String, Boolean, Int32, Boolean ByRef, Boolean, Boolean)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at NServiceBus.Transports.SQLServer.TableBasedQueue.Send(System.Object[], System.Data.SqlClient.SqlConnection, System.Data.SqlClient.SqlTransaction)
   at NServiceBus.Transports.SQLServer.SqlServerMessageSender.Send(NServiceBus.TransportMessage, NServiceBus.Unicast.SendOptions)

Exception Info: System.Exception
   at NServiceBus.Transports.SQLServer.SqlServerMessageSender.ThrowFailedToSendException(NServiceBus.Address, System.Exception)
   at NServiceBus.Transports.SQLServer.SqlServerMessageSender.Send(NServiceBus.TransportMessage, NServiceBus.Unicast.SendOptions)
   at NServiceBus.Unicast.Subscriptions.MessageDrivenSubscriptions.SubscriptionManager.SendSubscribeMessageWithRetries(NServiceBus.Address, NServiceBus.TransportMessage, System.String, Int32)
   at NServiceBus.Unicast.Subscriptions.MessageDrivenSubscriptions.SubscriptionManager+<>c__DisplayClass12_0.<Subscribe>b__0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

What we have tried

  1. Restart all the endpoints to resubscribe
  2. Check log files but nothing found in it

Observation - It runs for 1 or 2 minutes and then stop with above exception in event viewer

Any help on this is appreciated.

2

Hi Raj

NServiceBus is now out of general support but looking at the code here it seems that the process is killed because in V5 only the QueueNotFoundException was handled properly. In case of any other exception, it is propagated to the calling code (which in this case is the Thread Pool) and the process is killed.

I suspect there is some connectivity issue with the SQL Server database here but in order to diagnose it you have two options:

  • connect the debugger and put a breakpoint in SendSubscribeMessageWithRetries
  • modify the code in SendSubscribeMessageWithRetries to catch all types of exceptions rather than only QueueNotFound and at least log the exception details. Then build and run your custom version of NServiceBus. Note: after diagnosing the problem you can revert back to the official package.

Szymon

3

Hi @SzymonPobiega - Thanks for the response. Actually these are the production servers where I am not able to debug. Also this is happening with only this endpoint whereas, other endpoints are running fine.

4

You could probably use the SQL Profiler and log what is happening during that query on the database and find out the potential cause. Maybe you have a permission issue, a table doesn’t exist, or a timeout issue.