Hi
Our function app using NServiceBus using Azure Servicebus as transport has lately start throwing NullReferenceExceptions (see call stack below). We can not found any reason for this behaviour. The format of the messages has not been changed and we do not have updated any NServicebus assemblies for a while. The version we are using are
NServiceBus 7.8.0
NServiceBus.AzureFunctions.InProcess.ServiceBus 3.0.0
There is also a lot of warnings like this one:
A duplicate diagnostic entry was renamed from Hosting to Hosting-2.
Any ideaas of what this could be?
Kind regards
Lars Kolsaker
Noa Ignite AS
Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
at Microsoft.Azure.WebJobs.ServiceBus.MessageProcessor.CompleteProcessingMessageAsync (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.2.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener+d__26.MoveNext (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.2.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Messaging.ServiceBus.ServiceBusProcessor+d__104.MoveNext (Azure.Messaging.ServiceBus, Version=7.5.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Messaging.ServiceBus.ReceiverManager+d__16.MoveNext (Azure.Messaging.ServiceBus, Version=7.5.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Messaging.ServiceBus.ReceiverManager+d__15.MoveNext (Azure.Messaging.ServiceBus, Version=7.5.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception System.NullReferenceException handled at Microsoft.Azure.WebJobs.ServiceBus.MessageProcessor.CompleteProcessingMessageAsync:
at NServiceBus.SendComponent+<>c__DisplayClass1_0.b__1 (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: //src/NServiceBus.Core/Pipeline/Outgoing/SendComponent.cs:35)
at NServiceBus.Extensions.DependencyInjection.ServiceCollectionAdapter+<>c__DisplayClass3_01.<ConfigureComponent>b__1 (NServiceBus.Extensions.DependencyInjection, Version=1.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c) at DryIoc.Microsoft.DependencyInjection.DryIocAdapter+<>c__DisplayClass3_0.<RegisterDescriptor>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/DryIocAdapter.cs:162) at DryIoc.Registrator+<>c__DisplayClass27_0.<RegisterDelegate>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:4550) at lambda_method762 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null) at DryIoc.Factory+<>c__DisplayClass26_0.<ApplyReuse>b__2 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6605) at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7849) at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7834) at DryIoc.Factory.ApplyReuse (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6604) at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6564) at DryIoc.Factory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6634) at DryIoc.DelegateFactory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7740) at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:210) at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:195) at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:99) at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.21.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:77) at NServiceBus.Extensions.DependencyInjection.ServiceProviderAdapter.Build (NServiceBus.Extensions.DependencyInjection, Version=1.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c) at NServiceBus.Extensions.DependencyInjection.ServiceProviderAdapter.Build (NServiceBus.Extensions.DependencyInjection, Version=1.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c) at NServiceBus.SendComponent+<>c.<Initialize>b__1_0 (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Pipeline/Outgoing/SendComponent.cs:31) at NServiceBus.Pipeline.PipelineSettings+<>c__DisplayClass8_01.b__0 (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: //src/NServiceBus.Core/Pipeline/PipelineSettings.cs:141)
at NServiceBus.Pipeline.RegisterStep.CreateBehavior (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Pipeline/RegisterStep.cs:143)
at NServiceBus.Pipeline1+<>c__DisplayClass0_0.<.ctor>b__0 (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Pipeline/Pipeline.cs:25) at System.Linq.Enumerable+SelectListIterator2.ToArray (System.Linq, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Linq.Enumerable.ToArray (System.Linq, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at NServiceBus.Pipeline1..ctor (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Pipeline/Pipeline.cs:18) at NServiceBus.SendComponent.CreateMessageOperations (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Pipeline/Outgoing/SendComponent.cs:62) at NServiceBus.StartableEndpoint+<Start>d__1.MoveNext (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/StartableEndpoint.cs:44) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at NServiceBus.HostingComponent+<Start>d__7.MoveNext (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Hosting/HostingComponent.cs:76) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at NServiceBus.ExternallyManagedContainerHost+<Start>d__9.MoveNext (NServiceBus.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.Core/Hosting/ExternallyManagedContainerHost.cs:49) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at NServiceBus.InProcessFunctionEndpoint+<InitializeEndpointIfNecessary>d__19.MoveNext (NServiceBus.AzureFunctions.InProcess.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.AzureFunctions.InProcess.ServiceBus/InProcessFunctionEndpoint.cs:249) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at NServiceBus.InProcessFunctionEndpoint+<ProcessNonAtomic>d__13.MoveNext (NServiceBus.AzureFunctions.InProcess.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c: /_/src/NServiceBus.AzureFunctions.InProcess.ServiceBus/InProcessFunctionEndpoint.cs:115) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2+d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.37.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:20)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2+d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.37.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.37.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.37.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.37.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)