NServiceBus.FunctionEndpoint.Process return NullReferenceException

1

I’m running AzureFunction v4 .NET6 isolated as a host (with Service Bus as transport) on NSB 8.0.1 and 4.0.0

Sometimes I’m getting:
Exception while executing function: Functions.NServiceBusTriggerFunction Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException

Looking at the aggregated exceptions it is a NullReferenceException from with
NServiceBus.FunctionEndpoint.Process. Within its exception handeling it is throwing this.
Any idea what is causing this?

image
image1159×739 113 KB 

Exception while executing function: Functions.NServiceBusTriggerFunction Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at NServiceBus.FunctionEndpoint.Process(Byte[] body, IDictionary`2 userProperties, String messageId, Int32 deliveryCount, String replyTo, String correlationId, ITransactionStrategy transactionStrategy, PipelineInvoker pipeline, CancellationToken cancellationToken) in /_/src/NServiceBus.AzureFunctions.Worker.ServiceBus/FunctionEndpoint.cs:line 84
   at NServiceBus.FunctionEndpoint.Process(Byte[] body, IDictionary`2 userProperties, String messageId, Int32 deliveryCount, String replyTo, String correlationId, FunctionContext functionContext, CancellationToken cancellationToken) in /_/src/NServiceBus.AzureFunctions.Worker.ServiceBus/FunctionEndpoint.cs:line 41
   at FunctionEndpointTrigger.Run(Byte[] messageBody, IDictionary`2 userProperties, String messageId, Int32 deliveryCount, String replyTo, String correlationId, FunctionContext context) in D:\a\SpotlerAutomate.Service\SpotlerAutomate.Service\src\AV.Spotler.FunctionHost\NServiceBus.AzureFunctions.Worker.SourceGenerator\NServiceBus.AzureFunctions.SourceGenerator.TriggerFunctionGenerator\NServiceBus__FunctionEndpointTrigger.cs:line 27
   at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82
2

Hi

Thanks for reporting this. I have some questions that could help us find the issue. How often does it happen? Is this something that corrupts the host until it is killed and restarted or subsequent calls succeed? Have you seen this with previous version?

Szymon

3

Frequency around once every couple of days.

It doesn’t kill the host. In my case I see it happening when a handler executes a http call where a 403 ar 408 is given.

I see retries happening and in the end the call is succeeded into a 200.

image
image922×593 45.1 KB

4

Thanks. Do you know if these errors are correlated in time with the duplicate diagnostic entries you reported here A duplicate diagnostic entry was renamed from Azure Service Bus transport to Azure Service Bus transport-X ?

I am asking because we are investigating a possibility of a defect in the code that assigns the singleton instance of the processing pipeline.

5

Yes, it is correlated in time with my other issue about duplicate diagnostic entries.

But those duplicate diagnostic entries also happen in other situations, that I’m still trying to analyze.

6

fix by answer in With NSB 8 I'm getting ServiceBusException: Cannot allocate more handles. The maximum number of handles is 255 - #4 by danielmarbach