We are currently experiencing a slightly strange error in production that I am unsure how to continue investigating. Our application is a modular monolith that starts multiple endpoints on startup. We recently noticed some errors on startup as shown below.
System.Exception: Management rights are required to run this endpoint. Verify that the SAS policy has the Manage claim.
at NServiceBus.Transport.AzureServiceBus.QueueCreator.d__11.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NServiceBus.HostingComponent.d__6.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.HostCreator.d__1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Endpoint.d__1.MoveNext()
Of course the first thing I checked was the SAS policy being used, and as shown it does have the manage claim.
From some further investigation I have found
The endpoint that is starting (named “customers”) already has a queue created in the service bus namespace. This is the first processing endpoint that is started in the application.
This error is only thrown when starting this endpoint. All other endpoints build and start without any errors with the same credentials. We have some common code to make NSB endpoints easier to configure/start, and as a result all endpoints are created with the same code, with the exception of a couple of parameters for the endpoint name and assembly scanning.
All of our existing endpoints have created their relevant queues/subscriptions automatically via NSB.
I spotted the latest release on the AzureServiceBus transport listed here, but it sounds like this is not the issue we are experiencing because we have the manage claim. Any suggestions on where to look next would be appreciated?
This issue definitely sounds like a different problem from what we fixed with the latest 3.0.1 release that you mentioned (as that fix was focused on enabling endpoints to run in minimum-privilege configurations).
The described behavior sounds very odd to me, in case all the endpoints use the same connection string/credentials. I assume you have verified that there is no chance that some sort of configuration override (e.g. a config file setting) might override the expected connection information? It also looks like the stack trace isn’t too helpful about the specific location the error occurs.
Unfortunately, our verification code that checks the manage rights doesn’t show the actual error message from the transport, so we could try to see what the reason for failure really is by executing this code yourself:
var administrationClient = new ServiceBusAdministrationClient(<connectionstring>);
await administrationClient.QueueExistsAsync("$nservicebus-verification-queue");
Can you check if this code fails when you try to run it with the connection information of your customers endpoint?
In case that doesn’t fail, can you share your configuration customization with us?
Good idea, I have carried out the above and I have a much more in depth error that points to some IP restrictions that seem to be impacting. Let me take it away, get more info and I can report back when I hopefully have a fix.
FYI our issue was in our infrastructure code, essentially we are accessing ASB through a private endpoint and there were some restrictions on which deployment slots had access to that.
Do you think it is worth opening a PR to surface the underlying error when this occurs along with the existing message being returned currently? It might help others that hit similar issues accessing ASB for reasons other than not having the manage claim.
Good to hear that this helped to resolve the issue
Do you think it is worth opening a PR to surface the underlying error when this occurs along with the existing message being returned currently? It might help others that hit similar issues accessing ASB for reasons other than not having the manage claim.
