NServiceBus.Extensions.Hosting - Is it possible to stop and restart an endpoint?

1

I am using multiple endpoint hosting in the same process with the goal of helping building a redundancy system. One endpoint only receive Start/Stop messages and trigger Start/Stop on the real endpoint with the background worker services. The goal is to stop processing messages and keep it in “idle” state while it is not elected as the main instance, while still having the Windows Service application running and ready to start processing.

Previously, using self-hosting, I could easily create two IEndpointInstance and call Start/Stop on them as often as I wanted and it worked perfectly. I’m trying to achieve the same goal using the .NET Generic Host and I am having issues.

First, with NServiceBus out of the image, it seems that .NET Generic Host by themselves can’t be stopped and restarted within the same application lifetime or it crash and throw exceptions. I created a StackOverflow question about it here.

So my first instinct as a workaround was to leave the .NET Generic Host running, but using the IServiceProvider instance to grab every hosted services (mine, and the NServiceBus ones) and call start/stop on them instead.

Problem is, the IServiceProvider.GetService() method require a type. IEndpointInstance isn’t registered. When debugging, I can see multiple services registered by NServiceBus:

NServiceBus.HostAwareMessageSession
NServiceBus.Extensions.Hosting.NServiceBusHostedService
NServiceBus.LearningTransportDispatcher
NServiceBus.Hosting.HostInformation
NServiceBus.StorageInitializer.CallInit

I see that NServiceBusHostedService have a “Endpoint” property that I could probably call Start/Stop on it. Unfortunately, I can’t resolve an instance of this type because it is marked as internal. This blocks my attempt to retrieve the NServiceBus endpoint.

But still, as a test I called GetService<IHostedService>() with only this service registered and managed to get the instance of NServiceBusHostedService anyway (typed as a IHostedService). I can’t access the Endpoint sub-property, but I can call StopAsync/StartAsync on it passing a new CancellationToken. Doing so seems to work for the Stop, but unfortunately when calling Start again afterward I get this exception:

System.Collections.Generic.KeyNotFoundException : 'The given key (NServiceBus.Transport.TransportInfrastructure) was not present in the dictionary.'

So I really don’t think this is the proper method to achieve my goal.

TLDR; Is stopping and restarting endpoint message processing possible to do with the .NET Generic Host method? And how?

2

Hi,

The problem you are seeing is that the EndpointConfiguration object cannot be reused to start the same instance again. The reason is that we are cleaning all the previously applied settings and the lifetime of the endpoint configuration object is tied to the lifetime of an endpoint. If you want to achive what you are planning to do you need to take it in your hands implement the hosted services and custom hosting of NServiceBus yourself as well as making sure the EndpointConfiguration object is not reused. You are free to copy code from the existing NServiceBus.Extension.Hosting and tweak it to your needs locally.

Hope that helps

Regards
Daniel

3

Hello Daniel!

Thanks for the reply.

Just a quick reminder, my goal is simply to be able to Start/Stop message processing on demand. I don’t need to delete/re-create all the configuration.

I would be willing to download the project from source and tweak it to my needs if it was easy, but looking at the code on GitHub is seems to go much deeper than just modifying a few lines in NServiceBus.Extension.Hosting. I’m not very familiar with your project structure and would love a few more pointers if that’s possible. If required, I can go through the e-mail support service (we paid for a support license) instead of this forum.

Here’s a quick analysis of what I can understand:

If it were “easy”, the EndpointConfiguration object would be used in the UseNServiceBus() extension method only, and then the StartAsync() method on NServiceBusHostedService wouldn’t need to use it.

But instead, it seems like UseNServiceBus() goes through EndpointWithExternallyManagedContainer.Create() which call HostCreator.CreateWithExternallyManagedContainer() which end up creating a ExternallyManagedContainerHost object, which keep a copy of a EndpointCreator object. The ExternallyManagedContainerHost object is the one returned as a IStartableEndpointWithExternallyManagedContainer that is stored in NServiceBusHostedService on which Start() get called on.

Unfortunately, ExternallyManagedContainerHost.Start() does not just Start the endpoint, but also call endpointCreator.CreateStartableEndpoint() to re-create it every time, probably re-using the EndpointConfiguration in one form or another doing it.

Needless to say, this is extremely complex from my point of view and can’t just be quickly “tweaked”. This also seems to requires change in The NServiceBus.Core project more than just in the NServiceBus.Extension.Hosting project. I hope I missed something and I am wrong, but in the current form I can’t seem to do anything to fix that.

4

Hi Dunge

I gave this some more thoughts and I came to the conclusion it is not possible to achieve what you want to achieve. Despite what I said in the previous answer the problem is indeed more than just EndpointConfiguration. The way that the generic host and service collection/provider works is that there is a dedicated configuration phase where dependencies are registered but no resolve is allowed up to the point when the service provider is built. From that time on dependencies can be resolved but no registrations can be done anymore unless you rebuild the service provider which might not be possible depending on the underlying DI container you are using. Therefore I think it is best to assume once the provider is built the container is sort of immutable.

NServiceBus before it is being started needs to register dependencies on the collection during the registration phase which is

var startableEndpoint = EndpointWithExternallyManagedContainer.Create(endpointConfiguration, serviceCollection);

once that is done the actually starting of the endpoint will no longer try to add anything to the collection. That’s why NServiceBusHostedService passes the service provider to

await startableEndpoint.Start(serviceProvider)

but in order to restart the endpoint you would need to basically do the following again:

            var endpointConfiguration = endpointConfigurationBuilder(ctx);
            var startableEndpoint = EndpointWithExternallyManagedContainer.Create(endpointConfiguration, serviceCollection);
            await startableEndpoint.Start(serviceProvider)

but because the provider is already resolved calling EndpointWithExternallyManagedContainer.Create(endpointConfiguration, serviceCollection) would effectively try to reregister the same dependencies and then you would need to recreate a new provider and pass it to start. As far as I’m aware now this is not possible due to restrictions of how the generic host and the MS DI works.

I’ll double check this with someone else though.

So I think it is probably simpler to move the endpoint that needs to be started and stop into it’s own service and then start and stop that one on demand as needed.

Regards
Daniel

5

All the way reading your reply I keep thinking “but why can’t you simply just leave everything registered/resolved and just expose IEndpointInstance so I can call Start/Stop on it that would just pause its internal message pump loop without going through all that dependency injection nightmarish scenario?”

Now I realized that even in my old self-hosting code, I would create a new DI container and configuration on every re-start because RunningEndpoint can only be stopped, it does not have a Start method. There’s probably a reason why you designed the system in a way that endpoint startup can’t be decoupled from its creation, and its creation can’t be decoupled from the DI registration, but this goes outside my field of expertise.

So I think it is probably simpler to move the endpoint that needs to be started and stop into it’s own service and then start and stop that one on demand as needed.

Do you have a quick sample that would show me how to do that? Because as you mentioned, this seems impossible to work around with the way MS DI works.

To be clear, are you suggesting to forget about using the NServiceBus.Extension.Hosting nuget and UseNServiceBus() altogether and instead control the endpoint lifetime in one of my service? Maybe by using NServiceBus.Extensions.DependencyInjection and a EndpointWithExternallyManagedServiceProvider? If you remember, I had quite the issues trying to use it previously (message pump would start before DI resolving would finish) and put it on hold until I had the budget to move my solutions to Net5 and use NServiceBus.Extension.Hosting which I hoped would help get over it since you put a lot of work into it.

Now I finally got the budget to port csproj, configurations, updating nugets for compatibility and refactoring my services to use IHostedService which keeps my hands full already. I was glad to finally move to a proper structure with MS Host, DI, Logging and such and hoped that this time the NServiceBus transition would be painless, but it’s not the case. I already spent way too many hours trying to make sense of it all, and feel like I’m driving straight into a wall.

6

There’s probably a reason why you designed the system in a way that endpoint startup can’t be decoupled from its creation, and its creation can’t be decoupled from the DI registration, but this goes outside my field of expertise.

The way the current lifecycle of NServiceBus is configured, starting and stopping an endpoint does not just affect the message pump’s behavior to receive messages or not but does a lot more infrastructure related stuff. Stopping the endpoint also does shutdown some infrastructure that might be required e.g. by the transport. There is currently no API to solely stop (or even better, pause) the message pump.

As Daniel pointed out, the endpoint itself is very much tied towards the DI container, and the DI container is managed by the generic host in this case. This means, if you want to start/stop the endpoint more dynamically during runtime, you’d have to provide a new DI container (ServiceCollection and ServiceProvider) to every endpoint that you start (or let the endpoint just manage it’s own internal container which would require less effort in this case).

In short: When using the generic host, the NServiceBus endpoint is tied very much into the lifecycle of the host because the host owns the DI container. NServiceBus itself doesn’t really have an API to control the message pump during runtime, which definitely is an interesting feature request.

I’m not familiar with your exact needs, but I’d probably recommend to split the endpoint in some way that you can shut down the whole host when you want to stop receiving message and start it up again when you want to resume and keep the rest of the application (and if necessary a send-only endpoint) on a second host that will always be running.

FYI: I’ve raised this feature request as an internal issue.

7

I’d probably recommend to split the endpoint in some way that you can shut down the whole host when you want to stop receiving message and start it up again when you want to resume and keep the rest of the application (and if necessary a send-only endpoint) on a second host that will always be running.

Yup, that was my original plan when I started the move to MS Generic Host, I would shut down the host (nservicebus and my services together) and re-create a new one. Unfortunately, after a few simple tests with an empty project with just MS Generic Host (no NServiceBus), it seems like this isn’t supported (see my post to StackOverflow above). Not only I can’t start/stop/start a host, but I also can’t start/stop, create a new one and start/stop again. As soon as you call Stop twice in the same application lifetime, no matter if it’s the same host or not, it crash. In fact, I’m not certain your suggestion of using two generic host for multiple endpoint is something that is supported by Microsoft. And it is why I then moved to trying to Start/Stop the services and endpoints instead of the host itself.

About the “less effort” solution to let the endpoint manage it’s own internal container, I’m not quite sure how to do it. And if I understand correctly, that would prevent me from injecting my services in the message handler classes? This is kind of a blocking issue.

Thanks for raising the “pause” feature request. Hopefully it will helps in the future.

As for my need why I need to pause an endpoint processing, I can do a quick explanation here:

I reached out to your support e-mail back in 2018 on how to design a redundancy system since NServiceBus wasn’t allowing me to everything I needed out of the box. I was suggested to build a multiple hosting endpoint in this manner, and after working hard I finally managed to get something working perfectly with self-hosting. Depending on the roles of the application, one of more endpoint are created in the application lifetime:

-Some endpoint type share the load (every instance use the same name and share the same queue). This act as some kind of load balancing.
-Some endpoint type act individually (every instance have a individual instance name and their own queue and receive every messages). This is mostly used for cache refresh.
-Some endpoint type have one “main” instance active. While all applications are “running”, only one out of all instances is active at one time and have a worker service running and an endpoint processing messages. All available instances periodically send heartbeats. Another endpoint available on all machine hosts (I called the orchestrator) listen to the heartbeats and use an algorithm to select the “main” instance. When the main is down, a new one is selected. At this point the old main instance stop receiving messages so it shutdown its endpoint and service, and the new one start the endpoint and services. This could theoretically toggle many times during an application lifetime if there’s network outages or new machines get added/removed from the cluster.

The last one is the type of endpoint where I need to start/stop the endpoints in the application lifetime. As I said, it is currently working perfectly with self-hosting, I just wish there was a way I can keep doing this on a NetCore/Net5 framework.

8

Hi dunge

I am using multiple endpoint hosting in the same process with the goal of helping building a redundancy system.

The highlight is mine. What Tim and I have trying to highlight is the fact that most of the complexity you are currently facing is because you are trying to host multiple endpoints in the same process. Why not isolate each of them into a dedicated process and deploy them as individual services?

With that, your headaches that you are currently facing are out of the window and you can simply start and stop those services according to your needs or data you even gather at runtime.

Or maybe I’m misunderstanding things.

Regards
Daniel

9

The “internally managed container mode” is just the traditional way to use NServiceBus (basically using the Endpoint.Start API). Using the NServiceBus.Extensions.DependencyInjection package, you can also access the ServiceCollection used by NServiceBus, so you can register specific shared service instances as singletons in every endpoint’s ServiceCollection. See the documentation here on how that works: NServiceBus.Extensions.DependencyInjection • Particular Docs

1 Like