I’m developing a service which processes a number of documents.
Each document type processing is quite unique and isolated so I decided to have endpoint per document type architecture.
Each endpoint creates at least 2 long live connections (for listening/publishing) to RabbitMQ server.
Having approximately 40 documents types means we have 40 endpoints and 40x2 (80) connections.
Scaling such service creates more and more connections.
RabbitMQ has a physical limit on connections number so I’m wondering if it is possible to share/re-use an existing connection between multiple endpoints?
Personally, I’d take a slightly different route - have handlers/sagas per document type in the same endpoint and scale it out rather than creating an endpoint per document. Whenever you find yourself needed to split certain documents into their own endpoints, you still would be able to do it, but you’d have less resources wasted on having so many endpoints co-hosted in a single host.
RabbitMQ is extremely efficient in managing connections. It can handle several thousands of connections concurrently if configured properly. I would not worry too much about that.
What is the reason you are co-hosting all those endpoints together in a single process? Because it seems a bit of an antithesis to your initial split. Although your documents are separate and split out they are still tight together in the same host. Thus if you need to change a process for a single document type you have to take all the endpoints down. If they are truly separate and unique why not have single deployable endpoints and hosts?
Currently in my setup I already see 1000 connections, that’s why I’m worry.
Endpoints are co-hosting together only temporarily, from scaling perspective I can host one endpoint per process.
I would say co-hosting is done more for grouping and easy managing windows services.
Because of expecting high load we want to scale processing of documents independently.
To meet the expected load we went with idea that we need separated RabbitMq queue per document type.
In NServiceBus (normally) each queue means a new endpoint.
In another setup I’m afraid it can be harder to scale processing independently (f.e put processing of some document on completely different machine)