We are running NServiceBus 4.6.5 in a distributor/worker configuration and are preparing for our peak sales period where the system will be under much heavier load than normal.
One thing we have found during our performance tests is that the distributor tends to go into high CPU usage if a backlog of messages appears on the distributor’s input queue. We have seen it go as high as 90% if there are as few as 1,000 messages on the input. The distributor is running on a reasonably spec’d machine: it’s a 2-node Windows 2012 cluster, with each node a 2 x dual-core 2.3HGz CPU and 8GB of RAM.
Is this normal? What is the process doing that requires such high CPU usage?
Version 4.6.5 is from September 2014, I quickly browsed through the release notes of the releases after 4.6.5 and found atleast one bug related to the distributor and would recommend updating to the latest patch release in the v4.x.x range which is 4.7.12
Also, the distributor got forked into its own package in the 4.x range. Please add the package NServiceBus.Distributor.MSMQ to your project and use the new profiles as documented here:
Also a tip, have you configured the maximum concurrency level on the distributor and the workers? If not, then both the distributor and worker nodes are processing messages sequentially. Increasing the maximum concurrency on the distributor will forward messages to the workers in parallel and increasing it on the workers results in your workers to process messages in parallel. Only increase this on the workers if the workload allows for it.
Let us know if upgrading to these packages resolved your CPU issue!