We have a simple architecture with 2 endpoints using nservicebus 7. and sql persistence. Our events are marked with an Interface IEvent. Now and then our events end up in the dead letter. When we look at the exception it says : No handlers could be found for message type.. If we retry the message then it is handled correctly. This behavior is not typically for one of our message but infects all of them. We noticed that most of the time it is not one message that appears but several after each other with a very small time frame between. It is not that we have a heavy load system. The exception mentions our generic Interface but not the concrete message type. Maybe that is an indication of the issue?
I used to work on a more complex system with 6 endpoints and more heavy load. This was together with a MSMQ system instead of sql. During the period of 7 years, I never saw the above described behavior.
NServiceBus registers all handlers once at startup. It would either always work or always not find the handler. Intermittent behaviour is definitely not right.
Is logging a centralized logging for scaled out endpoint?
Is your endpoint scaled ou manually? If yes, might be that one of the logical endpoint instances is missing the necessary handler and occasionally fails, when gets to process messages of that type.
As Sean already mentioned, intermittent behavior during runtime is definitely not something that is expected. It would be very helpful to have a sample that reproduces the issue somehow.
However, a potential root that might cause such cases could be:
- Different endpoints somehow consuming from the same input queue. In such a case, it is not clearly predictable which endpoint will actually receive the message from the shared queue and if an endpoint without the necessary handler receives it it could cause the described error. A way to check for this is to carefully inspect the headers of the failed message in the error queue, it should contain information about the queue the message was received from and the endpoint that failed to process the message. Double-check these headers to verify that the header values are what you’d expect them to be.
- It could be possible that the failing endpoint somehow subscribed to certain event messages (e.g. due to auto-subscribe if it was deployed with a matching handler once). However, in this case, the endpoint would receive the subscribed message types consistently. You can inspect the existing subscriptions of an endpoint directly in the SQL database.
These are only two potential ideas and more information about your system would be required to investigate that issue.
Thanks, Tim (and Sean). for your swift answers. We found the issue and it was caused by what Tim said (different endpoints that were looking at the same queue). We fixed this and now it works fine 