Message Type 'unknown' in ServiceInsight

Garran-SHFT · October 26, 2023, 9:12am

Description

We have a system with many applications that publish events and other applications that are able to subscribe to. We are using Azure Service Bus.

When we view ServiceInsight we are having a hard time seeing our working messages through all the Unknown Message Type messages. I have gone as far as deleting all the queues and subscriptions and started over but we still seem to see a lot of the Unknown Message Type messages. At this point, we are not sure if this is a code or setup problem we are having.
These Unknown Message Types do not have a body, no errors, or sequence diagrams, in the logs we see no errors, and only limited header information, like the ones listed below:

Diagnostic-Id
NServiceBus.MessageId
NServiceBus.ProcessingStarted 2023-10-26 07:21:25:672515 Z
NServiceBus.ProcessingEnded 2023-10-26 07:21:25:672558 Z
$.diagnostics.hostid
$.diagnostics.hostdisplayname
NServiceBus.ProcessingMachine
NServiceBus.ProcessingEndpoint
NServiceBus.Version 8.0.3
NServiceBus.TimeSent 2023-10-26 07:21:25:672578 Z

Here is a screenshot of the issue I am finding and we can see the Time Sent is a few seconds apart:

We would really appreciate the help or point us in the right direction.

Versions

ServiceInsight: 2.13.0
ServiceControl: 4.32.2
ServicePulse: 1.37.0

Nuget Packages:
NServiceBus: 8.0.3
NServiceBus.RabbitMQ: 8.0.1 (Used for local)
NServiceBus.Metrics.ServiceControl: 4.0.0
NServiceBus.Newtonsoft.Json: 3.0.0
NServiceBus.ServicePlatform.Connector: 2.0.1
NServiceBus.Transport.AzureServiceBus: 3.2.0

Dennis · October 27, 2023, 11:25am

Hi,

Could you investigate one of the messages and paste more information about a single one of them here? Like the headers of the messages that are available? If there’s not much information in ServiceInsight, perhaps in ServicePulse or even ServiceControl itself, using the maintenance mode and going into RavenDb?

Regards,
Dennis

Dennis · October 27, 2023, 11:32am

Also, the message identifiers in the example are custom identifiers, not provided by NServiceBus. They’re not in the format we provide. Are you using native integration? Possibly you’re not including the EnclosedMessageType header?

You can find more information about NServiceBus headers here. There are also several samples on our docs website called “native integration” which show how to natively integrate with NServiceBus by providing the right headers.

Does that help?

Garran-SHFT · October 31, 2023, 6:27pm

Hi Dennis,

Thank you for getting back to me. Seems I need to put this in multiple messages due to the screenshots.

The problem we’re encountering revolves around a lack of information. Here are some details and screenshots in the hope that they help you better understand the issue:

Garran-SHFT · October 31, 2023, 6:28pm

Example of sequence diagram with no information:

Garran-SHFT · October 31, 2023, 6:28pm

Example of the body with no information:

Garran-SHFT · October 31, 2023, 6:29pm

Example of message properties:

Garran-SHFT · October 31, 2023, 6:29pm

Messaging Process Overview:

From an application standpoint, we publish a message event using _messageSession.Publish(CaseAllocatedEvent) from NServiceBus.IMessageSession.
CaseAllocatedEvent is a basic class containing several properties.
NServiceBus.IMessageHandlerContext processes the message from the message handler.

Custom Identifiers:

You mentioned “custom identifiers,” but we couldn’t find where these are being set. Are these related to Azure services or something else?

Among the unknown message types we have working messages with headers and a JSON body:

Garran-SHFT · October 31, 2023, 6:30pm

Could you perhaps point me in the right direction of where to look in RavenDb? Here are some screenshots:

I am aware of these 4 failed messages but I can’t see how this error could be causing this issue.

The four errors are:

"Exception": {
                    "ExceptionType": "NServiceBus.MessageDeserializationException",
                    "Message": "An error occurred while attempting to extract logical messages from incoming physical message dc8e3090-xxxx-xxxx-xxxx-b0a800c636ef",
                    "Source": "NServiceBus.Core",
"StackTrace": "NServiceBus.MessageDeserializationException: An error occurred while attempting to extract logical messages from incoming physical message dc8e3090-xxxx-xxxx-xxxx-b0a800c636ef\n ---> System.ArgumentException: Declined user id is invalid\n ...

We will read the “native integration” section to see if it provides any insights into our issue.

If you need more information or if anything is unclear, please let us know. Your assistance in resolving this issue is greatly appreciated.

Best regards,
Garran

Dennis · November 15, 2023, 4:02pm

For some reason, I didn’t receive an email for your replies. Apologies for replying so late.

First of all, you’re sharing only parts of the information. Not entire screenshots, not the entire stacktrace, that makes it hard to understand what’s going on.

The error

The error might have something to do with it. It should be nearly impossible to get this error when you’re using vanilla NServiceBus. But the error An error occurred while attempting to extract logical messages from incoming physical message is accompanied by Declined user id is invalid and then the stacktrace is cut off.

It feels like this indicates you have entered some data in a message and when the message is deserialized something else happens? Because Declined user id is invalid is not a regular deserialization message! Maybe a custom (de)serializer? Maybe some custom code in your message classes?

The weird messages in ServiceControl

What I mean with native integration, is that some other component is sending messages without using NServiceBus. So it’s creating messages on its own, but forgot to add appropriate headers. If NServiceBus fails to deserialize those messages, it will send them to the error queue.

They then end up in ServiceControl. ServiceControl is less strict on deserialization, because if you encrypt the message body (for example), it will never be able to show that in ServiceControl.

The thing to note about those message is:

They hardly have headers. This is a very, very strong indication they were not send by NServiceBus
The identifiers of the messages are different. Instead of 9f0d8767-eaa2-441e-9b18-de0406de9f38 I see 9f0d8767eaa2441e9b18de0406de9f38. But 9f0d8767eaa2441e9b18de0406de9f38 looks like an actual GUID without dashes, but your identifiers don’t even look like GUIDs?

What to do?

First of all, on follow-up replies, please provide us with complete information. If for some reason you don’t want to share that publically, which I can totally understand and appreciate, send an email to support@particular.net because your company has support! This is a community forum that we also monitor, but way less strict that actual support.

Another thing you can do, is stop the audit instance of ServiceControl, so that messages will remain inside the queue. And then investigate them with ServiceBusExplorer or Cogin Queue Explorer or something and see what’s actually inside those messages?

Hope this helps.