ServiceInsight retry failed for message send via sql-Switch

, ,
1

Facing an issue while retrying message from ServiceInsight, below is scenario which not working.

ServiceControl DB installed on DEV3 database server. (error , audit queue configured here)
SourceEndpoint DB installed on DEV3 database server.
DestinationEndpoint DB installed on DEV2 database server.

SourceEndpoint Send command to DestinationEndpoint via Switch(switch@[nsb]@[SourceEndpointDB]) (Connecting multiple SQL Server instances with NServiceBus.Router • NServiceBus.Router Samples • Particular Docs), Command failed at DestinationEndpoint. As error queue on DEV3 DB,message stays at DEV3 DB

Solution tried:

  1. Retry directly form ServiceInsight - It search for DestinationEndpoint queue on ServiceControl DB & failed.
  2. Configured retry redirect(Redirects • ServicePulse • Particular Docs) - From DestinationEndpoint to switch@[nsb]@[SourceEndpointDB]
    & try redirect again from ServiceInsight, it failed with error at Router side & router send it to poison queue.
NServiceBus.Router.UnforwardableMessageException: Routing cycle detected: via|6|Switch
   at DetectCyclesRule.Invoke(PreroutingContext context, Func`2 next)

Please suggest how message can be retried when its flow via sql-switch.

SourceEndpoint : on .Net core, NSB 7, SQL transport
DestinationEndpoint : on .Net framework, NSB 5, SQL transport

Update:

Gone through code for NServiceBus.Router, as per that if message has header value for “NServiceBus.Bridge.Trace” : “via” , routerName then it throw “Routing cycle detected” error, this header value added by router when message first time process by router. so basically router not allowing to process message second time but in case of retry we need that.

2

Hi @Nirav,

I discussed this with @SzymonPobiega, there are 2 ways forward:

  • to unblock you from this situation, you could manually remove the NServiceBus.Bridge.Trace header and retry the message
  • if that doesn’t help, let us know and we’ll try to figure out how to move forward.

Kind regards,
Laila

3

Thanks for your response, I already tried this solution, it works fine but I see two issues with this approach.

  1. Need to manually remove header for each message, which may add extra overhead.
  2. This solution work for first retry, again it will add header, when we retry message & need to remove again

Exploring other option as well, what if we have different instance of service control for each database instance so that failed message will remain at same database instance where we have endpoints queue so retry message would be as normal message retry, please suggest on this as well.

4

Hi

You mentioned

how did the message that failed at the DestinationEndpoint get back to the router? Have you configured the router’s queue as the error queue?

We are trying to understand better you usage scenario. The removal of headers is just a temporary workaround. The ultimate solution is most likely a change in the way Router handles the cycle detection header.

Szymon

5

Hello,

We have error queue & DestinationEndpoint queue on different database instance.
When we retry failed message from ServiceInsight, It try to find DestinationEndpoint queue on same database instance where we have error queue, so for retry to work we have configured redirect from DestinationEndpoint queue to router queue( using Redirects • ServicePulse • Particular Docs) & from router queue , router endpoint again try to send to DestinationEndpoint.

Thanks,
Nirav.

6

OK, but how the error message ended up in a different database than the DestinationEndpoint database in the first place?

7

We have Configure centralized error & audit queue on seperate database on Dev3 server for all endpoints & endpoints queue are on DEV2 database instance

8

Can you post your DestinationEndpoint configuration? Is it configured to use the multi-instance mechanism to send the messages to the error queue on a separate instance?

Szymon

9

Hi @Nirav,

We saw that you raised a dedicated support case for this issue.
Can you please reply to @SzymonPobiega’s question on the support case so that we can gather all the information there?

For reference, v5 used to support configuring the error queue on a separate instance. However, this feature was removed in later versions and v5 is out of (extended) support.

Kind regards,
Laila

10

@laila.bougria, @SzymonPobiega, @Nirav Can we have a zoom call to discuss this next week as we have dedicated support available.

11

Hi @Arunkumar @Nirav,

We’re happy to jump on a call, let’s schedule a time through the support case you already raised.

Kind regards,
Laila

12

Was a solution ever reached for this? I have a similar issue where I can see the following error message

2022-09-23 15:52:37.622 ERROR Moving poison message to the error queue
NServiceBus.Router.UnforwardableMessageException: Routing cycle detected: via|37|MY.SERVICE.NAME.HERE
   at DetectCyclesRule.Invoke(PreroutingContext context, Func`2 next)
   at Interface`1.<>c__DisplayClass3_0.<.ctor>b__1(MessageContext context, IDispatchMessages _)
   at ThrottlingRawEndpointConfig`1.<>c__DisplayClass1_0.<<PrepareConfig>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at NServiceBus.Transport.RabbitMQ.MessagePump.Process(BasicDeliverEventArgs message, Byte[] messageBody) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 265
13

Hi @pbennington

These kinds of errors are usually related to the Router routing topology directing messages to the wrong queue. Can you share your Router configuration?