Is there a way to prevent timeouts while I'm debugging?

1

In short, if I run a command and set a break point, then step through my code, if I take to long, the bus starts throwing and retrying which is problematic. Is there a way to prevent this while I’m debugging?

2

Hi Jeremy!

This is a tricky topic, we did have some code in the past trying to mitigate for long debugging sessions but there is a trade off between that and production safety as the code gets more complex. You can attach a debugger in production and we don’t want to alter behavior in those scenarios. Also not all timeouts are under our control, eg. TransactionScope still have a machine wide max limit that we can’t extend.

What transport/persistence are you using in this case?

Cheers,

Andreas

3

We’re actually using rabbitmq. And fwiw, we don’t necessarily need this ability in production, only during development. I was thinking of something where I can have an alternate dev configuration during startup. (e.g. if(!environment.IsProduction()) { setUnlimitedTimeout(); }

4

RabbitMQ has no receive timeout to my knowledge

I have verified this using our RabbitMQ simple sample with a never ending handler and it just keeps on going

public class MyHandler :
    IHandleMessages<MyMessage>
{
    static ILog log = LogManager.GetLogger<MyHandler>();

    public async Task Handle(MyMessage message, IMessageHandlerContext context)
    {
        var waited = 0;
        while (true)
        {

            await Task.Delay(TimeSpan.FromSeconds(10));
            waited += 10;

            log.Info("Waited: " + waited);
        }
    }
}

What exception are you getting when it times out?

5

I don’t know what the exact conditions are that cause it to happen so it’s difficult to reproduce. But from memory, the error is a message about the circuit breaker tripping and I believe there was a message about a timeout. And in all cases, the solution is to stop it and start over. If you let it run, it’ll complete with no errors. The most I can repro right now is the initial message that the circuit breaker is armed for messagepump, timeoutsdispatcher, and timeouts.

2018-03-13 09:45:38.664 WARN NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker The circuit breaker for ‘DAS.Services.Surveys.TimeoutsDispatcher MessagePump’ is now in the armed state
2018-03-13 09:45:38.695 WARN NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker The circuit breaker for ‘DAS.Services.Surveys.Timeouts MessagePump’ is now in the armed state
2018-03-13 09:45:38.664 WARN NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker The circuit breaker for ‘DAS.Services.Surveys MessagePump’ is now in the armed state

Other times, it’ll actually throw an exception with similar messages.

Right now, I’m on a break point and just kinda moving the debug cursor around to simulate my taking a long time to debug but it’s not tripping. I wonder… if I’m currently using InMemoryTransport, if that would be enough to fix it?

I was assuming that after x amount of time, NSB sees the the message isn’t finished processing yet and then retries with a new handler and then after so many failures to finish, it finally throws.

6

I can’t seem to reproduce this, would you be able to run Simple RabbitMQ Transport Usage • RabbitMQ Transport Samples • Particular Docs and see you are able to reproduce this there?

7

I can’t reproduce it right now either. It happens pretty often during development thought (we’re currently closing up this release so not a ton happening right now), but I’ll let the guys know to let me know when it happens again so that we can capture it.

1 Like
8

I have had similar issues, as well as the message being dispatched to the handler a second time. I believe it is because I am using the outbox feature along with RabbitMQ.

Are you using the outbox feature Jeremy?

9

I’m not using outbox, no.

So it’s happened to me a few times in the last hour and this seems to be the exception that I’m getting:

2018-04-11 10:06:34.392 WARN NServiceBus.Transport.RabbitMQ.MessagePump Failed to acknowledge message ‘ae320abf-838c-457c-84e4-a8bf0114f758’ because the channel was closed. The message was returned to the queue.
RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=0, text=“End of stream”, classId=0, methodId=0, cause=System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
at RabbitMQ.Client.Impl.Frame.ReadFrom(NetworkBinaryReader reader)
at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd)
at RabbitMQ.Client.Impl.ModelBase.ModelSend(MethodBase method, ContentHeaderBase header, Byte body)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.RabbitMQ.MessagePump.d__29.MoveNext() in C:\BuildAgent\work\a9e6741f41af7061\src\NServiceBus.Transport.RabbitMQ\Receiving\MessagePump.cs:line 267
UJSTAFFORD-DT [VRB 2018-04-11 10:06:34.402 -07:00] [Thread 47] InventoryExtractor::Processing DAS.Data.ETL.HomeNet.Messaging.ServiceBus.ExtractRooftopInventory…
UJSTAFFORD-DT [VRB 2018-04-11 10:06:34.404 -07:00] [Thread 47] InventoryExtractor::Waiting our turn for HomeNet extraction “039170”…
UJSTAFFORD-DT [INF 2018-04-11 10:06:34.406 -07:00] [Thread 12] Inventory extraction for “039170” began at 04/11/2018 17:06:34 +00:00.
2018-04-11 10:06:35.522 INFO NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker The circuit breaker for ‘DAS.Data.ETL.HomeNet.TimeoutsDispatcher MessagePump’ is now disarmed
2018-04-11 10:06:35.527 INFO NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker The circuit breaker for ‘DAS.Data.ETL.HomeNet.Timeouts MessagePump’ is now disarmed
2018-04-11 10:06:35.555 INFO NServiceBus.Transport.RabbitMQ.MessagePumpConnectionFailedCircuitBreaker The circuit breaker for ‘DAS.Data.ETL.HomeNet MessagePump’ is now disarmed

10

If you spend too long at a breakpoint without moving to the next, this will happen. There is some background task that pulls data from RabbitMQ and if it doesn’t operate in too long of a period, you will get this exception. I don’t recall that happening unless it’s been a minute or more, but I could be wrong.

11

Yeah this has been my experience as well. If I had to guess what was happening, maybe there’s some sort of NSB service trying to read the queue before the stream has been advanced; a race condition perhaps.

12

RabbitMQ does have a heartbeat mechanism, so if you’ve paused execution, the broker is going to detect the missing heartbeat and close the connection on you.

If you need to debug things, you can increase the heartbeat timeout value: Connection settings • RabbitMQ Transport • Particular Docs

I can’t link directly to it, but there is a note on that page that mentions needing to increase this value when debugging.

If you’re already setting this value and still having this problem, let me know!

13

We had the same issue when pushing quite a few messages to a queue.

I just applied a rule by following this: Installing VMware Tanzu RabbitMQ for Kubernetes | Tanzu RabbitMQ

All good now.