Saga TimeOut() is not working in Azure Service Fabric deployed cluster

, , , ,
1

I have created a micro-service using Service Fabric orchestrated using Docker container.
I am using NSB and transport as RabbitMQ to sending messages. In this have a Saga for my business rule and over the same endpoint I am sending message using the method SendLocal().

The handler is getting called. I have below code in my handler,

    public Task Handle(BagOpenMessageContract message, IMessageHandlerContext context)
    {
            // Set the timeout Event time.
            this.Data.RemainingTime = message.RemainingTime;
            this.Data.LastUpdatedDateTime = DateTime.UtcNow.AddMinutes(message.RemainingTime);

            return this.RequestTimeout<TimeoutModel>(context, TimeSpan.FromMinutes(1));
    }

and TimeOut method below,

   public Task Timeout(TimeoutModel state, IMessageHandlerContext context)
    {
        // Check the current time matches with the time out set in Saga Data.
        if (DateTime.UtcNow.Minute == this.Data.LastUpdatedDateTime.Minute)
        {
            this.inventoryManager.BlockInventory(this.Data.InventoryId);
            this.MarkAsComplete();
            return Task.CompletedTask;
        }

        return Task.CompletedTask;
    }

So ideally, this TimeOut() method should get invoked after say 1 min (that is what i have in handler). When I debug is locally, this works well, but once deployed on Service Fabric Cluster on Azure, TimeOut() method is not getting called.

I have logs (Application Insights) in Saga, which verifies the control is coming to handler but after the specified timespan it just doesn’t work.

Can any one please tell me what I am missing here ?

2

Hi

What is the instance count of the service you deployed inside the cluster?

Do you use the new native delayed delivery or the timeout manager based delivery?

What kind of persistence are you using?

Regards
Daniel

3

Thanks for the reply Daniel. And Apologies from me for responding late.

This issue is resolved now.
It was basically the persistence issue. I was using InMemory and now changed it to SQL persistence and its working all fine.

In service Fabric, nodes keeps on switching internally, so using InMemory was flushing out the saga data which caused the issue.

Now using SQLPersistence, the Saga Data is maintained even if nodes are changed, which resolved the issue.

Thanks for your help. Appreciated !!!

1 Like