Using callbacks in ASP.NET Core MVC application in scaled-out Azure App Service

Architecture & Design
,
1

Hi there, I tried summarizing my conversation with Copilot and attach it as a pdf here, but apparently I’m unable to do so as I’m a “new user”. Therefore I paste the contents below, to explain what I’m working on right now.

:white_check_mark: Context

  • Running an Azure App Service that scales out to multiple processes.

  • Need NServiceBus callbacks in a web application where the HTTP request waits for a backend response.

  • The reply is only relevant while the user/browser session is active (loss is acceptable if the process recycles).

:white_check_mark: Problem

  • NServiceBus callbacks are ephemeral and tied to the process instance.

  • In Azure App Service:

    • Processes recycle or move to another VM unpredictably.

    • Instance identifiers change on restart, breaking callback routing.

  • I considered implementing a SlotManager:

    • Uses SQL to assign one of 3 logical slots (SlotA, SlotB, SlotC) to each process.

    • Maintains stable identity across restarts using leases.

    • Passes the slot name as the endpoint unique address for callbacks.

:white_check_mark: Questions Explored

  1. Is there an off-the-shelf solution?

    • NServiceBus supports callbacks via EnableCallbacks() and MakeInstanceUniquelyAddressable().

    • But callbacks do not survive restarts by design.

    • Official guidance: If loss is acceptable, callbacks are fine. Otherwise, use request/response with a handler or a saga.

  2. Alternatives to callbacks:

    • Request/Response with a handler:

      • Send a message, store correlation ID, and poll or push updates to the browser.

    • Saga:

      • Durable state machine that tracks workflow and can be queried by the web app.

    • Both require async UI (polling or SignalR) instead of blocking the HTTP request.

  3. Polling Saga State:

    • Saga persists state in SQL or another persistence store.

    • Web app queries saga state by RequestId via an API endpoint.

:white_check_mark: Current Understanding

  • If loss is acceptable, the simplest solution is:

    • Use EnableCallbacks() and MakeInstanceUniquelyAddressable().

    • Accept that callbacks will be lost on recycle.

  • If loss is not acceptable, use:

    • Request/Response with a handler or

    • Saga with correlation and polling.

  • There is no built-in or community-supported feature for stable slot-based identity across restarts (like my SlotManager idea).

:white_check_mark: What I Need from NServiceBus Support

  • Confirm that callbacks are intended for ephemeral scenarios and that my understanding is correct.

  • Validate whether there is any built-in or community-supported feature for stable identity across restarts such as the SlotManager idea.

  • Recommend the least-effort, reliable approach for legacy apps where:

    • Loss is acceptable.

    • Minimal code changes are preferred.

Thusfar the summary that I had with Copilot about this. There are two scenario’s of scaling-out that I’m considering:

  1. One app service running in multiple (fixed amount) of processes using the SlotManager idea to uniquely address an instance.
    Positive aspect is that only one app service needs to be defined and can relatively easily expanded with additional processes when necessary.
    Negative aspect is the fact that the slot leases need to be maintained and there doesn’t seem to be an off-the-shelve solution to manage this - I would want to avoid custom developing this.
  2. Multiple app services (deployments of the same application each time) running only 1 process, in which the name of the app service can be used to uniquely address an instance.
    Positive aspect is that there is no need for a slot lease / slot manager solution that needs be custom maintained (when no off-the-shelve solution exists).
    Negative aspect is that when the web application needs to scale out, an additional app service needs to created and deployment pipelines adjusted so that it can additionally deploy to this extra app service. Also, load balancing needs to be added and configured to distribute load across the different app services where otherwise the app service handles this in the previous option.

This allows for creation of a predictable set of reply-queues so that an app service does not need queue-creation privileges and the names of the queues are deterministic.

Are both approaches valid? Are there better approaches that I’m not thinking of right now?

Thanks!
Roel

2

Roel:

CoPilot is correct, and we can confirm that callbacks are intended for ephemeral scenarios. They will not survive a process restart (e.g. a crash or an IIS recycle) as they are held in memory.

Are both approaches valid? Are there better approaches that I’m not thinking of right now?

Slots or separate app services will not change this behavior.

However, when scaled out you will need to use the MakeInstanceUniquelyAddressable() API to ensure the message reply is directed to the web server instance that has the callback function instance in memory.

Your instinct is correct that finding a suitable name for that unique instance address is the challenge.

Another option would be to use the WEBSITE_INSTANCE_ID environment variable provided by Azure as the unique identifier.

When scaling up this would create the appropriate queues. However, be aware that when scaling down NServiceBus will not clean up those queues.

  • Bob