Challenges with the migration to the topic-per-event topology

This discussion topic serves as a reference point to capture challenges with the migration to the topic-per-event topology. Please share with us the challenges you faced or rough edges you discovered during the migration.

There is also a github issue you can use should you prefer this avenue.

For cases when specific data or insights of the project needs to be disclosed, feel free to raise a support request.

Additional Context

Additional ideas

I can’t upgrade yet because
NServiceBus.AzureFunctions.Worker.ServiceBus doesn’t support it yet.

Will this also be upgraded?

Other question:
It’s not clear to me from the manual. if we need todo this manual or if the installer takes care of this. I assume the later?
topology.SubscribeTo<IOrderStatusChanged>("Shipping.OrderAccepted");

Hi

Will this also be upgraded?

We have not yet made the decision when we’ll upgrade the Functions support. We expect to assess its priority next week.

It’s not clear to me from the manual. if we need todo this manual or if the installer takes care of this. I assume the later?

The topology.SubscribeTo<>() API is used to override the default mapping, which is the event’s full name. In this case if you have a handler for Shipping.IOrderStatusChanged like this:

public class OrderStatusChangedHandler : IHandleMessages<IOrderStatusChanged>
{
}

and auto-subscribe enabled (default), the automatic subscription mechanism will create topic shipping.iorderstatuschanged and a subscription to that topic with forwarding to its input queue. This topic, however, will never receive any messages because publisher always publishes concrete events e.g. OrderAccepted. That events, by default, will be published to shipping.orderaccepted topic.

This why the mapping via topology.SubscribeTo is necessary in this case.

1 Like

Hi @remyvd

I can’t upgrade yet because
NServiceBus.AzureFunctions.Worker.ServiceBus doesn’t support it yet.

Wanted to give you a heads-up that we are making good progress with the function integration, and we are looking at an ETA of a week(ish) until it is completed if nothing else pops up during our testing.

Daniel

We will be releasing things on Monday

Hi @remyvd

We have just released the Azure Functions hosting packages. Feel free to give it a try and looking forward for your feedback!

1 Like

@SzymonPobiega thanks. I rolled it out today and I will monitor it. I am happy with this change because we have hit the 5Gb limit in the past :-).

When not using interfaces, but concrete classes, do I also need to specify it explicitly?

I have the following msgs:

public record InteractionEvent : IEvent
public record InteractionSent : InteractionEvent
public record InteractionOpened : InteractionEvent
public record InteractionClicked : InteractionEvent

I only published the events derived from InteractionEvent, never the InteractionEvent itself.

I subscribe only to InteractionEvent (for now) in my handler.

I see that only the interactionevent topic is auto created (but not the derived msg types). Will this work ok, or do I need to specify explicit to publish all derived msg to interactionevent?

1 Like

Hi

You need a bit of mapping here. This can be done in two ways.

Subscriber-side

On the subscriber side you can inform the endpoint that, although the type accepted by the handler is InteractionEvent, the actual topics you are interested in are named after the derived types:

topology.SubscribeTo<InteractionEvent>("InteractionSent");
topology.SubscribeTo<InteractionEvent>("InteractionOpened");
topology.SubscribeTo<InteractionEvent>("InteractionClicked");

This will make auto-subscriber to create these three topics instead and wire the subscription to them.

Publisher-side

Alternatively, you can tell the publisher to publish all its derived events onto the single interactionevent topic:

topology.PublishTo<InteractionSent>("interactionevent");
topology.PublishTo<InteractionOpened>("interactionevent");
topology.PublishTo<InteractionClicked>("interactionevent");

This second option requires less entities on the broker side but forces the subscribers to all have handlers for all the multiplexed derived events.

2 Likes

Thanks @SzymonPobiega!

With your explanation, it finally clicked for me :-).

Only concrete classes work, inheritance or any interfaces will not auto work, and you need to wire them up yourself. I think adding an example like your explanation to the manual would make it a little bit clearer (the manual doesn’t make it clear enough what does work automatically and what doesn’t).

A third option would be then to not subscribe the handler to the base class, but all the derived classes. Then I don’t need to do the manual wiring (my preference I think).

@remyvd We started tweaking the documentation slightly Extended sample for the mapping by danielmarbach Ā· Pull Request #7055 Ā· Particular/docs.particular.net Ā· GitHub

While you’re at that, it would be valuable to provide a sample of the mapping via configuration, briefly mentioned in the upgrade guide, under more advanced scenarios, but not in the topology document.

For clarification do you mean the options API?

Something similar, like we have here? Configuration • Azure Service Bus Transport • Particular Docs

Beautiful. I was missing the link to this Options API from both, the Topology page and the Upgrade page.

I think there is definitely room for improvement on the documentation page. I have already noted down once we have the two other PRs merged, it might be beneficial to go over the pages again to consolidate things a bit.

1 Like

Hi, I’ve been doing some preliminary investigation into how our migration process will look. We have a reasonably complex landscape with a decent bit of cases of events passing in-between services (or teams), so we’ll definitely have to make use of the migration topology as we move towards the upgrade.

I wanted to check a couple of conclusions/points from my investigation with you directly:

  • Is there any documentation on configuring the migration topology with NServiceBus on Azure Function app?

After looking at the IHostbuilder extension implementation I figured out a working setup for my spike by setting specific paths in configuration to make it create the topology as Migration topology and then modifying that topology through code later, but I’m not sure if there are better ways to configure this.

  • Is there any support for configuring events to ā€˜integrate’ with both the new and old topology at the same time?

Given that in many cases publisher and subscriber of our events are separate applications (often managed by different teams), getting all of their upgrade deployments synced would be quite hard.

Preferably we would use a configuration setup where we could mark an event as either to be published on both topics (so listeners on either can pickup the event), or to mark an event as subscribed on both topics (so startup creates two rules, 1 one of each topic). Second case probably has less chance of unnecessary double processing of messages in our case.

I did notice that a migrated event (in code configuration) which still has its rule configured on the old event topic (since we don’t automatically clear these) will still pick up events from the old topic, effectively creating a doubly subscribed situation. While this could maybe work as a solution for us, it would feel a bit shaky that part of the subscription only works due to ā€˜lingering’ configuration, rather than being consistent on a fresh environment.

Thanks in advance for any advice/information!

Hi!

Thanks for reaching out! Let me try to answer your questions.

  • Is there any documentation on configuring the migration topology with NServiceBus on Azure Function app?

Yes, we have an upgrade guide that specifically targets Azure Functions. It shows how you can load the topology from the configuration.

  • Is there any support for configuring events to ā€˜integrate’ with both the new and old topology at the same time?

We cover this one in-depth in the main upgrade guide where we stated

While it is possible to migrate events individually, it is currently not supported to partially migrate a delivery path of a single event.

This is based on the assumption that a mixed delivery path has all the downsides of the old topology and no benefits of the new one.

For autonomous team setup we recommend the following migration process

  • Each team upgrades to the latest version of the transport running the migration topology configured to common topic delivery (the ā€œold topologyā€).
  • Once all teams involved in a given event path (all publishers and subscribers) agree they want to migrate that path to the new topology, the migration process is kicked-off.
  • The teams that manage the subscribers need to act first, switching their configuration of the migration topology to topic-per-event. At this point the subscribers are receiving events thanks to the ā€œlingering configurationā€ you mentioned but this should be fine because all the teams are committed to migrating sooner rather than later
  • The team that managers the publisher updates their configuration of the migration topology to topic-per-event. After this change is deployed, the events bypass the bundle-1 topic and the subscriptions there (the ā€œlingering configurationā€) can be removed

Would that process work for you? We believe the most important part is building the consensus about committing to the upgrade within some defined time window so that the system does not rely on ā€œlingering configurationā€ forever.

Thanks for the extensive reply!

I completely missed that function upgrade page when I was looking around, that was exactly what I was hoping for so I can add it as a link to our internal documentation.

The suggested migration process sounds like a solid plan. I’ll have to discuss a bit internally to see if we can build the required consensus to migrate specific paths within reasonable timelines, but I think this should probably work for us.