Azure Service Bus with NSB 7 and .net core

danielmarbach · December 5, 2019, 5:41pm

Did you have the publish call in a synchronous execution path?

Could you elaborate a bit your scenario so that we can have a closer look? Maybe share some snippets?

Regards
Daniel

alexander-nanay · December 11, 2019, 5:10pm

Sorry for delayed response - missed a notification.
Unfortunately awaiting the publishing call solved the problem only temporary. Perhaps it was just a coincidence.
It seems like the root cause is in topic bundles (bundle-1, bundle-2) of Azure Service Bus which we use as a transport.
NSB 7 seem to create subscription only in bundle-1 while NSB 6 creates it in both bundle-1 and bundle-2.
When I looked into the publisher debug logs I’ve found the following line:
DEBUG DefaultOutgoingBatchRouter - Routing 1 messages to bundle-2.

Seems like new NSB 7 subscriber is not receiving the message because it’s listening only in “bundle-1”, while NSB 6 subscribers are listening in both “bundle-1” and “bundle-2”. Publisher is routing to one bundle or another - therefore it is unstable.

Publisher is .Net Framework 4.6.1 and using the following nuget packages:

<package id="NServiceBus" version="6.3.4" targetFramework="net461" />
<package id="NServiceBus.Autofac" version="6.0.1" targetFramework="net461" />
<package id="NServiceBus.Azure.Transports.WindowsAzureServiceBus" version="7.2.4" targetFramework="net461" />
<package id="NServiceBus.DataBus.AzureBlobStorage" version="1.1.0" targetFramework="net461" />
<package id="NServiceBus.Log4Net" version="2.0.0" targetFramework="net461" />

New subscriber is .Net Core App 2.2 with the following nuget packages:

<PackageReference Include="NServiceBus" Version="7.1.10" />
<PackageReference Include="NServiceBus.MSDependencyInjection" Version="0.1.4" />
<PackageReference Include="NServiceBus.Newtonsoft.Json" Version="2.2.0" />
<PackageReference Include="NServiceBus.Transport.AzureServiceBus" Version="1.2.1" />

Publisher config (NSB 6):

endpointConfiguration.UseTransport<AzureServiceBusTransport>()
                        .ConnectionString(Bus)
                        .UseForwardingTopology()
                        .Transactions(TransportTransactionMode.SendsAtomicWithReceive)
                        .Sanitization().UseStrategy<Sha1Sanitization>();

Subscriber config (NSB 7):

 configuration.UseTransport<AzureServiceBusTransport>()
                .ConnectionString(options.ConnectionString)
                .Transactions(TransportTransactionMode.SendsAtomicWithReceive)
                .RuleNameShortener(Sha1Sanitization.ShortenerFor(TransportEntity.Rule)) 
   .SubscriptionNameShortener(Sha1Sanitization.ShortenerFor(TransportEntity.Subscription))
                .EntityMaximumSize(1);

I feel like this is some common problem, but couldn’t find any docs on solution.
Would be very glad if you point me in the right direction.

Best regards,
Alex

danielmarbach · December 12, 2019, 10:55am

Hi

That’s interesting. As far as I remember the default number of entities in a bundle is set to 1. Is there any endpoint that actually requires bundle-2?

Can you check whether any of the endpoints has set the number of bundles explicitly?

forwardingTopology.NumberOfEntitiesInBundle(2);

if not and if all v6 compatible publishers are on 7.2.4 then you should be able to delete bundle-2 on the broker.

@SeanFeldman can you verify and confirm?

Regards
Daniel

alexander-nanay · December 12, 2019, 11:54am

@danielmarbach,

Thanks for your help.

No, we don’t use NumberOfEntitiesInBundle api.
Haven’t found docs, but according to the code it defaults to 2 bundles in v.7.2:

github.com

Particular/NServiceBus.AzureServiceBus/blob/release-7.2/src/Transport/Topology/Topologies/ForwardingTopology.cs#L43


      
          }
          
          
void ApplyDefaults(SettingsHolder settings)
          {
              new DefaultConfigurationValues().Apply(settings);
              // ensures settings are present/correct
              settings.SetDefault(WellKnownConfigurationKeys.Topology.Addressing.Composition.Strategy, typeof(FlatComposition));
              settings.SetDefault(WellKnownConfigurationKeys.Topology.Addressing.Individualization.Strategy, typeof(CoreIndividualization));
              settings.SetDefault(WellKnownConfigurationKeys.Topology.Addressing.Partitioning.Strategy, typeof(SingleNamespacePartitioning));
              settings.SetDefault(WellKnownConfigurationKeys.Topology.Addressing.Sanitization.Strategy, typeof(ThrowOnFailedValidation));
              settings.SetDefault(WellKnownConfigurationKeys.Topology.Bundling.NumberOfEntitiesInBundle, 2);
              settings.SetDefault(WellKnownConfigurationKeys.Topology.Bundling.BundlePrefix, "bundle-");
              topologySectionManager = new ForwardingTopologySectionManager(settings, container);
          }
          
          
void InitializeContainer(SettingsHolder settings)
          {
              // runtime components
              container.Register<ReadOnlySettings>(() => settings);
              container.Register<ITopologySectionManager>(() => topologySectionManager);
              container.RegisterSingleton<NamespaceManagerCreator>();

I can see the issue is already mentioned as “potential message loss”, but for different version though

github.com/Particular/NServiceBus.AzureServiceBus

The bundles feature for the ForwardingTopology can be missconfigured on different endpoints causing message loss

opened 02:11PM - 18 Apr 17 UTC

closed 06:00AM - 09 Jun 17 UTC

SeanFeldman

Bug

## Who's affected You are affected if you are using ForwardingTopology and cu…stom number of topics in bundle using [`NumberOfEntitiesInBundle(int)` API](https://docs.particular.net/nservicebus/azure-service-bus/configuration/full#configuring-the-topology-forwarding-topology). ## Symptoms Misconfigured publisher and subscriber cause events to be posted to the topics that are not listened to, causing events to be lost. ## More details As part of the conversation in #521 it was identified that the scenario for a message loss could exist when `ForwardingTopology` is used and number of topics in bundles are miss-configured on different endpoints (see [this visualization](https://github.com/Particular/NServiceBus.AzureServiceBus/issues/521#issuecomment-294614402)). We could have a startup check to verify that number of topics per bundle is aligned with the number of actually found topics. If that's not the case, an exception should be thrown preventing endpoint from starting until bundles configuration is resolved. ## Affected versions - [x] `support-7.2` --> PR #565 - [x] `develop` --> PR #570 - [x] Announcement --> PR https://github.com/Particular/Announcements/pull/149

I’ve tried to delete bundle-2 but then publisher is not able to publish a message:

Failed to dispatch a batch with the following message IDs: ...
40400: Endpoint not found., Resource:sb://<our-servicebus>.servicebus.windows.net/bundle-2.

It’s weird I haven’t found this problem mentioned anywhere in migration docs (or I’m just not that much into NSB terminology).

We have around 15-20 services running with legacy Azure ServiceBus Transport, but we want to use .Net Core for new endpoints (with new Azure ServiceBus Transport).
Most of our services are on 7.1-7.2 version of Legacy Azure Transport, but there are couple on 6.4 and couple on 8.
Any thoughts on how to make it work together or migration plan to (non-legacy) Azure ServiceBus transport?

danielmarbach · December 12, 2019, 12:20pm

Yeah sorry my research led me to this

github.com

Particular/NServiceBus.AzureServiceBus/blob/1d01629441ca5a0b50cf37d296f71383220c7fcd/src/Transport/Topology/Topologies/ForwardingTransportInfrastructure.cs#L15


      
          using Transport.AzureServiceBus;
          
          
class ForwardingTransportInfrastructure : AzureServiceBusTransportInfrastructure
          {
              ForwardingTopologySectionManager topologySectionManager;
              string bundlePrefix;
              volatile int initializeSignaled;
          
          
    public ForwardingTransportInfrastructure(SettingsHolder settings) : base(settings)
              {
                  settings.SetDefault(WellKnownConfigurationKeys.Topology.Bundling.NumberOfEntitiesInBundle, 1);
                  settings.SetDefault(WellKnownConfigurationKeys.Topology.Bundling.BundlePrefix, "bundle-");
              }
          
          
    protected override ITopologySectionManagerInternal CreateTopologySectionManager(string defaultAlias, NamespaceConfigurations namespaces, INamespacePartitioningStrategy partitioning, AddressingLogic addressing)
              {
                  var endpointName = Settings.EndpointName();
                  var numberOfEntitiesInBundle = Settings.Get<int>(WellKnownConfigurationKeys.Topology.Bundling.NumberOfEntitiesInBundle);
                  bundlePrefix = Settings.Get<string>(WellKnownConfigurationKeys.Topology.Bundling.BundlePrefix);
          
          
        topologySectionManager = new ForwardingTopologySectionManager(defaultAlias, namespaces, endpointName, numberOfEntitiesInBundle, bundlePrefix, partitioning, addressing);

Only realized later that we have shifted things a bit.

So the fix is the following

Set forwardingTopology.NumberOfEntitiesInBundle(1); on all the v6 publishers
Delete bundle-2

I’ll see if we can make that a bit clearer in the upgrade guides

Regards
Daniel

danielmarbach · December 12, 2019, 12:49pm

Alexander,

I did some more digging. Apparently we removed the documentation for NumberOfEntitiesInBundle at some point because we wanted customers to stop using it. v8 and v8 of the legacy transport already restrict the usage to one bundle and only allow to change the bundle prefix.

We already have compatibility guidance available here

but unfortunately that didn’t mention the fact that the number of entities in the bundle need to be set to one for v7 publishers. I’ve added a note

Regards
Daniel

alexander-nanay · December 12, 2019, 2:00pm

Thanks for all your help, we will plan and update our services.
As far as I understand it’s vital to set NumberOfEntitiesInBundle to 1 for publishers, but it’s not that important for subscribers which are not publishing (they will listen on both bundles, but receive actual messages only on first).