Azure Service Bus Limits

Architecture & Design
1

Hi, our implementation of NServiceBus uses Azure Service Bus.
We have now reached the limit of 2000 Sql Filters for a topic (the default topic of bundle-1)
Is there any guidance or pointers on how to implement multiple topics and use them correctly?
Any tips would be greatly appreciated

2

@Gobba, could you please elaborate how did you get to 2000 filters?
Was it 2K NServiceBus events or is the namespace is shared by other sub-systems in the organization and those rules are not NServiceBus specific?

3

They are NServiceBus events. We are using MS Service Fabric and multiple nodes. The issue arose when we increased the number of nodes to 5, with one of the services subscribing to a large number of Events (for end user notification). Each instance of this service has its own endpoint for a node, so the count of SqlFilters in Azure ServiceBus was essentially multiplied by 5
Hope this makes sense.
Is there a way we could subscribe to an interface or a base class used by all the events?

Thanks
Alan

4

Each instance of this service has its own endpoint for a node

Are you partitioning your data and is that the reason for this setup? One of the options you have with Service Fabric is partitioning and routing messages to the correct partitions. Would Service Fabric Partition-Aware Routing with one of the routing strategies work for you?

5

Thanks for your reply Sean, much appreciated

Are you partitioning your data and is that the reason for this setup?<<
Yes we are

Rather than have SqlFilters for each event, is there a way we could subscribe to an interface, base class or property used by all the events? That way we could actually reduce the number of filters but achieve the same result

6

Yes you could have:

class EventX : IEvent
{
}

class EventY : EventX
{
}

class EventZ : EventX
{
}

class Handler : IHandleMessages<IHandleMessages>
{
    public async Task Handle(EventX message, IMessageHandlerContext context)
    {
        // c# 7+
        switch (message)
            case EventX t1: 
            case EventY t2:
            case EventZ t3:
    }
}

However, that approach will make processing itself more complex. An alternative would be to disable auto subscribe and to explicitly only subscribe on the shared (marker) interface.

Guidance:

endpointConfiguration.DisableFeature<AutoSubscribe>();
// ...
await endpoint.Subscribe<EventX>();

A problem is that now the endpoint could receive messages but not have a handler for it. Such messages would currently be moved to the error queue. This could maybe be mitigated by using a custom recoverability policy.

Still I would not recommend doing any of this but address the cause of this which seems to be in the hosting and endpoint creation. So I also would like to know the answer regarding Sean’s question.

7

Thanks for your assistance @SeanFeldman and @ramonsmits.
It is an inherited system which I am trying to understand and simplify the processing.
I am now investigating reworking the system to reduce the number of subscriptions so that:

  • There are multiple instances of a WebAPI stateless service
  • Commands are routed from the WebAPI to appropriate endpoints for various services to subscribe to. These are mostly orchestrators of multiple sagas which can be long processes with components handled by services subscribing to different endpoints.
  • When all processing is complete, events are published and the WEBAPI subscribes to these events via a single endpoint. Previously there was a specific endpoint for each instance of the WEBAPI (which increased the number of subscriptions having one per event per instance of the WebAPI endpoint).
  • I now want to handle the event by the originating WebAPI service to send back to the calling client through websockets tied to this specific instance of a WebAPI
  • The WebAPI instances are competing consumers so any instance can handle the notificaton event, not necessarily the instance containing the websocket link. Subsequently, notifications are mostly lost

Does this make sense?
Am sure this is a common scenario and have seen various discussions but am yet to find a recommended approach.

Any pointers would be appreciated

8

@Gobba If your web api instances are fully indeed stateless then I don’t understand why each instance would require its own queue. It seems all these instances being “subscribers” isn’t really correct as logically they are the same but just scaled out.

You state that you have calling clients that you want to send data to. Maybe a better approach is to use SignalR for that notification mechanism to respond to the connected client?

We have a sample that shows how to use messaging with NServiceBus and notifications handling with SignalR:

1 Like