Our application runs on Azure Service Fabric and is separated into services that handle different aspects of the application. We’ve got a service running on a stateless ASP.NET MVC service that manages the web client. In order to access data in other services (such as fill a grid), we’ve had to rely on callbacks to get this data.
I’ve attended Udi’s distributed system design course and have learned that using the Callbacks package should be avoided. We’re experiencing some really sub-optimal performance as well, with callbacks with even small payloads taking seconds to complete.
The solution I have in my head is to transition our Web service to a stateful ASP.NET MVC service, and when objects are created in other services, the web service would subscribe to those changes and update the stateful reliable collections. The web service would only hold the data it needs to display in the web app, any create/update/deletes would get published to be handled by the other services.
Does this solution make sense? Is there something I’m missing? Accessing the reliable collections within the web service should be faster, and gets us around using the Callbacks package.
what I understand from your description is the following. You use Callbacks mechanism to fetch some data from some services to use them in another one (as you mention in the example). If that’s the case, I’d propose to do a different thing as we don’t encourage to use messaging for querying. If you need to use some data to handle a request, you should query the service directly, without using the messaging infrastructure.
If you need these data only for displaying them, there are several options:
you use one service to gather all the data and then respond to the client in a single HTTP request
you modify the client to ask different services for data
you use reliable collections as a kind of a cache that you mentioned
you use a distributed cache
The solution depends on a scale, size of the data, throughput requirements and the time that you could use to change your solution. The first is the easiest one (the queried service acts as a kind of proxy). The second requires doing some work on the client side. The third could increase the size of the data for the service but introduces some resiliency (cache like). The last one is a major effort.
I’d propose to evaluate the options above from 1 to 4, in this order.
I think the idea I’m shooting for is your first option, but the main issue is in the “gather all the data” step. We’re gathering the data by using callbacks. I definitely agree that we shouldn’t be querying using the messaging infrastructure. I suppose for Option #1 this would mean opening up HTTP listeners for each service, so that our main “API” service could query all the other services and gather the data that way.
Alternatively, without modifying all the other services, we create the one main “API” service, which is stateful and more or less caches the current state, being updated by subscribing to changes in the other services. I’m assuming this would be faster than gathering the data from other services via HTTP, but could run into storage issues in the main API service if the amount of data increases drastically.
You are correct, Option #1 would require an additional HTTP listener (or Service Fabric Remoting if you don’t need to access it in any other way. This could be faster). Maybe phrasing it as “gather all the data” was not that fortunate
The main API service is another way to go. There are things to consider though:
Storage size (as you pointed it), which might unbounded
A possible leaky implementation - when you apply events outside of a service, you may leak some logic to other parts of the system (how to apply events on a local state). If you stick to the query approach, the logic will be in the service that raises events.
Speed - you mention that querying a local state could be faster. That’s correct if queries are simple enough to get them translated to methods provided by Persistent Collections. IMO speed is the last thing that you should worry about.
