I’ve just started at an organization that seems to have an equal number of Java and .NET processes. They’ve started down the road of a homegrown “service bus” (really a single process that marshals RabbitMQ messages to webservice calls to Java or .NET back-end processes). I’m not sure how to recommend NServiceBus, as it would provide a .NET solution but still seem to require hand-rolled code for Java processes to interoperate with NServiceBus.
We are migrating to Azure, if that is considered a variable in the equation.
Thanks for reaching out. I am a bit worried by this statement
They’ve started down the road of a homegrown “service bus” (really a single process that marshals RabbitMQ messages to webservice calls to Java or .NET back-end processes)
because this approach is likely to destroy good part of the potential benefits of messaging. The great advantage of messaging systems is the ability to receiver a message, update the data store and send out other messages in a single atomic step.
This is how NServiceBus started back then with MSMQ but now it supports this mode of operation on all transports thanks to the Outbox.
Having a single process that receives messages and translates them to webservice calls breaks this because the service being called can’t really send anything to the next service in the chain, at least not in an atomic way and that atomicity is something that makes building complex distributed systems manageable. Without it you need to deal with partial failures (e.g. data updated but messages not sent or vice versa) in each and every component of the system.
In the principle NServiceBus is not that different from the good old days J2EE message-driven beans. I am not sure if there exists a modern version of this idea in the Java space but worst case you should be able to put together something that runs in Java and works like NServiceBus or message-driven beans i.e.
the infrastructure receives a message (from RabbitMQ)
it looks up for handlers that match the message
it opens a transaction with a data store
passes the invocation to these handlers along with the transaction context
the handlers return the messages they would like to send out
the infrastructure stores the outgoing messages in that transaction that the handlers used
it inserts a marker that the received message had been processed and commits the transaction
This is how the Outbox works. Such a lightweight framework would provide solid foundations for further extensions because it clearly separates the business code (the message handler) from the infrastructure (managing transactions and messages).
OK, so I got a more detailed explanation of their infrastructure. They’re actually using an open source tool written in python called Celery. Also, their process manager/sagas have no compensating behavior, and when I asked them about it, they cannot foresee the possibility of ever needing compensating behavior. They are purely sets of HTTP API calls that need to be made when a message is consumed. They recognize that all of these APIs need to be fully idempotent, and that if this process crashes or is recycled that the sequence of API calls will start over from the beginning when the message is consumed again (there is not any kind of saga state).
Anyway, I told the dev I was working with that my intuition was that this would work fine for awhile, but one day there would surely be an “OH F#$%” moment, i.e. Day of Reckoning. He wrote it off, as all software architectures work until they don’t. :-
So what are enterprises that run on a mix of Java and .NET actually doing these days? Hand-rolling their own cross-platform service bus?
Yeah, that’s a really good question. From time to time we get to hear from customers that they run NServiceBus as part a bigger and more technologically-diverse environment but we don’t have much information on what’s used in the other tech stacks.
Maybe some people in the community would be able to chime in?
I’ve spent a lot of time using Mulesoft in those heterogeneous environments but I’m looking for a change (very tired of Mulesoft), so I’m asking the same question you guys are.
We are seriously considering NServiceBus, but we have services written in Python that we wonder about the extra complexity to integrate. Wondering if we lose things like retries and some of the ServiceInsight tracing ability when some nodes are in Python.
all NServiceBus transports support (in some form) the concept of native integration. That means an NServiceBus endpoint can receive messages from endpoints not using NServiceBus and written in different languages.
If an NServceBus endpoint fails to process a native message, that message can be retried without any issue. For ServiceControl to retry a message, IIRC, the only requirement is that the failed message has a FailedQ header with the name of the queue where the message originally failed. That means that even a Python endpoint could be sending failed messages to ServiceControl via the error queue.
Tracing in ServiceInsight is a bit more involved and requires a deeper understanding of what kind of information you want to visualize there from non-NServiceBus endpoints.
