Switching transport might be something that should not be underestimated from an operational and migration standpoint especially if you need to keep things running while migrating.
Have you considered lowering the transactionality level? For example, do you need MSDTC everywhere? Could you slightly redesign your handler and business logic to be able to deal with potential duplicates? With V6 and higher with NServiceBus by default, we batch all the messages in a handler and only send them out when the message handler and the pipeline completed successfully. This already decreases the chance of things going wrong for example.
Based on my experience with MSMQ I can say MSMQ is a super robust transport infrastructure. It can get “wacky” with MSDTC, and the two-phase commit reduces the throughput quite “drastically.”
NServiceBus by default uses a safe-by-default approach. That’s why when the transport and persister support it tries to leverage the highest transactionality level possible. With SQL Transport and Persistence, this means it would use the transactionality level which allows escalating to distributed transactions. This is ideally suited for existing applications that are migrated to distributed systems or customers who are building smaller systems and that don’t want to adopt the distributed systems thinking just yet due to tight timelines or only little scaling needs at the moment. Similarly, the outbox is a feature that allows achieving distributed transaction like behavior without having to have a distributed transaction coordinator. Which makes the outbox a bit more convenient to operate as well as less complex to administrate (gone are the days when the DTC has to be clustered, or an administrator needs to log into a local box to manually resolved a distributed transaction in an in-doubt state). While the outbox is more convenient than the DTC both the DTC and the outbox have something in common: They are a technical solution that tries to address idempotency and consistency while inflicting scaling constraints into your distributed systems architecture. Coordination whether it is from a DTC standpoint or the outbox involves IO. IO is not cheap and therefore limits the throughput of your system. It might sound like I’m totally against DTC and Outbox. That is not the case since they are perfectly valid solutions depending in your needs and that is what I’m trying to outline and let’s say challenge conventional thinking.
Having said that with modern architectural approaches like Microservices, Containers and more where non-functional requirements push you towards building a robust distributed system that allows being scaled dynamically and potentially even be lifted-and-shifted to the cloud or PaaS running on premises a different mindset is required. Since you are building a new Microservice platform, you have the steering wheel in your hands to make sound architectural decisions (if your non-functional requirements demand so) to drive your Microservices towards a design which does not rely on technical solutions to idempotency and consistency. In my personal belief we have to free ourselves from the notion of technical transactions and the utopia of technical consistency. Business processes always had clever ways of embracing the fact that consistency can be achieved by issuing compensations. So I would encourage you to think about the question how you can evolve your Microservices so that they can cope with the fact that business commands might be retried and happen multiple times. By doing so, you achieve a much more scalable solution in the long run because it requires you to think about business transactions and race conditions up-front and explicitly design for those instead of relying on some “black magic” solution that handles it behind the scenes. And btw. even when you look at DTC or outbox there is, for example, nothing prevents a user to submit the “same business information” twice. Your system needs to deal with those kinds of scenarios as well.
We have an excellent article in the Azure documentation that explains that very well
Although the article is filed under Azure the same applies to on-premises solutions.
Furthermore, Pat Helland wrote an excellent paper about the life beyond distributed transactions.
Another thing: If the-the future you’d decide to switch the transport let’s say to RabbitMQ, that transport only supports the transaction mode ReceiveOnly. Therefore not even outgoing messages are transactionally safeguarded in some scenarios. Therefore your code would need to deal with such situations as well.
I hope that answers your questions or at least give you hints into the direction you want to take in your architecture