Integration of Kafka with NServiceBus: Your Input Needed!

1

Hi everyone,

We are currently exploring the possibility of integrating systems that use Kafka with those that use NServiceBus. As we continue to develop our solution, we want to ensure that we’re meeting the needs of our customers effectively.

If you’re using Kafka and also have systems that rely on NServiceBus, we’d love to hear from you:

  • Is integration between these two systems important to your workflow?
  • Do you anticipate integrating these technologies?
  • How are you currently managing interactions between Kafka and NServiceBus, if at all?
  • What challenges have you faced ?
  • What features or capabilities would you like to see in an integration solution?

Additionally, we’re interested to know if any of the following scenarios are relevant to your use case:

  • Importing and exporting messages to and from your NServiceBus system from an external system publishing/consuming messages from Kafka topics.
  • Automating long-running business processes via message queues, in other words do you want to run NServiceBus endpoints using Kafka?
  • Manipulating, processing, and reacting to the event streams in real-time as well as retrospectively. Are there problems or opportunities related to this you think the Particular Software team could address?

Your feedback will be invaluable as we work to create a seamless experience for those utilizing both Kafka and NServiceBus in their environments.

Thank you in advance for your insights!

2 Likes
2

Prior to joining Particular, I worked for a large system integrator.

We dealt a lot with production lines (in that particular case, it was food processing). In that scenario events (or better signals) coming from the production line, when looked at in isolation, were mostly insignificant.

For example, reading the temperature of an oven every 5 seconds can have lots of fluctuations and a single event has little meaning. We were mostly interested in trends, if the temparature went above a warning threshold during the last 5 readings then raise an alarm, because we’re burning food.

Those sensors readings were streamed from the production line and we were buffering/sampling them looking for business meaningful events. Picturing it today in Kafka, the production line would stream to Kafka every single event, a “Kafka” handler buffers a few of them and only if certain conditions are met it publishes the NServiceBus event.