RabbitMQ with rotating credentials


(Bill Staff) #1

My company is considering using HashiCorp Vault to handle RabbitMQ credentials. This would mean that the Rabbit logins would expire and be renewed. Is there a good way to handle this? I do not see a way to change the connection string in an endpoint. I could just create a new endpoint each time these credentials change.

Has anyone done something like this?

(Brandon Ording) #2

If you need to update your connection strings, you’ll need to restart any endpoints using the old connection string.

(Bill Staff) #3

What if I generated a new IMessageSession per scope. Are there any know adverse behaviors besides a performance hit?

(Brandon Ording) #4

If I understand what you’re saying, you’d need to call Endpoint.Start to create new endpoint every time you want to send a message. That is definitely not something we recommend. In fact, you’d have to create a new EndpointConfiguration instance each time as well since they can’t be reused.

You’d also need to manage shutting down all of those endpoints correctly as well, or they’d all still be active, with their own connections to the broker running.

That also assumes these endpoints would only be used for sending messages and aren’t also receiving them. If you’re also receiving messages, then this definitely would not make sense since endpoints need to be long-running to process messages in a queue.

Overall, I don’t think it makes sense to do what you’re suggesting. It’s a much better idea to allow the new credentials to be picked up naturally during a process restart.

(Bill Staff) #5

It sounds like I need to have my process die when it looses connection. Is there a good place to do this for a process that only waits for messages?

(Bill Staff) #6

Never mind. Did a 10 second google search and found this:

(Bill Staff) #7

I take that back. I still have no idea how to trigger a shutdown when my Handler process losses a connection.

(Brandon Ording) #8

The critical error documentation is what you need. If the receive connection goes down and stays down, that will cause the circuit breaker to trigger the critical error action. You can use the critical error action to do whatever you want, including shut down the process.

See https://docs.particular.net/nservicebus/hosting/critical-errors#a-possible-custom-implementation as an example of how to do this.

(Bill Staff) #9

I’ve been trying this out on a Process with only a RabbitMQ handler. If the username and password are deleted from the rabbit server, the process does not attempt to reconnect. Almost as if the heartbeat still works. If I restart my process it fails to connect due to the lack of credentials.

If I restart my local rabbit service while my process is running, I do get the critical error plus re-connection attempts.

(Brandon Ording) #10

That’s not really anything we can control. It sounds like RabbitMQ itself doesn’t invalidate existing connections using old/removed credentials.

If you’re wanting to update credentials on a regular basis, why would you want existing connections to stop working anyway? It seems more like you’d want both the old and new ones to co-exist for sometime, and only remove the old ones once everything is using the new ones.

Doing it that way allows you to have a seamless transition without causing connectivity issues.

(Bill Staff) #11

We are doing it that way. I’m trying to guard against a scenario where the HashiCorp Vault (rotates credentials) credential gets blown-out during some upgrade or for any other reason.
One reason is if Hashicorp Vault itself goes a little crazy and fails to remove its own expired credentials. The fix is to blow them all out. I know these are edge cases, but when these happen, a listener process will assume nothing is wrong.

I see what you are saying. You are bound to Rabbits Heartbeat functionality. One thought I now have is to have a background thread that tries to create a new throw away endpoint with the existing connection string. If it fails, then the one in use is most likely dead too.

(Ramon Smits) #12

If you do not require high availability with rolling keys then I would suggest to just add a critical error callback and let the host die to have it restarted in your environment. This works well if you only reset the password for a specific account.

If you have a new set of credentials so that you do not have downtime during key rotation then you would need to launch a new instance. The easiest way for this would be to just have a automated launch/teardown script on your server that is invoked in the same cadence of your key renewal.

I personally would try avoid leaking this operational concern into your endpoint code.