If an application uses threads such as a C# Parallel.ForEach to iterate through configurations which each setup an endpoint and start it, race conditions occur writing to the diagnostics file. From he below log you can see only one of the Endpoint.Start calls succeeds.
2018-11-05 12:10:40.874 INFO DefaultFactory Logging to ‘C:\Logs\AppName’ with level Info
2018-11-05 12:10:41.249 INFO NServiceBus.LicenseManager Selected active license from C:\license.xml
License Expiration: 2246-04-13
2018-11-05 12:10:41.249 INFO NServiceBus.LicenseManager Selected active license from C:\license.xml
License Expiration:2246-04-13
2018-11-05 12:11:01.093 ERROR NServiceBus.WriteStartupDiagnostics Failed to write startup diagnostics
System.IO.IOException: The process cannot access the file ‘C:\AppName\bin\Debug.diagnostics\AppName-configuration.txt’ because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream…ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream…ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)
at NServiceBus.AsyncFile.d__0.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.WriteStartupDiagnostics.d__1.MoveNext()
It sounds like you have more than one endpoint running in single process, right? If so, are you making sure each endpoint has a unique name? The default diagnostics file name is {endpointName}-configuration.txt, so each endpoint should get their own separate diagnostics file.
Would it be possible to share some of your startup code? Why are you using Parallel.ForEach for this? Since our APIs are async, it’s actually somewhat difficult to call them correctly from inside a Parallel.ForEach, so I’m not sure what benefit you’d be getting from using that.
That appears to be the issue. I was not aware that each endpoint’s name needed to be unique. I named each endpoint after the Queue name it’s connecting to. This queue was hosted on multiple endpoints, so it turned out to be a bad choice for an endpoint name, but it makes sense in my design.
I question this restriction, as I’m not aware of how the endpoint Name plays a part in anything that we actually do with NServiceBus. I understand why you chose to use it for each configuration file, and I have no helpful solution to provide off the top of my head, but I do know a different way is entirely possible and I encourage you to think on it some more.
To answer your question, the reason I connected to each endpoint in parallel is to start up the application quicker. I ended up reverting to a normal for each loop and took the hit of each endpoint having to start up sequentially.
Imagine the same queue on three different RabbitMQ instances/virtual hosts/what have you. We use a single application to send messages to the same queue across all three RabbitMQ brokers, depending on which environment it’s supposed to go to. The only difference between the queue is the broker. So all three configurations are for the same queue, and I chose to name the endpoint after the queue it’s connecting to.
You can certainly question why we have a single service handling messages for multiple environments at once. But crazy restrictions and financial considerations forced our hand, basically because it’s entirely possible to implement and saved money and headaches.
That does make a bit more sense as to why you’d have multiple endpoints with the same name, but yes having those be separate processes per environment would be a much better idea.
Until that becomes something you could consider doing, I would recommend using SetDiagnosticsPath to set a custom location for each diagnostics file. That way you get a distinct diagnostics file for each endpoint instead of overwriting it like is happening currently.