The team working on ServiceControl on Linux containers will be watching this thread for questions and feedback, since the documentation is not 100% complete.
While we aren’t 100% done with all the documentation, we know you’re all looking forward to having ServiceControl on Linux containers. So today, we’re doing a soft release so you can test things out.
Note that it’s highly recommended that the Error and Audit instances both have independent database containers.
We will be working on shaping up the official documentation over the next couple weeks, but we think the documentation for each image on Docker Hub should give you enough to play with if you just can’t wait. We’ll continue to post updates on this thread as work on the documentation progresses.
We’ve also created a dedicated thread on our discussion site for ServiceControl on Linux containers early access questions. We’ll be monitoring that thread carefully for questions and feedback. We’d love to hear from you there. There’s still the opportunity to incorporate feedback into additional releases during the month of July.
We know you’ve been waiting for this for a long time. Take it for a spin and let us know what you think.
Really excited to see this, and great work on such a huge task.
I’ve been able to successfully get the containers up and running, with the one exception of servicecontrol-monitoring - it’s running and port 33633 is open, but any requests to the port time out, either in ServicePulse or just browsing to http://localhost:33633
Config is basically the same as the main ServiceControl container, which is working fine… any tips here?
To add to this, in the logs I can see requests hitting ServiceControl monitoring:
eg. 2024-06-28 00:41:13.4886|21|Info|Microsoft.AspNetCore.Hosting.Diagnostics|Request finished HTTP/1.1 GET http://localhost:33633/connection - 200 - application/json;+charset=utf-8 2.1517ms but not getting any response.
A postman GET request to http://localhost:33633does return the expected JSON { "instanceType": "monitoring", "version": "5.3.0" } but an equivalent browser request just hangs… the ServicePulse test connection utility similarly just hangs
I tried running these in an Azure Container Apps environment. Feedback thus far:
I created an ingress to expose the service control API via HTTPS, however when the /api endpoint is called, the base URL for all of the URLs returned by the endpoint incorrectly have their scheme set as HTTP rather than HTTPS. Perhaps the X-Forwarded-Proto header is not being used to construct the base URL and UseForwardedHeaders needs to be enabled for service control?
I can’t seem to figure out how to customise the service name in order to customise the queue names. It doesn’t appear to be an environment variable. I must be missing something.
Your 2nd question is easier, I’ll take that first. The service name is a holdover from Windows Services that makes a lot less sense in a container world. So we think we’re going to re-analyze that a little bit, but as a workaround in the meantime, you can use these (quite inconsistently named) environment variables that happen to work to change the queue names:
SERVICECONTROL_INTERNALQUEUENAME
SERVICECONTROL_AUDIT_INTERNALQUEUENAME
MONITORING_ENDPOINTNAME
For the first question, I’d love to see your ingress implementation. Would you be willing to get on a call? I’ll send you a direct message with a link where you can set up a time with me.
EDIT: Actually, I think I understand what you’re saying now. You looked at the URLs output by the default route and they didn’t accurately reflect the reverse proxy settings. Those are just informational so they won’t (shouldn’t) break anything, but I think that will be fixed by this PR which we’ll include in the next release.
I scheduled a call but then after that saw you edited your post. The URLs output from the /api route and the ingress (reverse proxy) headers is exactly what I meant and the PR linked to would resolve the issue, thank you.
For the queue name customisation, I will use the provided environment variables for now. I assume they may change going forward if they are inconsistently named.
The next things I intend to look at are:
Persistent storage for the RavenDB containers
Adding authentication - this is probably a bigger topic and although authentication can be bolted on without any code changes in Azure Container Apps, I am not sure how the Service Pulse UI loaded in the browser would allow for authenticated calls to the service control and monitoring APIs
Authorisation, if at all possible. For example, certain users should be able to perform a subset of operations on the service control APIs
It is great that I can now use system assigned managed identities to connect to Azure Service Bus (the transport that we use) from the error, audit and monitoring instances, which allows for granular role assignment on queues that are pre-created for each instance. I am not sure if it was already possible in the past, but it it is certainly easier now.
When the docs are ready, would there be guidance on migrating existing embedded databases (we’re already on the new format) and preserving failed/archived messages?
I took a stab at getting this to run in kubernetes (AWS EKS) using the SQS transport. I’ve hit an issue. For some background, we use IAM Roles for Service Accounts to provide AWS credentials. When I try to run the init / setup container under these credentials, the AWS SDK throws an error about a missing assembly. I think, from experience, the fix is to just include this assembly in the build output (AWSSDK.SecurityToken)
stack trace
servicecontrol-audit-8445b8c47c-4775x setup 2024-07-08 15:14:49.8180|1|Error|Program|Unhandled exception was caught.|System.InvalidOperationException: Assembly AWSSDK.SecurityToken could not be found or loaded. This assembly must be available at runtime to use Amazon.Runtime.AssumeRoleAWSCredentials.
servicecontrol-audit-8445b8c47c-4775x setup ---> System.IO.FileNotFoundException: Could not load file or assembly 'AWSSDK.SecurityToken, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
servicecontrol-audit-8445b8c47c-4775x setup
servicecontrol-audit-8445b8c47c-4775x setup File name: 'AWSSDK.SecurityToken, Culture=neutral, PublicKeyToken=null'
servicecontrol-audit-8445b8c47c-4775x setup at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
servicecontrol-audit-8445b8c47c-4775x setup at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.GetSDKAssembly(String assemblyName)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.LoadTypeFromAssembly(String assemblyName, String className)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.LoadServiceConfigType(String assemblyName, String serviceConfigClassName)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.CreateServiceConfig(String assemblyName, String serviceConfigClassName)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.CreateClient()
servicecontrol-audit-8445b8c47c-4775x setup --- End of inner exception stack trace ---
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.CreateClient()
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.GenerateNewCredentialsAsync()
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.RefreshingAWSCredentials.GetCredentialsAsync()
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at NServiceBus.Transport.SQS.QueueCreator.CreateQueueIfNecessary(String address, Boolean createDelayedDeliveryQueue, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.SQS/QueueCreator.cs:line 37
servicecontrol-audit-8445b8c47c-4775x setup at NServiceBus.SqsTransport.Initialize(HostSettings hostSettings, ReceiveSettings[] receivers, String[] sendingAddresses, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.SQS/Configure/SqsTransport.cs:line 260
servicecontrol-audit-8445b8c47c-4775x setup at ServiceControl.Transports.TransportCustomization`1.ProvisionQueues(TransportSettings transportSettings, IEnumerable`1 additionalQueues) in /_/src/ServiceControl.Transports/TransportCustomization.cs:line 132
servicecontrol-audit-8445b8c47c-4775x setup at ServiceControl.Audit.Infrastructure.Hosting.Commands.SetupCommand.Execute(HostArguments args, Settings settings) in /_/src/ServiceControl.Audit/Infrastructure/Hosting/Commands/SetupCommand.cs:line 36
servicecontrol-audit-8445b8c47c-4775x setup at ServiceControl.Audit.Infrastructure.Hosting.Commands.CommandRunner.Execute(HostArguments args, Settings settings) in /_/src/ServiceControl.Audit/Infrastructure/Hosting/Commands/CommandRunner.cs:line 15
servicecontrol-audit-8445b8c47c-4775x setup at Program.<Main>$(String[] args) in /_/src/ServiceControl.Audit/Program.cs:line 27
servicecontrol-audit-8445b8c47c-4775x setup at Program.<Main>(String[] args)
servicecontrol-audit-8445b8c47c-4775x setup Unhandled exception. System.InvalidOperationException: Assembly AWSSDK.SecurityToken could not be found or loaded. This assembly must be available at runtime to use Amazon.Runtime.AssumeRoleAWSCredentials.
servicecontrol-audit-8445b8c47c-4775x setup ---> System.IO.FileNotFoundException: Could not load file or assembly 'AWSSDK.SecurityToken, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
servicecontrol-audit-8445b8c47c-4775x setup
servicecontrol-audit-8445b8c47c-4775x setup File name: 'AWSSDK.SecurityToken, Culture=neutral, PublicKeyToken=null'
servicecontrol-audit-8445b8c47c-4775x setup at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
servicecontrol-audit-8445b8c47c-4775x setup at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.GetSDKAssembly(String assemblyName)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.LoadTypeFromAssembly(String assemblyName, String className)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.LoadServiceConfigType(String assemblyName, String serviceConfigClassName)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ServiceClientHelpers.CreateServiceConfig(String assemblyName, String serviceConfigClassName)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.CreateClient()
servicecontrol-audit-8445b8c47c-4775x setup --- End of inner exception stack trace ---
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.CreateClient()
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.GenerateNewCredentialsAsync()
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.RefreshingAWSCredentials.GetCredentialsAsync()
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
servicecontrol-audit-8445b8c47c-4775x setup at NServiceBus.Transport.SQS.QueueCreator.CreateQueueIfNecessary(String address, Boolean createDelayedDeliveryQueue, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.SQS/QueueCreator.cs:line 37
servicecontrol-audit-8445b8c47c-4775x setup at NServiceBus.SqsTransport.Initialize(HostSettings hostSettings, ReceiveSettings[] receivers, String[] sendingAddresses, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.SQS/Configure/SqsTransport.cs:line 260
servicecontrol-audit-8445b8c47c-4775x setup at ServiceControl.Transports.TransportCustomization`1.ProvisionQueues(TransportSettings transportSettings, IEnumerable`1 additionalQueues) in /_/src/ServiceControl.Transports/TransportCustomization.cs:line 132
servicecontrol-audit-8445b8c47c-4775x setup at ServiceControl.Audit.Infrastructure.Hosting.Commands.SetupCommand.Execute(HostArguments args, Settings settings) in /_/src/ServiceControl.Audit/Infrastructure/Hosting/Commands/SetupCommand.cs:line 36
servicecontrol-audit-8445b8c47c-4775x setup at ServiceControl.Audit.Infrastructure.Hosting.Commands.CommandRunner.Execute(HostArguments args, Settings settings) in /_/src/ServiceControl.Audit/Infrastructure/Hosting/Commands/CommandRunner.cs:line 15
servicecontrol-audit-8445b8c47c-4775x setup at Program.<Main>$(String[] args) in /_/src/ServiceControl.Audit/Program.cs:line 27
servicecontrol-audit-8445b8c47c-4775x setup at Program.<Main>(String[] args)
@jfeinour is correct about mounting the license file - that location /usr/share/ParticularSoftware/license.xml is the machine-wide license location for Linux-based systems.
@jfeinour Ah yes, I’ve run into this SecurityToken problem in other projects. We’ll have to get that added to ServiceControl as it’s not in there yet. But, I fear there is another SQS-specific bug in ServiceControl, unrelated to containers, that may be a blocker for you after that. I don’t see a public bug issue posted for it yet, but it’s been flagged as a critical bug and we already have people working on a fix.
So while you could probably hack together a fix by mounting the SecurityToken assembly as a “volume” in the right place, I’m not sure it’s worth your time as I think the other bug would block you from starting anyway.
@DavidBoike this is more of a minor suggestion, but any thought to enabling the option to output the logs as structured json format? this would make the logs easier to consume in cloud-based environments
@jfeinour JSON structured logs is a great idea. Unfortunately we have other issues with our current logger and need to effect a larger change to the whole logging infrastructure before we’d be able to entertain that.
Something that came up for me and a client - is there a way to have the containers run their setup code and then continue running as normally? Combine the two steps into one? Basically we usually do EnableInstallers in our prod environments so is there way to do similar for the service control error/audit/monitoring instances.
Just a quick one to let you know that we have just released a new version of ServiceControl that fixes the issues with SQS transport and iam roles, see ServiceControl 5.4.1 – Patch release available
Your comment about running setup as part of regular startup is related to:
You might want to leave a comment there or upvote it.
A workaround to always run setup is to add a custom entrypoint bash that will execute both setup and normal operation. Workaround is mentioned in the issue.
ServiceControl cannot setup the RavenDB. Do I need to specify some credentials to connect ro Raven?
Here logs:
2024-07-18 11:20:06.6355|1|Info|LoggingConfiguration|Logging to console with LogLevel ‘Info’
22024-07-18 11:20:06.8267|1|Info|ServiceBus.Management.Infrastructure.Settings.Settings|No settings found for error log queue to import, default name will be used
32024-07-18 11:20:11.0313|1|Info|ServiceControl.Persistence.PersistenceManifestLibrary|Found persistence manifest for RavenDB
6Raven.Client.Exceptions.RavenException: An exception occurred while contacting http:/**/servicecontrol-ravendb.particular.svc.cluster.local:8080/databases/primary/admin/indexes?raft-request-id=d26792be-6c66-4e6b-83b7-366f16cbb8ff.
From that same pod, I can connect to Raven without problem:
$ wget -v http://servicecontrol-ravendb.particular.svc.cluster.local:8080/setup/alive
–2024-07-18 12:59:18-- http://servicecontrol-ravendb.particular.svc.cluster.local:8080/setup/alive
Resolving servicecontrol-ravendb.particular.svc.cluster.local (servicecontrol-ravendb.particular.svc.cluster.local)… 172.28.66.202
Connecting to servicecontrol-ravendb.particular.svc.cluster.local (servicecontrol-ravendb.particular.svc.cluster.local)|172.28.66.202|:8080… connected.
HTTP request sent, awaiting response… 204 No Content
2024-07-18 12:59:19 (0.00 B/s) - ‘alive’ saved [0]