ServiceControl stay at 50% CPU and log files get filled with errors

1

ServiceControl v4.5.3

It was fine yesterday and suddenly it use 50% CPU usage. If I restart the service it goes back to 0 for a few minutes before starting again.

When looking at the log files (C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl\Logs) I see about 1MB growth on the log file ever 5sec. If get flooded with line error:

2020-02-27 11:21:05.4683|18|Info|ServiceControl.Infrastructure.RavenDB.Expiration.ErrorMessageCleaner|No expired error documents found
2020-02-27 11:21:05.4683|18|Info|ServiceControl.Infrastructure.RavenDB.Expiration.EventLogItemsCleaner|No expired event log documents found
2020-02-27 11:21:05.4683|18|Error|ServiceControl.Infrastructure.RavenDB.Expiration.ExpiredDocumentsCleanerBundle|Error when trying to find expired documents
Raven.Abstractions.Exceptions.IndexDisabledException: The index has been disabled due to errors
à Raven.Database.Indexing.Index.IndexQueryOperation.d__17.MoveNext()
à Raven.Database.Util.ActiveEnumerable1..ctor(IEnumerable1 enumerable)
à Raven.Database.Actions.QueryActions.DatabaseQueryOperation.Init()
à Raven.Database.Actions.QueryActions.<>c__DisplayClass2_1.b__0(IStorageActionsAccessor accessor)
à Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action1 action, EsentTransactionContext transactionContext) à Raven.Storage.Esent.TransactionalStorage.Batch(Action1 action)
à Raven.Database.Actions.QueryActions.Query(String index, IndexQuery query, CancellationToken externalCancellationToken)
à ServiceControl.Infrastructure.RavenDB.Extensions.Query[TState](DocumentDatabase db, String index, IndexQuery query, CancellationToken externalCancellationToken, Action`2 onItem, TState state)
à ServiceControl.Infrastructure.RavenDB.Expiration.AuditMessageCleaner.Clean(Int32 deletionBatchSize, DocumentDatabase database, DateTime expiryThreshold, CancellationToken token)
à ServiceControl.Infrastructure.RavenDB.Expiration.ExpiredDocumentsCleaner.RunCleanup(Int32 deletionBatchSize, DocumentDatabase database, Settings settings, CancellationToken token)
à ServiceControl.Infrastructure.RavenDB.Expiration.ExpiredDocumentsCleanerBundle.<>c__DisplayClass1_0.b__0(CancellationToken token)
à ServiceControl.Infrastructure.AsyncTimer.<>c__DisplayClass1_0.<b__0>d.MoveNext()

2

I stopped the service, deleted the DB folder and started it up again and the problem stopped.

Still not giving confidence that the database got corrupted for no apparent reason and that ServiceControl started doing an infinite error loop.

3

Hi Dunge,

We would love to help you regain find out what caused the issue to give you hopefully back the confidence. In order to do so would you mind bundling the service control logs, the ravendb logs and the startup diagnostics file into a zip file and then send it via email to support at particular dot net?

Then we can look into it. Unfortunately the stack trace provided here isn’t enough because it only tells us that for some reason one of the indexes was disabled but we don’t know why.

Regards
Daniel

4

Hello,

Unfortunately as I mentioned above I deleted the whole Particular.ServiceControl folder in ProgramData (DB+Logs) this morning while I was trying to repair the problem and of course I also emptied the trash… I realize in hindsight this was stupid and that it would have been necessary for any following analysis.

Thanks for offering analyzing the issue, but sorry if I can’t provide anything.