Some failed messages were not handled by the recoverability feature

I appreciate this is not the intended use of this framework and it is internal. I’m using the NServicebus.AcceptanceTesting framework to create an integration test suite. I have a couple of EndpointConfigurationBuilders and I’m using message mutators to record in the scenario context which messages are received and sent. However I have an issue when I add this mutator, the message is received, processed and the scenario context updated, however the message is received multiple times, assume because its failing and the test eventually fails with the following error “Some failed messages were not handled by the recoverability feature”. I’ve struggled to identify why this is happening and this is why I’m reaching out to yourselves.

  [TestMethod]
    public async Task When_RegistrationAccepted_is_recieved()
    {
        var testContext =
        await Scenario.Define<SagaContext>()
            .WithEndpoint<Endpoints.SendingEndpoint>(b => b.CustomConfig(c =>
                {
                    c.OnEndpointSubscribed<SagaContext>(
                        (s, ctx) => ctx.RegistrationAcceptedSubscriptionReceived = true);
                })
                .When(c => c.RegistrationAcceptedSubscriptionReceived,
                    (session, c) => session.Publish(new RegistrationAccepted
                    {
                        Reg_ID = "1900060609789"
                    }))
            )
            .WithEndpoint<RecieverEndpoint>(
                b => b.When(async (session, c) =>
                {
                    await session.Subscribe<RegistrationAccepted>();

                    if (c.HasNativePubSubSupport)
                    {
                        c.RegistrationAcceptedSubscriptionReceived = true;
                    }
                }))
            .Done(c => c.RegistrationAcceptedSubscriptionReceived
                    && c.RegistrationAcceptedRecieved)
            .Run();
    }

    public class SagaInspector : IMutateIncomingMessages, IMutateOutgoingMessages, INeedInitialization
    {
        public SagaContext Context { get; set; }

        public Task MutateOutgoing(MutateOutgoingMessageContext context)
        {
            return Task.FromResult(context);
        }

        public void Customize(EndpointConfiguration configuration)
        {
            configuration.RegisterComponents(c => c.ConfigureComponent<SagaInspector>(DependencyLifecycle.InstancePerCall));
        }

        public Task MutateIncoming(MutateIncomingMessageContext context)
        {
            if (context.Message is RegistrationAccepted)
                Context.RegistrationAcceptedRecieved = true;

            return Task.FromResult(context);
        }
    }

hey @garyamorris

The exception you’re describing is thrown when a message fails to process and isn’t successfully processed a second time and also isn’t moved to the error queue, so technically that would be a “message loss”. There can be many reasons for this (e.g. another endpoint consuming the message, some misbehaving custom behaviors in the pipeline and more…)

It would be helpful if you could share the full test code with us (ideally so that we can run and reproduce the issue you’re seeing). Is this code available on github or somewhere else?

So I now have further insight into the issue, when I define a scenario that sends and receives a message between two endpoints. My mutator, is recording the sending and receiving and the done is marked as successful but the message processing is then triggered.

This results in an intermittent error, sometimes the done is marked successful before the processing is started and the test is green. How could I re-orientate my test to validate that when message A is received its sends message B and have the test marked as complete regardless of what message B processing does?

@garyamorris I’m having a hard time following your description to fully understand the issue. I understand that you don’t want message B to be processed at all? Why does this cause issues and can you replace the handler of message B with a dummy handler not doing anything with the message?

If you explicitly want to test for outgoing messages, it might be a lot easier to test a specific handler using the official testing package: Testing NServiceBus • Testing • Particular Docs

I might be better if I explain what I want to achieve and you may be able to guide me, we have unit testing on our sagas but we what to test the communicative of multiple services end-to-end in terms of integration, so we can identify that the behaviour is as expected. We’re persistently seeing issues of mis-configured queues and behaviours. Currently I’m looking at the following sample and adapting it for our purposes.

@garyamorris at this point I cannot really help you much with guidance without understanding your problem and without seeing the full test code (including endpoint configuration).