Wondering if there is an option to customize my original message and send to the error queue when there are exceptions.
ex: I receive an event with 100 items in it. I processed 80 items and 81 got failed. Since automatic retries are enabled, the message goes to the error queue and comes for re-processing. Problem is i try to reprocess the first 80 messages again. Is there a way to customize the message to put only failed items in the event for reprocessing.
Why not sending 100 commands to process 1 single item to yourself in response to the event. Then each of the commands can be retried without affecting the other ones. If needed, the overall progress can be tracked by a saga that is started by the event.
1 Like
To add to Marc his good suggestion, splitting in individual messages also allows for concurrent processing where each individual item that is being processed will just be a small transaction if it would require to access a database.
Marc S,
Exactly I have the same thoughts as you. In fact, I started there, I am trying to look out for options so i don’t have to do code changes. My biggest concern with this approach is, Error handling.
Now with this approach, the pseudocode will be
if(event.vehicles.count > 1)
SplitAndRepostIndiviudalMessageToSourceQueue()
else
continue processing.
await Task.Completed.
SplitAndRepostIndividualmessageToSourceQueue()
{
SendLocal(IndividualVehicle)
}
Now if SendLocal fails at 80th message, how to handle retry just for that vehicle while not stopping the loop to continue till end, I might end up in the same situation. This is where i was like to how to test these scenarios and does NService bus offer any thing out of these box for these scenarios.
To add More: Transport is Rabbit MQ. publisher publishes events to RMQ. we subscribe to events and acknowledge them on success and retry 3 times on failures.
SendLocal only sends a message to RabbitMQ. The only reason this could fail is if RabbitMQ is out.
But if you really need to have all messages sent in 1 single atomic transaction, I think you need the Outbox feature.
This would not send the messages directly to RabbitMQ, but write the messages to a table in your business database using a transaction that is wrapped around your message handler. No need to retry then: either the handler successfully completes and the transaction is committed, or it fails and all database actions are then rolled-back.
After your message handler is finished and the transaction is committed, the Outbox feature will work behind the screens to retrieve the message from the table and actually send them to RabbitMQ. Each of those messages is fetched and sent in a single transaction that will be retried when it fails.