To start, I’m using nservicebus 6, non-core, and Azure SB, Azure Storage.
There are normally 2-3 copies of each nservicebus message processor running
My system looks like:
- Users can upload files that contain multiple email messages to their clients. The sending of these emails can be staggered through the working day (send a fixed amount per minute calculated to send the last of the messages at the end of the day), sent at a particular number per minute, or just all sent at once (the easy case).
- So far easy enough. The problem comes in because the users can perform multiple uploads per day, change the way that emails are sent (between staggered through the day, number per minute or all at once), change the fixed number per minute, or delete emails - meaning that the number to be sent each minute varies dynamically.
- There are normally 10’s thousands of emails sent per day per user, but sometimes this goes into the 100’s thousands.
- The exact timing of the messages is not important, as long as they are all sent when staggered through the day and that they are roughly sent at the rate specified (± 5%)
Things that I need to avoid at all costs:
- Any email being sent twice (except maybe one email if the process restarts)
What is the best design for this system?
My current thought is for each user to have a saga
- When a user uploads a file all the messages are stored in a database and a reference (pk?) for each message is sent to the saga for that user
- The saga will have a state variable of the emails to send, and a total number remaining to send
- When the user modifies their office hours, the saga will be notified
- When the user modifies the send frequency type, the saga will be notified
- When the user deletes emails, the saga will be notified with the reference for each one (and will decrement the total)
- The saga will have a one minute timer and the handler for the timer will calculate the number of emails to send, and then batch the emails into 99 email batches, mark them as sent and decrement the total
Things I’m concerned about:
- How do I store > 100,000 references in the saga?
- If it takes longer than 1 minute to calculate how many to send, what will happen when the timer fires (ASB, Azure Storage and transactions)
- Could I store all the information for each email in the saga?
This has been going around my whiteboard for a week now, so I thought I’d ask the far more knowledgable community. Any help is gratefully received!