Is this scenario a reasonable use case for sagas, or am I being too clever by half?
I have a set of files which I download from a vendor (around 80), each of which, or more precisely a set of, may have its own publication time - hence download - schedule. The file data is then staged into SQL Server tables, which are then used to update an online catalog.
I am using NServiceBus for some of the steps, but I am now reconsidering using a saga where
- A scheduler triggers the file download from an ftp site
- an ftp completed message triggers an unzip request
- Unzip completion triggers a downloaded message which
- Triggers a progress log update and
- Triggers moving the zip file to backup
- and triggers a data loader to load the data into the associated staging table
- whose completion creates a data-loaded event which triggers
- The unzipped file is to be copied to the processed folder and triggers
- Update/addition of a record of completion of the data loaded event. (the download, unzip, data load, backups/moves should be recorded with the same id if possible.
- The data loaded event triggers creation of load files for the online catalog. and
- Its progress is logged in a process table.
- Finally, the catalog files are processed by the cataloger process,s which could be its own beast.
I was thinking of “platforming” these steps/processes on a saga. yes/no/maybe? Can a saga easily follow, for lack of a better term, a correlation ID that unifies each of the messages/states? So there could be a dozen of these sagas occurring in parallel, and there should be no confusion among them. They are to be isolated, self-contained operations that are unaware of one another.