Unable to RequestTimeout for more than 15 minutes on SQS Transport

,
1

We are attempting to request a timeout for a number of days in the future on an NServiceBus Saga using the SQS Transport. Every time we try this we get the following exception.

Amazon.SQS.AmazonSQSException: Value 804544 for parameter DelaySeconds is invalid. Reason: DelaySeconds must be >= 0 and <= 900. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.HttpRequest.<GetResponseAsync>d__16.MoveNext() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_bcl\HttpWebRequestFactory.cs:line 230
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpRequest.<GetResponseAsync>d__16.MoveNext() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_bcl\HttpWebRequestFactory.cs:line 247
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.SQS.Internal.ValidationResponseHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\HttpErrorResponseExceptionHandler.cs:line 114
   at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ExceptionHandler.cs:line 38
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 204
   at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 104
--- End of stack trace from previous location where exception was thrown ---

We are not setting the NativeDeferral option, as we have multiple Timeouts that go beyond the native limit of 15 minutes on SQS for delaying delivery of messages. Our understanding is that this should use the Timeout Manager with the configured Persistence (which in this case is RavenDB). It is our understanding that this should be supported on the SQS Transport, however we can’t figure out why it is trying to use native delays for timeouts over 15 minutes.

Thank you,

C. J.

2

Hi @imonthercks,

What version of SQS transport are you using?

Thank you,
Sean

3

Hello @SeanFeldman,

We are using NServiceBus.AmazonSQS version 3.1.0.

Thank you,

C. J.

4

It looks like there are two issues here:

  1. The transport doesn’t support delayed sends for more than 15 mins (opened an issue here).
  2. Delayed operation attempted despite no native deferrals being enabled (opened a bug issue here).

To answer your question - you can’t issue delays longer than 15 mins with SQS at the moment.
I would suggest to track GitHub issues linked above for more details.

Thank you,
Sean

5

In the meantime, are there any suggestions for how to deal with timeouts that need to be scheduled multiple day’s out for purposes of reminders or ageing out sagas? Should be use the built in scheduler to query custom timeouts that we have defined in our own datastore?

Thank you,

C. J.

6

Unfortunately, currently timeouts longer than 15 mins are not supported by the transport and the underlying AWS service. We’ll be adding support for delays exceeding 15 mins with coming releases.

Until then, please follow issue No support for delays longer than 15 minutes · Issue #190 · Particular/NServiceBus.AmazonSQS · GitHub for progress updates.

Sorry for the inconvenience. Please chime on GitHub issue to add details (urgency, is it a complete road-blocker, etc).

Sean

7

Hi @imonthercks,

While this is being addressed on the transport level, would a scheduling saga help you?

Another option could be to use a scheduling technology like Quartz.NET, Hangfire, or FluentScheduler in conjunction with NServiceBus to schedule your timeouts.
We’ve got samples for these if you’d like to have an idea how it works. They could be used as a base starting point.

These are scheduler samples, but they don’t have to be static in their behaviour.
Your code could create a scheduled job from a message handler and the job would be responsible to send a command or raise an event.

Thank you,
Sean

8

Hi,
While not ideal Sagas can still be used in combination with Quartz, Hangfire or FluentScheduler like @SeanFeldman pointed out in the previous post. We are doing a multi phase approach of adopting SQS as a fully supported transport. One of the first steps we took is to take over the community transport in collaboration with Adrian Hoffman, get the documentation up to date, introduce only required changes and then follow along with customers adopting the transport. For example we are about to release ServiceControl Monitoring for SQS. More steps will follow in the near future such as optimizing the native delayed delivery support.

Regards
Daniel

9

@danielmarbach Thanks for the update. We are experimenting with FluentScheduler at the present to see if we are able to satisfy our requirements.

Thank you,

C. J.

10

For all interested, Unrestricted Delayed Delivery feature is now available on NuGet as 4.1.0 beta. For more details, see documentaiton.