Help getting AmazonSQS transport native integration sample running with json serialization on NSB 6

1

I downloaded and successfully got the AmazonSQS transport native integration sample (AmazonSQS transport native integration sample • Amazon SQS Transport Samples • Particular Docs)) running with json serialization using the existing package versions. NSB 7 is used by default.

image
image902×938 52.5 KB

(screengrab of existing package versions)

However, I have a hard dependency on NServiceBus 6 in our on-prem sites, so I need to run the native integration with NSB 6, not NSB 7.

When trying to run the native sqs integration with NSB 6, I’m getting this error from the framework when it goes to deserialize the native message:

NServiceBus.Transports.SQS.MessagePump Treating message with SQS Message Id cc38161d-a572-4a74-84c5-fec69ee7a39f as a poison message due to exception Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: e. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at NServiceBus.Transports.SQS.MessagePump.<ProcessMessage>d__6.MoveNext() in C:\BuildAgent\work\a3a7604cdca0ab8\src\NServiceBus.AmazonSQS\MessagePump.cs:line 169. Moving to error queue.

These are the highest package versions I can use based on the NSB 6 constraint:

image
image922×810 53.3 KB

I’m dispatching the message using sqsClient like this:

var someNativeMessage = new SomeNativeMessage { ThisIsTheMessage = "Hello!" };
var messageSerialized = JsonConvert.SerializeObject(someNativeMessage);
var body = Convert.ToBase64String(Encoding.UTF8.GetBytes(messageSerialized));
var sendMessageRequest = new SendMessageRequest
{
    QueueUrl = getQueueUrlResponse.QueueUrl,
    MessageAttributes = messageAttributeValues,
    MessageBody = body
};

await sqsClient.SendMessageAsync(sendMessageRequest).ConfigureAwait(false);

My guess is NSB 6 handles the deserialization of messages different than NSB 7 (and/or the NServiceBus.Newtonsoft.Json library does)? Maybe I need a custom serializer based on my NSB 6 constraints?

Thanks!

UPDATE 6/7: it seems that the message pump code of version 3.3.5 of NServiceBus.AmazonSQS assumes that all messages received by the pump are base64 encoded. You can see the same assumption being made in the latest version of the library here:

The message bodies that are being enqueue’d to SQS from sqsClient calls running in lambdas (aka, not an NSB endpoint) are not being base64 encoded, they’re just straight up json. I just learned this today. The above code that shows SendMessageAsync calls was just me trying to set up the code to test all of this.

Do all NServiceBus transport message pumps make the assumption that the transport’s native message body is base64 encoded?

If so, I won’t be able to use NSB native integration for SQS as the messages I need to consume will not be base64 encoded.

2

Hi Paul,

Do all NServiceBus transport message pumps make the assumption that the transport’s native message body is base64 encoded?

My memory might fail me but I have reasons to believe this was adopted from the original community version to be backward compliant. The original version used base64 encoding as a way to avoid running into the character limitations of SQS.

image
image971×188 25.3 KB

The base64 encoding by default was always something that I found not so good because it also bloats quite a bit the payload size. Valid JSON would also be sufficient to comply with the above character limitations.

I have raised an issue to allow opt-out out from sending and receiving everything in base64 encoding.

Meanwhile, is it possible for you to use another lambda to base64 encode the payloads?

Regards,
Daniel

3

HI @danielmarbach. Thanks you so much for opening that issue!

I’m in conversations now around base64 encoding message payloads before dispatching using the SQS client from lambda processes.

I guess the one drawback I can see to base64 encoding you won’t be able to see the contents of a message payload in the AWS console, but I think it’s a minor blocker.

Also, even if I were using the latest SQS transport lib and NSB 7, I believe this would still be an issue. Just want to make that clear as the title of this post mentions NSB 6.

1 Like