Hi there,
we recently ran into an issue where we attempted to process a large order in our system (20k messages). The way this works is a parser handler reads information from a file, where each row translates into a message that gets processed (a file with 20k rows results in 20k messages). Given that we have outbox enabled the handler attempts to serialize all 20k messages to the outbox table. The payload of each message is not large per se (we don’t include any files) but the accumulated volume of all messages exceeds the maximum of 72MB, which is why we get the following exception:
insert into Nsb_OrderProcessing_OutboxData
(
MessageId,
Operations,
PersistenceVersion
)
values
(
@MessageId,
@Operations,
@PersistenceVersion
)
—> MySqlConnector.MySqlException (0x80004005): Error submitting 72MB packet; ensure ‘max_allowed_packet’ is greater than 72MB.
—> System.IO.IOException: Unable to write data to the transport connection: Broken pipe.
—> System.Net.Sockets.SocketException (32): Broken pipe
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory1 buffer, CancellationToken cancellationToken)
at MySqlConnector.Protocol.Serialization.StreamByteHandler.g__DoWriteBytesAsync|7_0(ReadOnlyMemory`1 data) in /_/src/MySqlConnector/Protocol/Serialization/StreamByteHandler.cs:line 105
One solution would be to introduce batching. For example process 1k messages, and then have the handler send a message to itself to process the next 1k messages until we reach the end.
I was wondering if there are any other solutions out there, maybe something configurable within outbox that would allow us to solve this globally and not on a per use case basis.
Thanks,
Serge