It seems I found the cause of this error, at least for my setup. Posting here in hope it could help someone else.
The class SSLHandler (from Apache Mina library) contains the following code:
| if (src.remaining() > ((outNetBuffer.capacity() - outNetBuffer.position()) / 2)) { | ||
| // We have to expand outNetBuffer | ||
| // Note: there is no way to know the exact size required, but enrypted data | ||
| // shouln't need to be larger than twice the source data size? | ||
| outNetBuffer = SSLByteBufferPool.expandBuffer(outNetBuffer, src.capacity() * 2); |
It seems that for some messages, the encrypted data ARE longer than twice the source. It actually relates very well to the this "avatar" topic: Your scaled PNG image cannot be compressed (you can try, the file gets larger if you (g)ZIP it) and thus it is a good candidate to occupy much space when encrypted.
Unfortunately, I do not see any other solution that a programmatic one - though it is relatively easy. In my case, I changed the multiplying factor to 4 instead of 2 (and added some fixed number to be safer).
Then it is necessary to recompile at least that one class.
I wonder if this problem would disappear by upgrading to a newer version of Apache Mine...