[zeromq-dev] Question about zmq_recv len and truncate

Bruno D. Rodrigues bruno.rodrigues at litux.org
Sat Oct 12 12:10:53 CEST 2013


This evening I've read some more from the zeromq book and now I understand why the zmq_recv discards the remaining data "by design" and why we should instead use the zmq_msg_* variations.

The question is not receiving the bytes directly into my buffer. The question is that zmq receives the bytes on its own buffer, copies "length" bytes into my buffer, and discards the remaining ones. I now understand this.


I had a simple patch to fix the socket.recvByteBuffer() causing the ByteBuffer.setPosition(x) to exceed its own size, but I'm now verifying that the whole ByteBuffer part of the code is quite broken, as it does not follow the "not receiving(sending) bytes directly from the buffer" - stops working silently if I pass some GC'ed direct buffers, and albeit the documentation mentions the ZeroCopy with direct buffers, the other one doesn't mention anything and should work with non direct buffers, like the byte[] does.

Who is responsible for the jzmq code and how can I try to help with it? Looking at ZMQ JIRA I assume the bindings are not being managed there, so where?





A 12/10/2013, às 01:07, KIU Shueng Chuan <nixchuan at gmail.com> escreveu:

> zmq_recv() doesn't really receive directly into your buffer, it just does the copying from a zmq_msg_t to your buffer.
> It was probably created to mimic the BSD sockets API. 
> 
> You could prevent out-of-memory with socket option of ZMQ_MAXMSGSIZE.
> 
> 
> On Sat, Oct 12, 2013 at 4:45 AM, Bruno D. Rodrigues <bruno.rodrigues at litux.org> wrote:
> Hello all,
> 
> I'm curious about a comment on a certain behaviour of zmq_recv that affected me when using it over the Java bindings.
> 
> The sentence in question is the following: http://api.zeromq.org/3-2:zmq-recv
> "The zmq_recv() function shall return number of bytes in the message if successful. Note that the value can exceed the value of the len parameter in case the message was truncated."
> 
> and on the code, zmq.cpp 493
>     //  At the moment an oversized message is silently truncated.
> 
> The context I was trying was a PUB server where I inject random messages, which I can't control (JSONs with some hundred bytes to several hundred kilobytes), and on the other side a SUB consuming them. This was over the Java bindings, using zmq git master (or 4.0.1) and jzmq 3 git master (or 3.0.1).
> 
> If I use the simpler memcpy heavy versions (send(byte[]) and byte[] x = socket.recv()), everything works as expected, but I was trying the other alternatives that allows to use direct ByteBuffers and avoid the memcpy operations.
> Or explicitly, the recv(byte[] buffer, int len) and the recv(no copy)(ByteBuffer but, int len)
> 
> What happens is that unless I pre-create a buffer of the maximum expected size (like 16MB), messages bigger than my buffer size will not only be truncated, but will also break on java when the callback tries to set the ByteBuffer position to a value larger than the limit.
> jzmq Socket.cpp
> 703     setByteBufferPosition(env, buffer, rc);
> or
> 728         env->CallVoidMethod(buffer, setPositionHandle, pos + rc);
> 
> On top of this, as my C/C++ is quite forgotten since I was forced to move to Java, I couldn't follow exactly where the zmq_msg_t content is filled up, but a grep by malloc scared me as it seems possible to telnet into the server, send the right header prefix, and then announce an incoming message of several GB, which is blindly followed by the malloc, to then be discarded by the truncate reported above. (this can be mitigated by setting a maxMessageSize somewhere depending on the bindings, it seems, but hadn't tested it yet)
> 
> In simpler words:
> - is it possible to receive a message of unknown size using the methods that receive my own buffers? If not, why do those methods exist at all?
> - is it possible to crash a server with out of memory by sending a couple message headers with a huge len value?
> 
> I'd like to help fixing whatever I can do, but I'm lost now on where to pick stuff up, architecturally speaking. could the recv(buffer) return the next part of the same message, as I expected?
> 
> Thanks in advance
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131012/7c641dd5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131012/7c641dd5/attachment.sig>


More information about the zeromq-dev mailing list