[zeromq-dev] Problem with zeromq and large messages

Bruno D. Rodrigues bruno.rodrigues at litux.org
Fri Jan 3 20:10:18 CET 2014


http://api.zeromq.org/4-1:zmq-recv

Any bytes exceeding the length specified by the len argument shall be truncated. 

The zmq_recv() function shall return number of bytes in the message if successful. Note that the value can exceed the value of the len parameter in case the message was truncated. If not successful the function shall return -1 and set errno to one of the values defined below.

Right there on the function documentation.

But I hope you understood that I agreed with you that it would be nice to be able to know the message size before passing the buffer into it.


On Jan 3, 2014, at 18:55, Brant Merryman <brantm at phmgmt.com> wrote:

> Sorry, I did miss that part of the documentation where it says that the return value is the number of bytes in the message which can be greater than the length parameter.
> 
> So, if I call it and the return code is greater than my buffer size – then I need to call it again – and then does it give me the next length bytes? Or have I already just lost those bytes?
> 
> 
> From: "Bruno D. Rodrigues" <bruno.rodrigues at litux.org>
> Reply-To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> Date: Friday, January 3, 2014 at 12:14 PM
> To: "zeromq-dev at lists.zeromq.org" <zeromq-dev at lists.zeromq.org>
> Subject: Re: [zeromq-dev] Problem with zeromq and large messages
> 
> read the man page ;)
> 
> recv returns the size of the message, but if the buffer you give is smaller than the message, zmq won’t write more bytes than the ones available.
> 
> the more flag is unrelated to the buffer size. It just means that multiple messages (parts) can be gathered together as a whole message by setting the more flag to on for all parts except the last one. 
> 
> 
> ALL: what are your thoughts about some api that could return the size of the message before calling the recv? I have the same problem with jeromq where I can’t pass a buffer into recv because I don’t know the message size, so I need to rely on byte[] recv() which will create a byte array for each message. Some kind of rc = zmq_recv_size(socket, flags) which would have no side effects, in the sense it could be called multiple times without taking that message out of the queue. Then we could adjust the buffer size and call the current recv(socket, buffer, flags) 
> 
> 
> 
> On Jan 3, 2014, at 17:48, Brant Merryman <brantm at phmgmt.com> wrote:
> 
>> I have been using zeromq in my application and I’ve been having some problems with it.
>> 
>> Sometimes I need to send large messages. I have code that works kind of like this:
>> 
>> char eventBuffer[BUFFER_SIZE];
>> 
>> for (int more = 1; more; ) {
>> rc = zmq_recv(socket, eventBuffer, BUFFER_SIZE, 0);
>> if (rc > 0) {
>> <accumulate the data in the event buffer>
>> size_t moreSz = sizeof(int);
>> zmq_getsockopt(socket, ZMQ_RCVMORE, &more, &moreSz);
>> } else {
>> printf(“zmq_recv failed: %d\n”, rc);
>> more = 0;
>> }
>> }
>> 
>> <handle the accumulated data>
>> 
>> 
>> The process that is sending the data uses code like this:
>> 
>> rc = zmq_send(socket, data, data_length, 0);
>> if (rc <= 0) {
>> printf(“zmq_send failed: %d\n”, rc);
>> }
>> 
>> The problem is that when the data being sent is less than BUFFER_SIZE – no problem. But when the data sent is larger than BUFFER_SIZE, the zmq_recv call gets the same number of bytes that was sent. What I was expecting was that the number of bytes returned would be BUFFERSIZE and that zmq_getsockopt would result in more being 1 and we would repeat that data_length/BUFFER_SIZE times and then once more for the remaining data and then more would go to 0 and we would exit the loop.
>> 
>> How the heck is this supposed to work? What am I doing wrong?
>> 
>> Thanks.
>> 
>> Brant 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140103/962079a0/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140103/962079a0/attachment.sig>


More information about the zeromq-dev mailing list