[zeromq-dev] Problem with zeromq and large messages

[Bruno D. Rodrigues]

No, it’s not that clear at the beginning, as you state yourself on the book. There’s a reuse of the same words for slightly different things and that is quite confusing until the boom paw moment.

There is at least one thing called message. It contains a length, the data, and a flag stating if more messages will come.
Then there is an aggregation of messages, which are multiple messages with the more flag on, plus the last one with the flag more off, and ensured to be delivered all together.

So the more has nothing to do with receiving a single message in byte chunks (don’t want to use parts here).

Now the problem is that zmq_msg_recv receives a structure without a buffer, and it’s zmq itself that creates a buffer with the message size, and “forces” the usage of msg_close to free the buffer.

zmq_recv on the other hand receives our own buffer, but because we might receive a message bigger than expected, the data will be truncated and dropped inside zmq, albeit the return code will report the right size, so at least we know that we lost data.

The first case is simpler, but may generate too many mallocs and frees. The second case may loose data. For regular usage I’d personally recommend using the first pattern (zmq_msg_recv) and let zmq care about the messages. For advanced high-performance optimizations, like my case, it would be nice to pass our own buffer know the message size.

Don’t worry I’m not complaining about this. I’ll look at it myself and push a proper issue and pull request as soon as I’m back to work ;) 


On Jan 3, 2014, at 19:22, Pieter Hintjens <ph at imatix.com> wrote:

> Please read the man page for zmq_recv_send and zmq_msg_recv. It's
> pretty clearly explained. http://api.zeromq.org/3-2:zmq-msg-send
> 
> On Fri, Jan 3, 2014 at 6:48 PM, Brant Merryman <brantm at phmgmt.com> wrote:
>> I have been using zeromq in my application and I’ve been having some
>> problems with it.
>> 
>> Sometimes I need to send large messages. I have code that works kind of like
>> this:
>> 
>> char eventBuffer[BUFFER_SIZE];
>> 
>> for (int more = 1; more; ) {
>> rc = zmq_recv(socket, eventBuffer, BUFFER_SIZE, 0);
>> if (rc > 0) {
>> <accumulate the data in the event buffer>
>> size_t moreSz = sizeof(int);
>> zmq_getsockopt(socket, ZMQ_RCVMORE, &more, &moreSz);
>> } else {
>> printf(“zmq_recv failed: %d\n”, rc);
>> more = 0;
>> }
>> }
>> 
>> <handle the accumulated data>
>> 
>> 
>> 
>> The process that is sending the data uses code like this:
>> 
>> rc = zmq_send(socket, data, data_length, 0);
>> if (rc <= 0) {
>> printf(“zmq_send failed: %d\n”, rc);
>> }
>> 
>> 
>> The problem is that when the data being sent is less than BUFFER_SIZE – no
>> problem. But when the data sent is larger than BUFFER_SIZE, the zmq_recv
>> call gets the same number of bytes that was sent. What I was expecting was
>> that the number of bytes returned would be BUFFERSIZE and that
>> zmq_getsockopt would result in more being 1 and we would repeat that
>> data_length/BUFFER_SIZE times and then once more for the remaining data and
>> then more would go to 0 and we would exit the loop.
>> 
>> How the heck is this supposed to work? What am I doing wrong?
>> 
>> Thanks.
>> 
>> Brant
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140103/4d4212df/attachment.sig>