[zeromq-dev] Zero-copy and INPROC transport

[Wenzel Jakob]

Dear Charles,

note that that is quite different though. If I have to explicitly send pointers via ZeroMQ, then it means that INPROC communication uses a specialized protocol that would not work with external hosts.

I just ran some experiments to test the hypothesis myself, and it looks like ZeroMQ delivers zero copies through the entire chain of operations. On my machine, I get the following output when transferring data at a certain memory location through INPROC transport (program attached).

Sending message for data at 0x107726ef0
Received message which points to data at 0x107726ef0
free_fn called for data at 0x107726ef0

In other words, the address stays the same — no copies. :)

Best,
Wenzel

PS: Here is the program I used to test this.


#include <zmq.h>
#include <stdio.h>
#include <string.h>
#include <assert.h>
    
const char *msg_content = "A message";

void free_fn(void *data, void *hint) {
    printf("free_fn called for data at %p\n", data);
}

int main(void) {
    int rc;

    void *ctx = zmq_ctx_new();
    void *rep_socket = zmq_socket(ctx, ZMQ_REP);
    void *req_socket = zmq_socket(ctx, ZMQ_REQ);
    zmq_msg_t msg_out, msg_in;
    
    rc = zmq_bind(rep_socket, "inproc://test"); assert(rc == 0);
    rc = zmq_connect(req_socket, "inproc://test"); assert(rc == 0);

    printf("Sending message for data at %p\n", msg_content);
    rc = zmq_msg_init_data(&msg_out, (void *) msg_content, strlen(msg_content), free_fn, NULL); assert(rc == 0);
    rc = zmq_msg_send(&msg_out, req_socket, 0); assert(rc != -1);
    zmq_msg_close(&msg_out);

    
    rc = zmq_msg_init(&msg_in); assert (rc == 0);
    rc = zmq_msg_recv(&msg_in, rep_socket, 0); assert(rc != -1);

    void *ptr = zmq_msg_data(&msg_in);
    printf("Received message which points to data at %p\n", ptr);
    zmq_msg_close(&msg_in);

    zmq_close(rep_socket);
    zmq_close(req_socket);
    zmq_ctx_term(ctx);

    return 0;
}



> On Jan 16, 2016, at 7:54 PM, Brian T. Carcich <briantcarcich at gmail.com> wrote:
> 
> if all threads are in the same process, just use zmq to pass the pointer; that guarantees ZMQ will not duplicates the content of the message.
> 
> of course, then you have to be very disciplined about making sure only one thread ever frees the memory, but that should not be too hard.
> 
> On Sat, Jan 16, 2016 at 11:25 AM, Wenzel Jakob <wenzel at inf.ethz.ch <mailto:wenzel at inf.ethz.ch>> wrote:
> Dear Charles,
> 
> this page notes that the creation of the message itself is guaranteed not to cause a copy operation.
> 
> It does not, however, say anything about moving the message from one socket to the other, and about the behavior of zmq_recvmsg.
> 
> (My question is thus if the zero-copy guarantee applies to the whole chain of operations.)
> 
> Thank you,
> Wenzel
> 
>> On Jan 16, 2016, at 5:20 PM, Charles Remes <lists at chuckremes.com <mailto:lists at chuckremes.com>> wrote:
>> 
>> Look at this man page:
>> 
>> http://api.zeromq.org/3-2:zmq-msg-init-data <http://api.zeromq.org/3-2:zmq-msg-init-data>
>> 
>> Zeromq does zero copy.
>> 
>> However, before you go and write a bunch of complex code, just use the normal facilities first and benchmark it. You may not need to do this extra work.
>> 
>> 
>>> On Jan 16, 2016, at 10:13, Wenzel Jakob <wenzel at inf.ethz.ch <mailto:wenzel at inf.ethz.ch>> wrote:
>>> 
>>> Hello all,
>>> 
>>> I’m interested in using ZeroMQ to build a multithreaded application which uses INPROC transport to propagate messages between threads.
>>> 
>>> My plan was to send messages using zmq_sendmsg (created using zmq_msg_init_data with a custom deallocator), and to receive messages, I was planning to use zmq_recvmsg.
>>> 
>>> I’m concerned if INPROC will perform unnecessary copies of messages, which can be very large in my application (i.e. hundreds of megabytes). In principle, it should just be possible to do it without copies by just moving zmq_msg_t* from the inproc socket of one thread to that of the other — however, I am not familiar enough with the implementation of ZeroMQ to know if that is indeed what happens. Can anyone here advise?
>>> 
>>> Thank you in advance for your help!
>>> 
>>> Best regards,
>>> Wenzel
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev <http://lists.zeromq.org/mailman/listinfo/zeromq-dev>
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev <http://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev <http://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20160116/f7e4122c/attachment.htm>