[zeromq-dev] Zero-copy and INPROC transport

[Pieter Hintjens]

It's quite normal to use a different protocol for inproc work, in
certain cases at least. Passing pointers is an effective way to pass a
whole structure. We do this e.g. in Malamute for passing messages
between actors. There is no intention of ever making actors into
remote objects. So a custom inproc protocol is fair.

When you use inproc to simulate discrete components, then you would not do this.


On Sat, Jan 16, 2016 at 10:42 PM, Wenzel Jakob <wenzel at inf.ethz.ch> wrote:
> Dear Charles,
>
> note that that is quite different though. If I have to explicitly send
> pointers via ZeroMQ, then it means that INPROC communication uses a
> specialized protocol that would not work with external hosts.
>
> I just ran some experiments to test the hypothesis myself, and it looks like
> ZeroMQ delivers zero copies through the entire chain of operations. On my
> machine, I get the following output when transferring data at a certain
> memory location through INPROC transport (program attached).
>
> Sending message for data at 0x107726ef0
> Received message which points to data at 0x107726ef0
> free_fn called for data at 0x107726ef0
>
> In other words, the address stays the same — no copies. :)
>
> Best,
> Wenzel
>
> PS: Here is the program I used to test this.
>
>
> #include <zmq.h>
>
> #include <stdio.h>
>
> #include <string.h>
>
> #include <assert.h>
>
>
>
> const char *msg_content = "A message";
>
> void free_fn(void *data, void *hint) {
>
>     printf("free_fn called for data at %p\n", data);
>
> }
>
> int main(void) {
>
>     int rc;
>
>     void *ctx = zmq_ctx_new();
>
>     void *rep_socket = zmq_socket(ctx, ZMQ_REP);
>
>     void *req_socket = zmq_socket(ctx, ZMQ_REQ);
>
>     zmq_msg_t msg_out, msg_in;
>
>
>
>     rc = zmq_bind(rep_socket, "inproc://test"); assert(rc == 0);
>
>     rc = zmq_connect(req_socket, "inproc://test"); assert(rc == 0);
>
>     printf("Sending message for data at %p\n", msg_content);
>
>     rc = zmq_msg_init_data(&msg_out, (void *) msg_content,
> strlen(msg_content), free_fn, NULL); assert(rc == 0);
>
>     rc = zmq_msg_send(&msg_out, req_socket, 0); assert(rc != -1);
>
>     zmq_msg_close(&msg_out);
>
>
>
>     rc = zmq_msg_init(&msg_in); assert (rc == 0);
>
>     rc = zmq_msg_recv(&msg_in, rep_socket, 0); assert(rc != -1);
>
>     void *ptr = zmq_msg_data(&msg_in);
>
>     printf("Received message which points to data at %p\n", ptr);
>
>     zmq_msg_close(&msg_in);
>
>     zmq_close(rep_socket);
>
>     zmq_close(req_socket);
>
>     zmq_ctx_term(ctx);
>
>     return 0;
>
> }
>
>
>
>
> On Jan 16, 2016, at 7:54 PM, Brian T. Carcich <briantcarcich at gmail.com>
> wrote:
>
> if all threads are in the same process, just use zmq to pass the pointer;
> that guarantees ZMQ will not duplicates the content of the message.
>
> of course, then you have to be very disciplined about making sure only one
> thread ever frees the memory, but that should not be too hard.
>
> On Sat, Jan 16, 2016 at 11:25 AM, Wenzel Jakob <wenzel at inf.ethz.ch> wrote:
>>
>> Dear Charles,
>>
>> this page notes that the creation of the message itself is guaranteed not
>> to cause a copy operation.
>>
>> It does not, however, say anything about moving the message from one
>> socket to the other, and about the behavior of zmq_recvmsg.
>>
>> (My question is thus if the zero-copy guarantee applies to the whole chain
>> of operations.)
>>
>> Thank you,
>> Wenzel
>>
>> On Jan 16, 2016, at 5:20 PM, Charles Remes <lists at chuckremes.com> wrote:
>>
>> Look at this man page:
>>
>> http://api.zeromq.org/3-2:zmq-msg-init-data
>>
>> Zeromq does zero copy.
>>
>> However, before you go and write a bunch of complex code, just use the
>> normal facilities first and benchmark it. You may not need to do this extra
>> work.
>>
>>
>> On Jan 16, 2016, at 10:13, Wenzel Jakob <wenzel at inf.ethz.ch> wrote:
>>
>> Hello all,
>>
>> I’m interested in using ZeroMQ to build a multithreaded application which
>> uses INPROC transport to propagate messages between threads.
>>
>> My plan was to send messages using zmq_sendmsg (created using
>> zmq_msg_init_data with a custom deallocator), and to receive messages, I was
>> planning to use zmq_recvmsg.
>>
>> I’m concerned if INPROC will perform unnecessary copies of messages, which
>> can be very large in my application (i.e. hundreds of megabytes). In
>> principle, it should just be possible to do it without copies by just moving
>> zmq_msg_t* from the inproc socket of one thread to that of the other —
>> however, I am not familiar enough with the implementation of ZeroMQ to know
>> if that is indeed what happens. Can anyone here advise?
>>
>> Thank you in advance for your help!
>>
>> Best regards,
>> Wenzel
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>