[zeromq-dev] Devices, and pipeline again

[Jon Dyte]

Brian/ MinRK

It would be really great if you could get minimal examples of these 
device patterns you've revealed in the last few days up on the wiki

that is the  xreq/xrep swapped roles for routing
this heartbeat one and the keyed relay.

These are really clever uses

Thanks

Jon


MinRK wrote:
> Hello,
>
> I work with Brian on the pyzmq devices.
>
> On Thu, Jul 29, 2010 at 10:44, Brian Granger <ellisonbg at gmail.com> wrote:
>
>   
>> On Thu, Jul 29, 2010 at 8:46 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>
>>     
>>> Hi All,
>>>
>>> I'm writing a new generic zmq_device that will replace zmq_queue,
>>> zmq_forwarder and zmq_streamer.
>>>
>>> The code is here: http://dpaste.de/4DcP/
>>>
>>>
>>>       
>> It looks like you are only working on the command line part.  Is this
>> right, so the underlying zmq_device function remains the same?
>>
>>
>>     
>>> The zmq_device works a lot like the zmq::device method.  There is
>>> documentation at the start of the source code, which I'll expand to
>>> become a man page when this is stable.
>>>
>>> Anyone who is already using devices: your feedback would be useful.
>>> Especially, do you use multiple in/out sockets (I've renamed that to
>>> 'frontend' and 'backend' here)?  And do you connect outwards from your
>>> device, using it as a proxy, or do you connect into it, using it as a
>>> mini-broker?
>>>
>>>
>>>       
>> We always create devices by calling the zmq_device function in our code.
>>  We have written a set of functions that will run a device in a thread.  The
>> reason we do this is that it is almost always easier to pass arguments in
>> Python than muck with XML files.  Also, this let's us put multiple devices
>> in a a single process.
>>
>> We use devices in just about any way you can imagine.  We often pass the
>> wrong type of sockets to a device to make it do novel things.  For example,
>> if you pass a single XREQ socket to both arguments of a forwarder device you
>> get a REQ/REP style heartbeat socket.  We have also started to write our own
>> devices that take more than two sockets.  One example is a monitored queue.
>>  It works like a regular queue, but there is a third admin socket that you
>> can use to query the status of the device and the messages that have moved
>> through it.
>>
>>     
>
> Another device we wanted was a queue that worked with two XREP sockets. Of
> course, that doesn't normally work, since xrep messages arrive prepended by
> the requester's identity, so a normal queue will try to send out to the same
> identity that the message came from.  I wrote a copy of zmq::device that
> simply swaps the identities around at the beginning of each message before
> relaying:
>
> the device is insocket: xrep1, outsocket: xrep2 (and behaves symmetrically)
> xreq1 is connected to xrep1 and as identity 'id1', etc.
>
> the sequence of events in this device:
> xreq 1 sends [ 'id2', 'msg' ] to xrep 1
> xrep 1 recvs [ 'id1', 'id2', 'msg' ]
> <swap first two elements>
> xrep 2 sends [ 'id2', 'id1', 'msg' ]
> xreq 2 recvs [ 'id1' 'msg' ]
>
> And there can be many xreq sockets on either side of the device, keyed by
> their identities.
>
> The effect is that all of the XREQ sockets really behave like XREP, in that
> their messages are prepended with an ID specifying the destination, yet they
> each have only one connection (to the device), and each message they receive
> tells them who it came from.
>
> Further, if you pass the same XREP socket as both in and out, you have a
> fully symmetric keyed message relay - anyone can send a message to anyone
> else (as opposed to just between side A and side B and back). For N xreq
> sockets, you still only need to keep track of the address of the device's
> xrep.
>
> -MinRK
>
>
>   
>>
>>     
>>> Second thing, those pipeline sockets again.  Here is the core of the
>>> generic device, acting as broker (such short sweet code, thank you
>>> 0MQ!):
>>>
>>>        int device = parse_device (argv [1]);
>>>        switch (device) {
>>>            case ZMQ_QUEUE:
>>>                frontend_type = ZMQ_XREP;   //  Client
>>>                backend_type  = ZMQ_XREQ;   //  Service
>>>                break;
>>>            case ZMQ_FORWARDER:
>>>                frontend_type = ZMQ_SUB;    //  Subscriber
>>>                backend_type  = ZMQ_PUB;    //  Publisher
>>>                break;
>>>            case ZMQ_STREAMER:
>>>                frontend_type = ZMQ_PULL;   //  Puller
>>>                backend_type  = ZMQ_PUSH;   //  Pusher
>>>                break;
>>>            default:
>>>                return (1);             //  Invalid device
>>>        }
>>>        //  Create sockets, configure them, and start the device
>>>        zmq::context_t context (1);
>>>        zmq::socket_t frontend (context, frontend_type);
>>>        zmq::socket_t backend (context, backend_type);
>>>        if (frontend_type == ZMQ_SUB)
>>>            frontend.setsockopt (ZMQ_SUBSCRIBE, "", 0);
>>>        frontend.bind (argv [2]);
>>>        backend.bind (argv [3]);
>>>        zmq::device (device, frontend, backend);
>>>
>>> As you can see I'm using ZMQ_PUSH and ZMQ_PULL in place of
>>> ZMQ_DOWNSTREAM and ZMQ_UPSTREAM.  It seems to work.  A node pulls
>>> data, works on it, and pushes it on.  So nodes are pullers, or
>>> pushers, or both.  It's short, verbish (like the other socket types)
>>> and feels clear.
>>>
>>>
>>>       
>> I think that hardcoding the socket types in the command line program does
>> make sense, but not in the zmq_device function API.
>>
>>
>>     
>>> If there are no outright objections after a day or so I'll start to
>>> deprecate ZMQ_DOWNSTREAM/UPSTREAM and replace with ZMQ_PUSH/PULL.
>>> We'll keep the old definitions around, of course, this does not break
>>> the ABI.
>>>
>>>
>>>       
>> I think PUSH/PULL are fine.
>>
>> Brian
>>
>>
>>
>>     
>>> -Pieter
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>       
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>     
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>