[zeromq-dev] Greetings & questions about inproc PUB/SUB inside Twisted.

[Charles Remes]

You will save a ton of time if you read the guide (zguide.zeromq.org) and the FAQ. The guide may appear quite long, but for what you intend to accomplish you could probably stop after chapter 5.

I'll give you a head start on a few things.

1. inproc
0mq supports 4 transport types. 

inproc is for communicating between sockets *within the same* process. It essentially does some pointer trickery so it's very fast. It does *not* work between processes.

ipc is for inter-process communication. Most OSes support IPC via local unix domain sockets. 0mq is no different and just relies on the OS mechanism.

tcp is for communication between processes on the same or different boxes. All messages go up and down the entire TCP stack unless the OS has optimizations for "localhost" communication.

epgm is for multicast. It cannot be used for talking to processes on the same box. 

Luckily 0mq lets a single socket bind (and/or connect) to multiple transports at the same time so each limitation isn't a big deal.

2. Subscription forwarding
As you have discovered, the 3.2+ releases of libzmq make the PUB socket intelligent about where to send data. It can save quite a bit of bandwidth if you are on a constrained network.

3. Understanding 0mq
We encourage and invite you to read the source, but the project (and community) prides itself on having some of the best documentation around (on par with commercial products). You should not "need" to read the source to understand the library. Please join us on irc.freenode.net in #zeromq or post your questions here. Be warned that one of the first questions asked is "did you read the guide?" You'll get less help if you answer "no" because most of us old timers are tired of answering novice questions that are clearly answered in the excellent docs (guide, man pages, faq).

Welcome!

cr

On Jan 26, 2013, at 1:56 PM, David J W <zeromq-dev-subscribe at ominian.net> wrote:

> Hello,
>   My name is David W, I am a professional code monkey that was first
> introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 )
> but only now have gotten the chance to sit down and start learning the
> library.  I am on a sabbatical until ideal after PyCon since I got
> laid off and taking the time out to hit a bucket list ( including
> learning 0MQ ).
> 
>  On that note, a few years back as a re-invent the wheel project to
> learn Twisted and COMET I started writing a web MUD engine and
> centered the architecture around two message pipelines:  User action's
> were locked stepped ( User A moved left, tell server, wait for it to
> say yes/no), broadcast to other User's that User A moved left,
> broadcast to all of User A's group they moved left.  NPC's were just
> headless User's driven by a behavior time/tick subprocess that hooked
> upto the same pipelines.    I set that project aside because I
> realized I needed a message queue of some sort and really didn't want
> to setup Rabbit or anything super industrial.
> 
>  Now along arrives 0MQ and since this is a personal project the
> priority is more about understanding how 0MQ works then accomplishing
> the actual project.   In the above example I can imagine using 0mq's
> inproc socket's where client's are SUB types ( subscribe to
> map/domain, subscribe to group chat ) and their is a master process
> that has a router socket for incoming work and a pub socket for
> products [ User A in map 1 moved left] ).
> 
> So here's my questions:
>   For PUB/SUB the impression is that the actual queue sit's on the
> client socket.  PUB pushes a message to all client's [ regardless of
> setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket
> filter's/clears the queue down to what the client is subscribed to.
> Is this correct or is the subscription more intelligent ( PUB keep's a
> subscription roster, see's no one is subscribed, drop's the message OR
> client receives a message, isn't subscribed so it drops the message ).
> 
>  Has anyone had any experience running multiple SUB based client's
> inside of one process and are their any severe consequences.  I
> imagine a SUB socket is going to instantiate the needed structures to
> hold a queue, the actual socket, and other house keeping structures
> but so far small tests (1-10 sockets) hasn't show much memory use.
> 
>   Additionally, if I do get past digging through 0MQ's mechanics, I
> was thinking it would be best to spin off the PUB side to it's only
> process.  Which leads me to wonder if 0MQ inproc PUB/SUB actually
> relies on some clever memory mapping.  eg Push a message on an inproc
> PUB socket which goes to a shared/mutex locked list and client's just
> read from this one list.
> 
>   Apologies if some of these questions seem naive, I haven't gotten
> the chance to read 0mq's C source code yet.
> 
> Thanks,
>   Dave
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev