[zeromq-dev] zmq_poll

Serge Aleynikov serge at aleynikov.org
Fri May 21 07:46:38 CEST 2010


Making further progress with the Erlang 0MQ driver...  Have more 0MQ 
design questions.

It would be nice to have 0MQ sockets owned by Erlang light-weight 
processes (LWPs).  What's perceived in 0MQ world as application threads 
would by the OS kernel threads running Erlang VM schedulers.  The 
problem is that there's no control on which kernel thread LWPs would be 
scheduled and therefore we cannot assume that a socket gets created on 
some application thread and further send/receive calls would be done on 
the same application thread.  I am not quite clear why there is such a 
hard restriction in 0MQ to enforce a hard relationship between 0MQ 
sockets and an application thread.  I suppose it's done to avoid locking 
at some level, but could this restriction somehow be optionally lifted?

The alternative design is to implement Erlang driver without taking 
advantage of its great concurrency - make sure that all sockets are 
owned by one thread that starts the driver, and relay all communications 
through a proxy LWP (driver manager) that would serialize all 
send/receive requests on different sockets and forward results to 
different LWPs that would "logically" own the sockets by keeping their 
indirect references.

The 0MQ I/O threads are managed/balanced automatically, without any hard 
restrictions.  So why was it designed to enforce the user to follow a 
particular threading model instead of having 0MQ sockets be "active 
objects" that could be processed in different thread contexts in 
presence of intelligent schedulers?

Serge

On 5/20/2010 1:54 AM, Martin Sustrik wrote:
> Serge,
>
>> I am currently implementing the three functions we discussed in this
>> thread, and consequently figuring out the implementation of the
>> dispatcher.  You mentioned that there's only one file descriptor
>> involved in handling 0MQ sockets.  I must be looking in the wrong place
>> - from what I see a context (a.k.a. dispatcher) has a pool of
>> app_threads and io_threads, and each app_thread has a signaler from
>> which you can get a pollable fd.  Since there can be several app_threads
>> it looks to me that there's more than one potential 0MQ fd.  Or is it
>> that poll() must be called in each app_thread independently and will
>> only return status of sockets belonging to that app_thread?  I suppose
>> if that's the case, the Dhammika's driver for Erlang VM that I am fixing
>> now would always have to have one app_thread.
>
> Yes, correct. Witin the array of signalers in the dispatcher, each fd
> belongs to exactly one thread (whether app or i/o). It is used by other
> threads to send notifications to that thread.
>
> Thus, when doing poll in a particular thread the only fd to poll on is
> the one retrieved from the signaler attached to the thread.
>
>> Secondly in order to get the fd through
>> app_thread->get_signaler()->get_fd() you must know app_thread.  The
>> app_thread you can easily get from a socket, but if the only thing you
>> are given is a context, the dispatcher is missing
>> get_current_app_thread() method, so I suppose it has to be added (it'll
>> just iterate through app_threads looking for a match to thread_t::id()).
>> Correct?
>
> What's currently done in zmq_poll is:
>
> a.) If there's no 0MQ socket in the pollset we don't have to use 0MQ's
> fd at all.
>
> b.) If there's at least one 0MQ socket in the pollset make sure all the
> 0MQ sockets belong to the same app thread and poll on the associated fd.
>
> Obviously, this cannot be done for zmq_process as there is no socket
> passed in. Let's then think of a better API... what about zmq_wait_fd
> returning a handle aside of the fd? The handle would then be passed to
> zmq_process instead of context? That way the seatch for current app
> thread could be done once only at the beginning instead of repeating it
> over and over again in each zmq_process.
>
>> It would also be useful if you explained about the design how the
>> transport connections are managed when you have multiple app_threads
>> having different ZMQ_REQ sockets - would it involve different transport
>> connections to the peer ZMQ_REP socket, or they can all be sending
>> messages through a single transport connection (if we are talking, of
>> course, about homogeneous peer connections, e.g. TCP only)?  Is there
>> some architecture document that explains this?
>
> Unfortunately, design documents are missing. What happens is that each
> 0MQ socket (whatever type) has its own set of underlying TCP
> connections. These are never shared between distinct sockets.
>
> Thoughts?
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev



More information about the zeromq-dev mailing list