[zeromq-dev] zeromq & pylons

[Eric Bell]

First off, thanks to you all for your quick and helpful replies.

Nicholas --> thanks for your one-liner on what Pylons is doing ... that
really helps me understand things better. Several days ago I read your
article "Asynchronous Servers in Python" ... little did I guess that the
subjects would become relevant so quickly! If I recall, gevent was one
of the best performers of the servers you tested.

I am building an application that performs analysis tasks in response to
requests originating from a client browser. As the analysis results
become available, I want them to be displayed in the user interface.

I had planned on using a multiple-publisher multiple-subscriber topology
so that my "Job Controller(s)", which coordinate(s) back-end workers,
could simply publish status and results messages onto a common message
bus. When a new job is requested, a subscriber socket would be created
and placed into a dictionary stored in the app_globals object. The
client browser could then perform AJAX requests ... on the server side,
the request handlers would then perform RECV's on the appropriate
subscriber socket and pass that data back to the client.

I wanted to use the PUBLISH-SUBSCRIBE pattern so that any number of
consumers could process the results coming from one specific job. For
example, I could have a consumer whose job was to generate PDF reports,
or another consumer stored results in a database.

Thread.local won't solve the problem because I'll end up with a separate
copy of the subscriber socket for each thead in the web server thread
pool. Not only is that horrible, but it will cause the same message to
be repeated as different threads are called upon to service a client
request.

It sounds like I need to implement a "boundary layer" between the
multi-threaded web server and the single-threaded 0MQ sockets. This
boundary layer will be in charge of collecting messages on each job and
vending out information to whomever asks for it. This is a totally
viable approach, but requires a shift in my thinking ... originally I
was using 0MQ as both a communications and storage mechanism. If I
create a "boundary layer" then 0MQ is only a communications mechanism. 

Alternatively, I am just a totally confused newbie and have a lot to
learn. :-)

--eric

---------------------------

Hi Eric,

On Aug 3, 2010, at 9:43 PM, Eric Bell wrote:

> I am new to both pylons and zeromq, so I have a lot of murkiness about how
> all this works. I imagine that pylons "handlers" come from some sort of
> thread pool, but honestly I have no real idea what's happening under the
> hood.

The only thing what Pylons does (and most other Python web frameworks) is that it provides a WSGI application to some kind of server. 

How your application is actually served depends on the type of webserver you use. The most common (and also Pylons default) are threaded but you could just as well run it on an evented server such as fe Gevent or Tornado. 

> In my initial design, I create zeromq sockets and store them in the
> app_global object. My thinking was that then any handler "thread" could have
> access to the sockets. Upon reading the quote above I am now concerned that
> I have violated a basic restriction on how to use ZeroMQ sockets.

Pylons' app_global is not only shared between different threads but it is also not thread safe and should really only be used for read operations.

The best way to achieve what you are doing is to store the socket context on thread local storage. You can do that by putting something like this in your module file.

    import threading
    lts = threading.local()
    lts.ctx = zmq.Context()

You can then use lts.ctx as a reference to your ZeroMQ context.

Nicholas Piël