[zeromq-dev] Clean shutdown of malamute client zactor in python binding?

[Arnaud Loonstra]

As long as it is in Python it will be sequential anyway. In a real 
concurrent setup you probably want n:1 topology in between the 
webservers and the malamute instance and pass messages, right?

Rg,

Arnaud

On 2016-02-04 11:37, Aaron Sokoloski wrote:
> Ah, yes, I said what I want to do but not why.  The test script
> doesnt need a thread at all, youre correct.
>
> One of my real use cases is a web server process, where I want to 
> have
> a long-lived client that is shared among requests.
>
> On 4 February 2016 at 03:04, Arnaud Loonstra <arnaud at sphaero.org 
> [12]>
> wrote:
>
>> Im not sure if I can follow you but why do you want to use malamute
>> in
>> separate thread? In python this has no benefits.
>> The approach you have is correct but you dont need a thread for
>> that
>> if Im not mistaken. I would avoid any threading if you can.
>>
>> Rg,
>>
>> Arnaud
>>
>> On 2016-02-03 18:57, Pieter Hintjens wrote:
>> > Yes, its not simple yet. Were slowly building threadsafe sockets,
>> > which will let us deal with actors from any thread.
>> >
>> > On Wed, Feb 3, 2016 at 6:34 PM, Michal Vyskocil
>> > <michal.vyskocil at gmail.com [1]> wrote:
>> >> Hi,
>> >>
>> >> In czmq things are stopped when zsys_interrupted is 1. This
>> apply
>> >> for
>> >> malamute as well. It is modified by default signal handler, so
>> >> simple ctrl-c
>> >> ends your client cleanly, even if in recv call.
>> >>
>> >> I assume that the default signal handler of camp is not setup
>> >> correctly in
>> >> your case. Consult zsys man page for details.
>> >>
>> >> Dne 3. 2. 2016 6:07 PM napsal uživatel "Aaron Sokoloski"
>> >> <asokoloski at gmail.com [2]>:
>> >>>
>> >>> Hi folks,
>> >>>
>> >>> Im trying to make the python malamute binding as easy to use as
>> >>> possible.
>> >>> Im having trouble making sure that everything gets cleaned up
>> >>> correctly on
>> >>> exit, though, when the client is running in a thread other than
>> the
>> >>> main
>> >>> thread, and the client is blocking on a call to recv (aka
>> >>> mlm_client_recv).
>> >>>
>> >>> So far, I have something that works, but isnt ideal, which is
>> to
>> >>> create a
>> >>> "shutdown" inproc socket pair whose only purpose is to notify
>> the
>> >>> recieving
>> >>> end when the process gets interrupted, using a python signal
>> >>> handler.  Then
>> >>> the application code can poll both that receiving socket and
>> the
>> >>> malamute
>> >>> client msgpipe, and exit the receive loop when it gets a
>> message on
>> >>> the
>> >>> shutdown socket.
>> >>>
>> >>> Heres the gist of that approach:
>> >>> https://gist.github.com/asokoloski/02ab5affeca9be2bebdb [3]
>> >>>
>> >>> Although conceptually simple, this is a bit awkward, especially
>> for
>> >>> application code that may want to create multiple clients.  So
>> what
>> >>> Im
>> >>> trying to do is figure out a way to make everything get cleaned
>> up
>> >>> automatically.  Ideally, what would happen is that the zactor
>> >>> thread
>> >>> finishes whatever it was doing and the zactor gets destroyed,
>> then
>> >>> an
>> >>> exception is raised in the python thread.
>> >>>
>> >>> Destroying the zactor from another thread is a big no-no,
>> because
>> >>> zeromq
>> >>> sockets are not thread safe, correct?  So the thread that is
>> doing
>> >>> the recv
>> >>> has to wake up.  We could terminate the context directly,
>> which
>> >>> would make
>> >>> the recv return, but thats quite abrupt, and doesnt give the
>> >>> zactor a
>> >>> chance to shut down.  We could make the recv call poll the
>> socket
>> >>> internally, periodically waking up to check if its been
>> >>> interrupted, but
>> >>> that seems gross, especially as it would have to poll rather
>> >>> quickly -- the
>> >>> shutdown timeout is 200ms.
>> >>>
>> >>> One options is to take the same approach of having a shutdown
>> >>> socket pair,
>> >>> but try to hide it inside the python library code.  Maybe the
>> >>> application
>> >>> code just calls recv(), but inside recv() the object polls the
>> two
>> >>> sockets.
>> >>> But that doesnt really play nicely in the case where, say, the
>> >>> application
>> >>> code wants to poll the malamute client and some other sockets
>> at
>> >>> the same
>> >>> time.  Its doable, but feels complicated.
>> >>>
>> >>> So I have lots of questions now.  Is my reasoning above
>> valid?  Is
>> >>> this
>> >>> sort of thing a problem with pure c programs as well?  Does it
>> even
>> >>> matter
>> >>> if the zactor shuts down cleanly, aside from avoiding ugly
>> >>> warnings?  Is
>> >>> there another approach that I havent even considered?
>> >>>
>> >>> If there isnt a better way to do this, I can bite the bullet
>> and
>> >>> make the
>> >>> application code handle it, but it sure feels a bit lacking.
>> >>>
>> >>> Thanks in advance for any help anyone can offer,
>> >>> Aaron