[zeromq-dev] Inter thread communication for scalability

[Goswin von Brederlow]

On Wed, Jan 15, 2014 at 11:33:16PM +0800, KIU Shueng Chuan wrote:
> During socket shutdown, with linger set to 0, messages in-flight would be
> dropped?
> 
> I use synchronized queues to hold the buffer pointers like the OP but use
> zeromq to send a signal to the consumer thread to pop a buffer from the
> queue.
> On 15 Jan 2014 22:06, "Goswin von Brederlow" <goswin-v-b at web.de> wrote:
> 
> > On Tue, Jan 14, 2014 at 03:40:17PM -0500, Lindley French wrote:
> > > I'm going to caution you about passing pointers through inproc. It may be
> > > possible to do safely, but I haven't yet figured out how to manage
> > > ownership semantics in an environment where messages (pointers) can be
> > > silently dropped.
> >
> > Does a PUSH/PULL inproc socket ever drop messages?
> >
> > MfG
> >         Goswin

Well, don't set linger to 0 if you don't want to loose data.

Shutdown will be a bit complex for this setup. Lets try this.

I would have one controler thread that coordinates shutdown. The
controler thread has a PUB socket and a PULL socket. Every thread
conntects to both of those.

On startup each thread sends a "STARTED" message to the PULL socket so
the controler knows who is running (or you keep track of what threads
you start directly). Each pool sends the number of buffers it creates
to the PULL socket (or you have that as shared knowledge, too).

Each thread in the Thread Sets also sends a "STARTED" and "FINISHED"
message to the PULL socket for every buffer they process. That way the
controler can track the number of buffers on the fly through the
system.

On shutdown first all threads in Thread Set 1 needs to stop producing
more uncompressed buffers. So the controler publishes a "QUIT" message
with key "Thread Set 1". This causes all threads in Thread Set 1 to
finish their current buffer, send a "QUIT" message to the controlers
PULL socket and terminate.

The controler then waits for all buffers with data to be "STARTED" by
Thread Set 2 and then publishes a "QUIT" message with key "Thread Set
2". This signals that no more uncompressed data is available and
compressors will shut down when they finish their current buffer.

The controler then waits for all buffers with compressed data to be
"STARTED" by the writer thread and then publishes a "QUIT" message
with key "Writer". The writer writes the last buffer and terminates
after sending a "QUIT" message.

This leaves only the pools and all the buffers allocated. So the
controler connects to the PUSH sockets of both pools and drains them,
freeing each buffer as it gets them. You know how many there are so
you know when all bufferes are freeed. Then the controler sends a
"QUIT" message with key "Pools" causing the pools to terminate.

Now everything has shut down cleanly and the controler can close its
sockets and terminate the context. There won't be any messages left
floating around either so no need to set linger = 0.

Simple? No. But do you have a better idea?

MfG
	Goswin