[zeromq-dev] Terminating multithreaded 0MQ application, secure subnets

[Martin Sustrik]

Hi all,

I'm forwarding this discussion with kind permission of Peta Sio in hope 
some of you may find is useful.

Martin

 > 2. how to force a blocking receive() to exit when app exits?

 > > A receive call can be blocking or non-blocking.

 > > If i use a blocking receive on a Q in a client app then when the user
closes
 > > the client app, there is no obvious way to tell the thread waiting 
on the
Q
 > > to exit.  At the moment i post a message from the client main user 
thread
to
 > > the broker - broker sends it to all clients - the client gets the 
message
 > > and sees it says "Get_Out" and then the receive() call returns and 
then it
 > > knows to exit.  On exit, i wait for this to happen (a few seconds) 
then i
 > > terminate the thread.
 > > If i use the nonblocking receive() variant then i am polling either by
 > > Sleep() or something similar - this seems wasteful to me especially 
when a
 > > message could be waiting in the Q while the thread Sleeps.
 > >
 > > It would be nice to do a blocking call but allow the client to pass 
in a
 > > waitable synchronization object (like an event) to the receive method.
Now
 > > the receive () method waits for messages and returns either if 
there is a
 > > message in the Q *OR* if the event was signalled.
 > > I am not a Unix person and i know this is probably just Windows 
specific -
 > > but does this make sense?

There's a problem with the solution. 0MQ is designed in the way as to
make individual threads completely independent and thus movable from one
process to another or even from one box to another. This is often needed
as you scale you application up to meet increased performance needs.

Communicating between threads using event or any other non-remotable
communication mechanism breaks the concept. It strongly couples the
threads - you won't be able to get them to separate boxes without
modifying the business logic of the application.

The correct solution would be to send a message from the main thread to
the other thread. The receive will unblock and return the message. The
thread can check the message and seeing it is "stop" message it can exit
the thread.

Thus the main thread does something like this:

int stop_exchange = api->create_exchange ("STOP_EXCHANGE", scope_process);
... launch the threads ...
... do the work ...
message_t msg; // empty message
api->send (stop_exchange, msg);
... wait for threads to stop ...

In the worker thread:

int stop_queue = api->create_queue ("STOP_QUEUE", scope_local);
api->bind ("STOP_EXCHANGE", "STOP_QUEUE");
while (true) {
      message_t msg;
      int queue_id = api->receive (&msg);
      if (queue_id == stop_queue)
          break;
      ... process the message ...
}
... terminate the thread ...

 > * if u have different subnets on a network - what is best way to allow
 > > one-way transfer of messages to a secure subnet
 > > On our network we have various subnets. I assume for a message to be
 > > published and received by all requires a routing policy change that 
simply
 > > allows the port used by ZMQ.  What if one of the subnets is a secure
subnet
 > > where information can flow to it but not out of it.
 > > So imagine users on this secure subnet running the UI app - they 
wont be
 > > able to submit jobs but they want to receive messages that tell 
them how
 > > jobs are progressing, etc.  Is this possible?

AFAICS what you are talking about is that connections can be created
from secure subnet to unsecure subnet but not the other way round. Once
the connection is established data are passed in both directions. If
that wasn't the case event TCP wouldn't work.

To allow this kind of behaviour the services on the unsecure subnet
should expose global object (exchange or queue) and the UI app on the
secure subnet should bind it's local object to it.

Still, opening large number of ports on the secure subnet may be an 
administrative problem. Currently, there's no way to avoid it, however, 
feature allowing to merge several global objects (exhcnage, queues) to 
use the same port is on 0MQ roadmap.