[zeromq-dev] zproxy and curve

Mon Apr 27 10:25:17 CEST 2015

Hi Pieter,
Sorry for the delay, but I work only part-time on this.

My overall architecture is based on the Asynchronous Client/Server pattern from the Guide.

I have N (configurable) backend worker processes to provide solution scaling. They are processes rather than threads as they use postgres.

Until recently I was using zmq_proxy to handle the distribution of work to these workers and the retrieval of their replies. I found I had to deal with the curve security before using the proxy, but that is OK in my case.

When I moved to zsock, etc; zmq_proxy stopped working (as discussed previously) and I found there was no way of using zproxy with curve.

I was already using a zpoller solution in my client, so I transferred that logic to the server to handle the workers. This also enabled me to clean up my close-down as this code was able to action the STOP message.

As requested I have included the code I use to handle all this. [I have simplified it slightly and not tested it]

-----------------------------------------------------------------------------------------------------------------------------------

{
  char worker_endpoint [128], worker_endpoints [1024];
  int process_instance;

  worker_endpoints [0] = 0;

  // start async worker processes
  for ( process_instance = 0; process_instance < CFG_Backend_Threads; process_instance++ ) {
    pid_t pid = fork();

    if ( pid == 0 ) { 		/* now in the forked process	*/
      server_worker ( process_instance );  
      _exit (0);	
    }
    else
      // build string for worker IPC connections
      sprintf ( worker_endpoint, "ipc:///tmp/async%d.ipc", process_instance );
      strcat ( worker_endpoints, worker_endpoint );
      if ( process_instance < CFG_Backend_Threads - 1 ) strcat ( worker_endpoints, "," );
    }

  // Connect frontend socket that talks to clients
  zsock_t *frontend = zsock_new ( ZMQ_ROUTER );
  assert ( frontend );

  // set up 2-way curve security
  secure_connection ( frontend );

  int rc = zsock_bind (frontend, "tcp://*:%s", CFG_Server_Async_Port);
  assert ( rc != -1 );

  // connect socket for async backend workers
  zsock_t *backend = zsock_new ( ZMQ_DEALER );
  assert ( backend );
  rc = zsock_attach ( backend, worker_endpoints, 1 );
  assert ( rc != -1 );

  // Set up poller to handle 2-way traffic
  zpoller_t *poller = zpoller_new ( frontend, 
                                    backend, 
                                    NULL );
  assert (poller);

  while (1) {

    void *which = zpoller_wait (poller, -1 );

    if ( which == frontend )  {   

      // get incoming client message
      zmsg_t *msg = zmsg_recv ( frontend );
      assert (msg);

      // test for STOP command
      zframe_t *content = zmsg_last (msg);
      assert (content);
      char *message = zframe_data (content);
      if (streq (message, "STOP") ) {

        // send STOP message to each worker, before exiting
        for ( process_instance = 0; process_instance < CFG_Backend_Threads; process_instance++ ) {

          // need to copy message as it is destroyed by send
          zmsg_t *msg_copy = zmsg_dup ( msg );
          rc = zmsg_send ( &msg_copy, backend );
        }

        zmsg_destroy ( &msg );

        // wait for workers to get message before destroying socket
        sleep (1);
        zsock_destroy ( &frontend );
        zsock_destroy ( &backend );      
        return;
      }

      // not STOP, send message to worker process
      rc = zmsg_send ( &msg, backend );
      assert ( rc != -1 );
    }

    else if ( which == backend )  {

      // get reply mesage from backend process and send to client       
      zmsg_t *msg = zmsg_recv ( backend );
      assert (msg);
      rc = zmsg_send ( &msg, frontend );
      assert ( rc != -1 );
    }

    else if ( zpoller_terminated )  break;
  }
}

----------------------------------------------------------------------------------------------------------

Regards,
Alan

-----Original Message-----
From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter Hintjens
Sent: 22 April 2015 14:36
To: ZeroMQ development list
Subject: Re: [zeromq-dev] zproxy and curve

Nice. if you want to publish that piece of code, others may enjoy seeing how you did it.

On Wed, Apr 22, 2015 at 12:43 PM, Alan Ward <alan.ward at btinternet.com> wrote:
> Hi Pieter,
>
> Thanks for the replies.
> In the end I cooked my own proxy using a zpoller loop.
> This also allows me to handle close down more cleanly.
>
> Thanks,
> Alan
>
> -----Original Message-----
> From: zeromq-dev-bounces at lists.zeromq.org 
> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter 
> Hintjens
> Sent: 20 April 2015 12:25
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] zproxy and curve
>
> Maybe I should wait another 5 minutes so you can answer the question 
> :-)
>
> Indeed, there is no support for configuring CURVE security on the zproxy sockets.
>
> Two options. A is to add this, which is a fun exercise for someone who wants to explore how CZMQ actors work. B, use zmq_proxy as you were before, using zsock_resolve () to get the ZMQ socket handle.
>
>
>
>
>
> On Mon, Apr 20, 2015 at 11:54 AM, Alan Ward <alan.ward at btinternet.com> wrote:
>>
>> Hi again,
>>
>>
>>
>> OK. Having delved a bit deeper it looks like I should be using zproxy rather than zmq_proxy.
>>
>>
>>
>> I now have a problem in that I can’t determine how to connect my frontend socket with CURVE security as the socket I wish to secure is within zproxy.
>>
>>
>>
>> Thanks
>>
>> Alan
>>
>>
>>
>>
>>
>>
>>
>> From: zeromq-dev-bounces at lists.zeromq.org
>> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Alan Ward
>> Sent: 20 April 2015 09:12
>> To: zeromq-dev at lists.zeromq.org
>> Subject: [zeromq-dev] zmq_proxy problem
>>
>>
>>
>> Hi,
>>
>>
>>
>> I have a system that has been using zmq_proxy for a while to distribute work across a number of worker processes..
>>
>>
>>
>> It was using zsocket etc.
>>
>>
>>
>> I am in the process of upgrading the system to use zsock, zactor etc and all is going well – except that zmq-proxy is no longer working.
>>
>>
>>
>> Apart from the change to zsock the code has not changed and zmq_proxy is just falling through with no error message.
>>
>>
>>
>> The frontend is a normal tcp socket from an external client; the backed is a set of 3 ipc sockets.
>>
>>
>>
>> Any ideas?
>>
>>
>>
>> Thanks,
>>
>> Alan
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev