[zeromq-dev] How to properly close a socket and HWMRECV

[Antonio Teixeira]

OOpsy Partial E-mail

Hello Martin & List.

In my point of view , ZMQ newbie here , is that with REQ/REP

Broker is REP

Worker is REQ

It would be something like :

while True.
 worker = socket.send('NEED WORK')
 sleep(1)
 continue

and on the boker
message = socket.recv()

if message = "NEED WORK":
 // check if i have work to give
if not work :
 socket.send('NO WORK')
else
socket.send('WORK_PAYLOAD')

The problem with this :
You are constantly  ( i could do sleep(1) but this still applies ) touching
the Broker for more work , this takes impact in used RAM + CPU and of course
reduces Total Broker capacity.

You are constantly exchanging traffic , this impacts the network Level ,
Machine link will be occupied with both "actual work" but also with load  in
order to get Work.
I know this is extremey small but 19 bytes + ZMQ + Network overhead * 32
Cores / second for only one server.

This was reason i picked PUSH/PULL

Sorry to brother you Martin but just trying to squeeze as much power as
possible.

Regards
António Teixeira




2011/7/8 Antonio Teixeira <eagle.antonio at gmail.com>

> Hello List.
>
> Since i have been able to bypass my little ( Big ) Bug i have come for a
> more interesting subject.
>
> My APP uses the pipeline topology , due to all the load-balance  and other
> features gained from this topology but i have found a problem.
>
> My APP requires alot of CPU so i have bypassed that with MultiProcessing +
> IPC and it works great , but i have all my processes based on the
> muliprocesing.cpu_count() value.
>
> (JFYI Messaging processing only takes in my case 0.007 seconds :) With SQL
> Query Included. against 0.12 using rabbitmq :P )
>
> Example
>
> Core 2 DUo > Cpu Count = 2
>
> If you do ps aux you should see 2 workers all goes fine
>
> I send 2 tasks and both cpus get used , great now the problem.
> If i send a third task to the client the socket will accept it and will
> probably "buffer" it for latter use .
>
> But i would like to set HWM on my DOWNSTREAM ( At the Client side not the
> Broker ) socket ,  looking at the source i have noticed 2 values :
> zmq.HWM and zmq.RCVHWM
>
> My objective is the following :
> On a 2 core machine , broker sends one Task
> it will occupy Process 1 and it takes 30 minutes to end this task.
>
> So we have one more working slot , now broker sends another task , the HWM
> should be reached now and ZMQ ignores further Work from Broker so it can
> overflow to other standby servers,
> This new task will occupy Process 2.
>
> Now Process 1 ends  and the Task Worker resets into network_socket.recv()
> The HWM should now allow one more task to flow trough.
>
> Is this the correct way or i'm messing something here ? Should i use HWM or
> RECVHWM. ?
>
> ----
>
> Another question is any nice way to say hello socket please stop accepting
> data , clear any message on the buffer and terminate nicely without ugly
> Exception erros , specially if this socket is working with zmq.Device()
>
> Thank you all again
> Antonio Teixeira
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110711/610d24da/attachment.htm>