[zeromq-dev] ZMQ_ROUTER modification

[Andrew Hume]

whit,

	i believe this is a common mistake, with an easy solution.
the fundamental error is confusing message distribution
with job scheduling. zeromq is partially to blame
because it does a good job at what it does (fair share
and load balancing) and tempts you into thinking it
solves the job scheduling problem as well.

	in general, the best solution is that each worker
asks a job to do when it is ready for that work. typically,
we might use a REQ/REP for this. this works cleanly
if the request overhead is not significant (normally the case).
even when we get near the edge condition of the latency
becoming an issue, i normally solve that by keeping an internal
queue on the worker of 2-3 jobs (so that there is always something to do).
then, the only bad case is when the time to do a job is comparable
to teh time to transmit the job description. in this case, life is hard,
but generally in this case, volume is high, so you can afford to simply
batch jobs into groups (of 100 or somesuch) and then treat those
as a single managed unit.

		andrew


On Dec 14, 2011, at 7:58 AM, Whit Armstrong wrote:

> Well, let me explain what I'm trying to do.  Perhaps someone can show
> me a better way.
> 
> I have a client using a dealer socket. Talking to a mixed server
> environment,  a couple of 6 core machines and a 12 core machine.
> 
> Each of the servers uses a simple queue device to fan out the jobs to
> the workers over ipc:
> 
> So, basically this pattern, but the client connects to many machines
> w/ different numbers of cores.
> 
> client(DEALER)->Queue(ROUTER,DEALER)->worker(REP)
> 
> Because the dealer socket on the client fair queue's the messages to
> all the queue devices equally, so the 12 core machine quickly becomes
> idle after working off its queue while the 6 core machines continue
> work off their queues.
> 
> My thought was that I could set the HWM to 1 on the ROUTER socket
> which would prevent the messages from being read aggressively, but
> since ROUTER will drop on HWM, I can't do that.
> 
> Can anyone suggest a better pattern?
> 
> -Whit
> 
> 
> 
> 
> 
> On Wed, Dec 14, 2011 at 3:35 AM, Martin Sustrik <sustrik at 250bpm.com> wrote:
>> On 12/14/2011 11:49 AM, Whit Armstrong wrote:
>>> 
>>> Is it possible to construct a ZMQ_ROUTER socket that does not drop on HWM?
>> 
>> 
>> Technically it is possible. It can block instead of dropping. The question
>> is whether single peer being dead/slow should really block sending messages
>> to all the other peers.
>> 
>> Martin
>> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20111215/6ec4c5d4/attachment.htm>