[zeromq-dev] Fair client queue

[Andrew Hume]

without sounding like a broken record, you are conflating 0mq routing with job scheduling.
0mq does so much for you it is tempting to think it can do it all, but in this case, it can't.
and a fair bit depends on the relative cost of doing a task.

the simplest way to get pretty much what you want (and close to your architecture) is

1) task executers do a REQ-REP to ask for the next task from teh scheduler
	(this is the best self-adaptive way to load balance); the executers can be anonymous.

2) to fair share on the input side, just put a submitter id in the task and have the scheduler round robin
	on the id's.

	andrew

On Feb 2, 2012, at 8:52 AM, Jesper Petersén wrote:

> Hi,
> I'm trying to create a parallel task system. Tasks are initiated by and owned by a user. The divide and conquer example from the guide doesn't quite do it because if we have one user pushing loads of tasks, anyone pushing a task after him would have to wait until all his jobs are finished.
> 
> The goal is that tasks should be pulled fairly per user/client. I've created a diagram of what I think is a solution to this, but it doesn't feel optimal. I have roughly this diagram in code as well and it's working OK. Check the link below! 
> 
> https://docs.google.com/drawings/pub?id=1rQ0_CAyNs0nNYa6w5Hub5b73Afkv-mCDhl7JA2mdvYA&w=960&h=720
> 
> 
> The idea is that users push tasks to my client starter. The client starter then forwards these tasks to client threads (these are created if initially not existing and times out /stops if there's nothing to do for a while..) who then sends them to workers, one at a time by REQ. Actually, the only reason I do REQ here is because I don't want a client thread having more than one task being worked one. I'm not sure this is the best way to achieve fair client tasks..
> 
> Any suggestions/guidelines highly appreciated!
> 
> If you would like to view my diagram in Google docs, use this link: https://docs.google.com/drawings/d/1rQ0_CAyNs0nNYa6w5Hub5b73Afkv-mCDhl7JA2mdvYA/edit
> J
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120202/bca4f163/attachment.htm>