[zeromq-dev] ZeroMQ application design questions

[Kapil Thangavelu]

In python you should use multiple processes for true concurrency due to the
GIL, unless your doing processing via forking out or c extensions which
release the gil (gstreamer), however for video processing (cpu) bound you'd
want multi-machine processing if you have any significant volume. i've done
these sort of things with amqp impl (processing guarantees in case of
failure), afaics zeromq is lower level, but one benefit is the transparency
of switching transports from inproc to distributed.

cheers,
kapil

On Fri, Nov 19, 2010 at 12:24 AM, Marc Demierre <demierre.marc at lab.ntt.co.jp
> wrote:

> Thanks for your reply. I'm currently doing some tests and already have a
> simulation that works, but I still have to implement a clean exit
> procedure (that implies passing special messages to the threads to tell
> the to exit)... Anyway, I wanted to keep too may things from the first
> prototype and it is usually a mistake.
>
> As I am not really experienced with concurrent programming, I just
> wanted to know if some "standard" way to do this kind of things (getting
> jobs, doing LRU for workers, receiving updates and results) with ZeroMQ
> existed. I didn't ask for code, just for some advices about theory. I
> described my job in details to show exactly what I didn't understand.
>
> The examples found in The Guide always do an assume that we know how
> many tasks there will be or how many results will be received. I have
> some difficulties to build a complete thing, but I'll try until I
> succeed :).
>
> On Fri, 2010-11-19 at 09:06 +0100, Pieter Hintjens wrote:
> > On Fri, Nov 19, 2010 at 3:44 AM, Marc Demierre
> > <demierre.marc at lab.ntt.co.jp> wrote:
> >
> > > It works but it is not really scalable and I would like to totally
> avoid
> > > threads calling methods of other threads and setting attributes.
> >
> > Don't do that, use inproc to send messages between threads.
> >
> > > Any advice on how to do this with ZeroMQ ?
> >
> > We don't usually do your homework for you here... :-)  Unless it's
> > particularly interesting.  Start with a small mockup, try different
> > patterns until you find what works, then build that out.  There's no
> > substitute for learning this yourself.
> >
> > > A second thing: I wrote most of the programs of The Guide in Python. I
> > > can't contribute them now but I will as soon as my internship is
> > > finished (my boss allowed me to give it under my name after the end of
> > > my contract). Where should I send them and how (pastebin, attached to
> > > email, my Dropbox public folder, ...) ? I will do it in the beginning
> > > January.
> >
> > Make sure you have the latest git master, then send a signed-off patch
> > to this list, following the rules for 0MQ contributions (see
> > http://www.zeromq.org/docs:contributing)<http://www.zeromq.org/docs:contributing%29>
> ..
> >
> > -Pieter
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
>
> --
> Marc Demierre<demierre.marc at lab.ntt.jp>
> Intern,
> Human Interaction Project - NTT Cyber Solutions Laboratories, Yokosuka
> Phone : 080 3395 8049 (national) / +81 80 3395 8049 (international)
> Address : 1-24-5 214 Hayashi, Yokosuka-shi, 238-0315 Kanagawa
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20101119/f1c1a02e/attachment.htm>