[zeromq-dev] Is 0mq suitable for producer/consumer kind of problem?

Ryan Chan ryanchan404 at gmail.com
Sun Jul 11 08:54:28 CEST 2010


I have a customized crawler project which consists of 4 subsystems.

a. Links discovery (5+ EC2 instance)
b. Web crawlers (20+ EC2 instance)
c. Links DB (MongoDB x 1)
d. Contents DB (MongoDB x 1)

1. Web crawlers (b) fetch links in batch from Links DB (c) and do the
crawling, and save the result into the Contents DB (d).
2. Links discovery fetch content in batch from Contents DB (d) and do
the analyze and save in the interested links in Link DB (c)

Current system work fine, but I want to explore a more scalable way to
design the system such that (c) and (d) would become the bottom-neck
in the future.

Seems both (c) and (d) can be replaced by 0mq?

I haven't do any serious testing yet, but any recommendation from
experienced users?


More information about the zeromq-dev mailing list