[zeromq-dev] Design for a Pub/Sub system

Edwin van den Oetelaar edwin at oetelaar.com
Tue Nov 3 13:08:49 CET 2020


Define : *very* high number of subscribers

how about some RSS feed or NNTP ? Or does it have to be real time in
milli-seconds?

Why ZeroMQ?

Best regards,
Edwin #oetelx

On Tue, 3 Nov 2020 at 12:51, Yanone <post at yanone.de> wrote:

> Hello friends,
>
> I'm new to the world of messaging systems. I'm developing an open source
> app in the world of graphic design that aims to change the way fonts are
> installed on people’s computers and with that tool help small and
> independent publishers to a better market position against the big evil
> corporations in our industry.
>
> I need the GUI app to react to when either the user data or the font
> subscription data has changed. The central server will inform the GUI app
> of these changes and the app needs to react to them somewhat instantly. A
> few seconds lag are no problem, neither are missing messages as I can
> record the update timestamp for all datasets on the central server and have
> the GUI app ask the central server for the timestamps on startup or
> network-reconnect for all datasets that the GUI app instance is concerned
> with.
>
> Back to ZeroMQ.
> So I need to design a pub/sub system that is horizontally scalable if
> necessary. At first I'm sure I can do with a few tens of thousands of
> subscribers, but later that number could grow dramatically. The thing
> doesn't need to scale now, but eventually it may need to. So I'd rather
> make the right decisions now.
>
> I read the guide (discarded non-pub/sub topics), and from what I
> understand, I can do with the simple-most pub/sub pattern. I don't need to
> store anything as the central server can keep the update timestamps as
> described. Messages can go lost. Only as long as the client is running and
> connected, I want updates to trigger.
>
> I'm expecting extremely little throughput (could be one update per topic
> in many months), but I'm expecting high number of topics and *very* high
> number of subscribers. Theoretically, two topics are sufficient (user
> accounts + subscriptions) when applying filters. This will mean even higher
> number of subscribers per topic. Otherwise, each user account and each
> subscription gets their own topic. In that case, normal number of
> subscribers per topic will be very low, except in rarer cases of free font
> subscriptions, when a single subscription could still get a very high
> number of subscribers.
>
> I haven't found *any* information anywhere on ZeroMQ limitations. So I
> need to ask you for clarifications.
>
> Here are a precise set of questions:
> 1. What are the limitations of number of topics and number of subscribers,
> either per topic or in total?
> 2. Which setup scales better? Only two topics with all subscribers in them
> and filters, or subscribers spread out across a high number of topics? In
> any case, the total number of subscribers stays the same.
> 3. In case of hard limitations on the above, how do I scale a cluster
> horizontally? How are the subscribers and topics spread across containers
> and brokered between them?
> 4. What is a good environment in the Google Cloud for that? I was eyeing
> Kubernetes. I have no experience with it as I come from a Google App Engine
> (Python) background, but I can learn it. The real question is: Will it
> scale?
>
> Thank you so much for your input! 🙏
>
>
>
>
>
> --
>
> Yanone
> Dipl.-Designer,
> Master of Design in Type & Media
> https://yanone.de
> Twitter/Instagram: @yanone
> +49 176 24023268 (Germany)
>
> Type.World project:
> https://type.world
>
> Crowd funding on:
> https://www.patreon.com/typeWorld
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20201103/34906c2c/attachment.htm>


More information about the zeromq-dev mailing list