[zeromq-dev] Design for a Pub/Sub system

Edwin van den Oetelaar edwin at oetelaar.com
Tue Nov 3 15:49:35 CET 2020


On Tue, 3 Nov 2020 at 15:20, Arnaud Loonstra <arnaud at sphaero.org> wrote:

> Of course ZMQ can provide you with what you need but be aware that ZMQ
> provides you with the building blocks to build such infrastructure. It
> won't provide you with it out of the box. You also do not tell what
> infrastructure you are planning to use. I would guess you are going to
> use plain internet and thus are challenged by all the firewalls and
> crappy connections everywhere. Sure you can also go more highlevel using
> websockets or even MQTT but that would just require more of your
> backend. You really have to be able to distribure your load and ZMQ can
> provide you with building blocks for it.
>
> Btw ZMQ is also providing websocket transport, just only as a draft API
> currently.
>
> Rg,
>
> Arnaud
>
>
@Arnoud : thanks for the pointer to websockets transport.
@Yanone : Some caution is in order. Before you start thinking very big,
with millions of paying subscribers you should think smaller.
You want to solve some big problem that you have noticed, you talk about
big evil companies and then about them joining you. (your fundamental
values are clear ?)
In order for your project to have any chance you need to find a business
model.
You need to build, improve and support and support all the code you write
(and may contain some errors) and clients/customers, millions?
Your design (communication model for the networked devices) may scale, but
you do not.
I expect you will be working on creating a proven working model for a year!
Can you hire some smart people to do the implementation, then maybe you can
go to market sooner.
How are your clients getting value from the system, compared to existing
options? Can this project continue if you decide to walk away?
Just some questions, I have been doing software/product development for
some time, the enthusiasm is good, but not always enough.

Good luck, and tell us what you decide,
Edwin #oetelx


> On 03-11-2020 13:48, Yanone wrote:
> >
> >
> >> Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar <
> edwin at oetelaar.com>:
> >>
> >> Define : *very* high number of subscribers
> >
> > It needs to be able to handle let's say <100K subscribers from the
> beginning without changes. Later, that number could go up to many millions
> when the big corporations jump onboard, which would be cool because then I
> can generate funding for a tool that aims to level the power asymmetry
> between them and the small and independent publishers. I want them to jump
> onboard, so I need to be prepared.
> >
> > The design needs take this into account from the beginning. What this
> means for me is that I could continue to work on the messaging
> infrastructure without having to update the user-installed apps.
> > Sure, I *could* roll out updates, but I'd rather not break functionality
> for those apps that are not getting updated for whatever reason.
> >
> > So I can postpone the actual scaling, but I feel like I need to make the
> right decisions from the get-go. Any change to the messaging system on the
> client side yields a functionality break.
> >
> >
> >>
> >> how about some RSS feed or NNTP ?
> >
> > I'm not sure how NNTP works exactly, but for sure constant polling is
> out of question I believe (RSS is not a transport protocol). Because the
> updates in my system will be very infrequent but need to trigger a somewhat
> instant GUI app event when they do happen, frequent polling is out of
> question. My whole system is designed rather distributed, so the
> subscription update payload will be polled from the small publisher’s web
> servers directly (after the discussed update notification) and I can't
> burden them or my own central server with a high and growing polling load.
> >
> > Pub/Sub is perfect as long as the messaging system can handle high
> numbers of subscribers.
> >
> >
> >> Or does it have to be real time in milli-seconds?
> >
> > No, up to a few seconds lag is fine. Like a messaging app – as long as
> you can use it, it'll work. Messages can arrive a few seconds late and I
> can still have a conversation.
> >
> >>
> >> Why ZeroMQ?
> >
> > Honestly, I'm really hoping that you guys can answer that question for
> me. Really, that's why I'm here.
> >
> > I had already implemented Google’s commercial Pub/Sub, which works like
> a charm for my app, but topics and subscribers per topic are both limited
> to 10K each, so I can't even launch with that. AWS’s solution handles more,
> but also limited. After there, you need to roll your own for specs alone,
> plus it'll be cheaper in the long run.
> >
> > Thanks for your time.
> > Jan
> >
> >
> >>
> >> Best regards,
> >> Edwin #oetelx
> >>
> >> On Tue, 3 Nov 2020 at 12:51, Yanone <post at yanone.de> wrote:
> >> Hello friends,
> >>
> >> I'm new to the world of messaging systems. I'm developing an open
> source app in the world of graphic design that aims to change the way fonts
> are installed on people’s computers and with that tool help small and
> independent publishers to a better market position against the big evil
> corporations in our industry.
> >>
> >> I need the GUI app to react to when either the user data or the font
> subscription data has changed. The central server will inform the GUI app
> of these changes and the app needs to react to them somewhat instantly. A
> few seconds lag are no problem, neither are missing messages as I can
> record the update timestamp for all datasets on the central server and have
> the GUI app ask the central server for the timestamps on startup or
> network-reconnect for all datasets that the GUI app instance is concerned
> with.
> >>
> >> Back to ZeroMQ.
> >> So I need to design a pub/sub system that is horizontally scalable if
> necessary. At first I'm sure I can do with a few tens of thousands of
> subscribers, but later that number could grow dramatically. The thing
> doesn't need to scale now, but eventually it may need to. So I'd rather
> make the right decisions now.
> >>
> >> I read the guide (discarded non-pub/sub topics), and from what I
> understand, I can do with the simple-most pub/sub pattern. I don't need to
> store anything as the central server can keep the update timestamps as
> described. Messages can go lost. Only as long as the client is running and
> connected, I want updates to trigger.
> >>
> >> I'm expecting extremely little throughput (could be one update per
> topic in many months), but I'm expecting high number of topics and *very*
> high number of subscribers. Theoretically, two topics are sufficient (user
> accounts + subscriptions) when applying filters. This will mean even higher
> number of subscribers per topic. Otherwise, each user account and each
> subscription gets their own topic. In that case, normal number of
> subscribers per topic will be very low, except in rarer cases of free font
> subscriptions, when a single subscription could still get a very high
> number of subscribers.
> >>
> >> I haven't found *any* information anywhere on ZeroMQ limitations. So I
> need to ask you for clarifications.
> >>
> >> Here are a precise set of questions:
> >> 1. What are the limitations of number of topics and number of
> subscribers, either per topic or in total?
> >> 2. Which setup scales better? Only two topics with all subscribers in
> them and filters, or subscribers spread out across a high number of topics?
> In any case, the total number of subscribers stays the same.
> >> 3. In case of hard limitations on the above, how do I scale a cluster
> horizontally? How are the subscribers and topics spread across containers
> and brokered between them?
> >> 4. What is a good environment in the Google Cloud for that? I was
> eyeing Kubernetes. I have no experience with it as I come from a Google App
> Engine (Python) background, but I can learn it. The real question is: Will
> it scale?
> >>
> >> Thank you so much for your input! 🙏
> >>
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Yanone
> >> Dipl.-Designer,
> >> Master of Design in Type & Media
> >> https://yanone.de
> >> Twitter/Instagram: @yanone
> >> +49 176 24023268 (Germany)
> >>
> >> Type.World project:
> >> https://type.world
> >>
> >> Crowd funding on:
> >> https://www.patreon.com/typeWorld
> >>
> >>
> >> _______________________________________________
> >> zeromq-dev mailing list
> >> zeromq-dev at lists.zeromq.org
> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >> _______________________________________________
> >> zeromq-dev mailing list
> >> zeromq-dev at lists.zeromq.org
> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> >
> > --
> >
> > Yanone
> > Dipl.-Designer,
> > Master of Design in Type & Media
> > https://yanone.de
> > Twitter/Instagram: @yanone
> > +49 176 24023268 (Germany)
> >
> > Type.World project:
> > https://type.world
> >
> > Crowd funding on:
> > https://www.patreon.com/typeWorld
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20201103/f2a9e85d/attachment.htm>


More information about the zeromq-dev mailing list