[zeromq-dev] Design for a Pub/Sub system

Edwin van den Oetelaar edwin at oetelaar.com
Tue Nov 3 14:57:56 CET 2020


Maybe someone else can shine some light here?

I would be looking at a MQTT over websockets. (install mosquitto as a
server)
Using standard HTTPS/websocket upgrade will allow you to work on any
network, trafic looks like normal https (even behind firewalls (normal
ports open)  and NAT)
I find it works very reliably for what I need. (many devices, pub sub
system, ACL for pieces of the hierarchy, low latency, simple to handle
large payloads)
I am not an expert, that is what I would do.
Best regards,
Edwin


On Tue, 3 Nov 2020 at 13:49, Yanone <post at yanone.de> wrote:

>
>
> > Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar <
> edwin at oetelaar.com>:
> >
> > Define : *very* high number of subscribers
>
> It needs to be able to handle let's say <100K subscribers from the
> beginning without changes. Later, that number could go up to many millions
> when the big corporations jump onboard, which would be cool because then I
> can generate funding for a tool that aims to level the power asymmetry
> between them and the small and independent publishers. I want them to jump
> onboard, so I need to be prepared.
>
> The design needs take this into account from the beginning. What this
> means for me is that I could continue to work on the messaging
> infrastructure without having to update the user-installed apps.
> Sure, I *could* roll out updates, but I'd rather not break functionality
> for those apps that are not getting updated for whatever reason.
>
> So I can postpone the actual scaling, but I feel like I need to make the
> right decisions from the get-go. Any change to the messaging system on the
> client side yields a functionality break.
>
>
> >
> > how about some RSS feed or NNTP ?
>
> I'm not sure how NNTP works exactly, but for sure constant polling is out
> of question I believe (RSS is not a transport protocol). Because the
> updates in my system will be very infrequent but need to trigger a somewhat
> instant GUI app event when they do happen, frequent polling is out of
> question. My whole system is designed rather distributed, so the
> subscription update payload will be polled from the small publisher’s web
> servers directly (after the discussed update notification) and I can't
> burden them or my own central server with a high and growing polling load.
>
> Pub/Sub is perfect as long as the messaging system can handle high numbers
> of subscribers.
>
>
> > Or does it have to be real time in milli-seconds?
>
> No, up to a few seconds lag is fine. Like a messaging app – as long as you
> can use it, it'll work. Messages can arrive a few seconds late and I can
> still have a conversation.
>
> >
> > Why ZeroMQ?
>
> Honestly, I'm really hoping that you guys can answer that question for me.
> Really, that's why I'm here.
>
> I had already implemented Google’s commercial Pub/Sub, which works like a
> charm for my app, but topics and subscribers per topic are both limited to
> 10K each, so I can't even launch with that. AWS’s solution handles more,
> but also limited. After there, you need to roll your own for specs alone,
> plus it'll be cheaper in the long run.
>
> Thanks for your time.
> Jan
>
>
> >
> > Best regards,
> > Edwin #oetelx
> >
> > On Tue, 3 Nov 2020 at 12:51, Yanone <post at yanone.de> wrote:
> > Hello friends,
> >
> > I'm new to the world of messaging systems. I'm developing an open source
> app in the world of graphic design that aims to change the way fonts are
> installed on people’s computers and with that tool help small and
> independent publishers to a better market position against the big evil
> corporations in our industry.
> >
> > I need the GUI app to react to when either the user data or the font
> subscription data has changed. The central server will inform the GUI app
> of these changes and the app needs to react to them somewhat instantly. A
> few seconds lag are no problem, neither are missing messages as I can
> record the update timestamp for all datasets on the central server and have
> the GUI app ask the central server for the timestamps on startup or
> network-reconnect for all datasets that the GUI app instance is concerned
> with.
> >
> > Back to ZeroMQ.
> > So I need to design a pub/sub system that is horizontally scalable if
> necessary. At first I'm sure I can do with a few tens of thousands of
> subscribers, but later that number could grow dramatically. The thing
> doesn't need to scale now, but eventually it may need to. So I'd rather
> make the right decisions now.
> >
> > I read the guide (discarded non-pub/sub topics), and from what I
> understand, I can do with the simple-most pub/sub pattern. I don't need to
> store anything as the central server can keep the update timestamps as
> described. Messages can go lost. Only as long as the client is running and
> connected, I want updates to trigger.
> >
> > I'm expecting extremely little throughput (could be one update per topic
> in many months), but I'm expecting high number of topics and *very* high
> number of subscribers. Theoretically, two topics are sufficient (user
> accounts + subscriptions) when applying filters. This will mean even higher
> number of subscribers per topic. Otherwise, each user account and each
> subscription gets their own topic. In that case, normal number of
> subscribers per topic will be very low, except in rarer cases of free font
> subscriptions, when a single subscription could still get a very high
> number of subscribers.
> >
> > I haven't found *any* information anywhere on ZeroMQ limitations. So I
> need to ask you for clarifications.
> >
> > Here are a precise set of questions:
> > 1. What are the limitations of number of topics and number of
> subscribers, either per topic or in total?
> > 2. Which setup scales better? Only two topics with all subscribers in
> them and filters, or subscribers spread out across a high number of topics?
> In any case, the total number of subscribers stays the same.
> > 3. In case of hard limitations on the above, how do I scale a cluster
> horizontally? How are the subscribers and topics spread across containers
> and brokered between them?
> > 4. What is a good environment in the Google Cloud for that? I was eyeing
> Kubernetes. I have no experience with it as I come from a Google App Engine
> (Python) background, but I can learn it. The real question is: Will it
> scale?
> >
> > Thank you so much for your input! 🙏
> >
> >
> >
> >
> >
> > --
> >
> > Yanone
> > Dipl.-Designer,
> > Master of Design in Type & Media
> > https://yanone.de
> > Twitter/Instagram: @yanone
> > +49 176 24023268 (Germany)
> >
> > Type.World project:
> > https://type.world
> >
> > Crowd funding on:
> > https://www.patreon.com/typeWorld
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> --
>
> Yanone
> Dipl.-Designer,
> Master of Design in Type & Media
> https://yanone.de
> Twitter/Instagram: @yanone
> +49 176 24023268 (Germany)
>
> Type.World project:
> https://type.world
>
> Crowd funding on:
> https://www.patreon.com/typeWorld
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20201103/6eeb256b/attachment.htm>


More information about the zeromq-dev mailing list