[zeromq-dev] Design for a Pub/Sub system

Tue Nov 3 15:20:29 CET 2020

Of course ZMQ can provide you with what you need but be aware that ZMQ 
provides you with the building blocks to build such infrastructure. It 
won't provide you with it out of the box. You also do not tell what 
infrastructure you are planning to use. I would guess you are going to 
use plain internet and thus are challenged by all the firewalls and 
crappy connections everywhere. Sure you can also go more highlevel using 
websockets or even MQTT but that would just require more of your 
backend. You really have to be able to distribure your load and ZMQ can 
provide you with building blocks for it.

Btw ZMQ is also providing websocket transport, just only as a draft API 
currently.

Rg,

Arnaud

On 03-11-2020 13:48, Yanone wrote:
> 
> 
>> Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar <edwin at oetelaar.com>:
>>
>> Define : *very* high number of subscribers
> 
> It needs to be able to handle let's say <100K subscribers from the beginning without changes. Later, that number could go up to many millions when the big corporations jump onboard, which would be cool because then I can generate funding for a tool that aims to level the power asymmetry between them and the small and independent publishers. I want them to jump onboard, so I need to be prepared.
> 
> The design needs take this into account from the beginning. What this means for me is that I could continue to work on the messaging infrastructure without having to update the user-installed apps.
> Sure, I *could* roll out updates, but I'd rather not break functionality for those apps that are not getting updated for whatever reason.
> 
> So I can postpone the actual scaling, but I feel like I need to make the right decisions from the get-go. Any change to the messaging system on the client side yields a functionality break.
> 
> 
>>
>> how about some RSS feed or NNTP ?
> 
> I'm not sure how NNTP works exactly, but for sure constant polling is out of question I believe (RSS is not a transport protocol). Because the updates in my system will be very infrequent but need to trigger a somewhat instant GUI app event when they do happen, frequent polling is out of question. My whole system is designed rather distributed, so the subscription update payload will be polled from the small publisher’s web servers directly (after the discussed update notification) and I can't burden them or my own central server with a high and growing polling load.
> 
> Pub/Sub is perfect as long as the messaging system can handle high numbers of subscribers.
> 
> 
>> Or does it have to be real time in milli-seconds?
> 
> No, up to a few seconds lag is fine. Like a messaging app – as long as you can use it, it'll work. Messages can arrive a few seconds late and I can still have a conversation.
> 
>>
>> Why ZeroMQ?
> 
> Honestly, I'm really hoping that you guys can answer that question for me. Really, that's why I'm here.
> 
> I had already implemented Google’s commercial Pub/Sub, which works like a charm for my app, but topics and subscribers per topic are both limited to 10K each, so I can't even launch with that. AWS’s solution handles more, but also limited. After there, you need to roll your own for specs alone, plus it'll be cheaper in the long run.
> 
> Thanks for your time.
> Jan
> 
> 
>>
>> Best regards,
>> Edwin #oetelx
>>
>> On Tue, 3 Nov 2020 at 12:51, Yanone <post at yanone.de> wrote:
>> Hello friends,
>>
>> I'm new to the world of messaging systems. I'm developing an open source app in the world of graphic design that aims to change the way fonts are installed on people’s computers and with that tool help small and independent publishers to a better market position against the big evil corporations in our industry.
>>
>> I need the GUI app to react to when either the user data or the font subscription data has changed. The central server will inform the GUI app of these changes and the app needs to react to them somewhat instantly. A few seconds lag are no problem, neither are missing messages as I can record the update timestamp for all datasets on the central server and have the GUI app ask the central server for the timestamps on startup or network-reconnect for all datasets that the GUI app instance is concerned with.
>>
>> Back to ZeroMQ.
>> So I need to design a pub/sub system that is horizontally scalable if necessary. At first I'm sure I can do with a few tens of thousands of subscribers, but later that number could grow dramatically. The thing doesn't need to scale now, but eventually it may need to. So I'd rather make the right decisions now.
>>
>> I read the guide (discarded non-pub/sub topics), and from what I understand, I can do with the simple-most pub/sub pattern. I don't need to store anything as the central server can keep the update timestamps as described. Messages can go lost. Only as long as the client is running and connected, I want updates to trigger.
>>
>> I'm expecting extremely little throughput (could be one update per topic in many months), but I'm expecting high number of topics and *very* high number of subscribers. Theoretically, two topics are sufficient (user accounts + subscriptions) when applying filters. This will mean even higher number of subscribers per topic. Otherwise, each user account and each subscription gets their own topic. In that case, normal number of subscribers per topic will be very low, except in rarer cases of free font subscriptions, when a single subscription could still get a very high number of subscribers.
>>
>> I haven't found *any* information anywhere on ZeroMQ limitations. So I need to ask you for clarifications.
>>
>> Here are a precise set of questions:
>> 1. What are the limitations of number of topics and number of subscribers, either per topic or in total?
>> 2. Which setup scales better? Only two topics with all subscribers in them and filters, or subscribers spread out across a high number of topics? In any case, the total number of subscribers stays the same.
>> 3. In case of hard limitations on the above, how do I scale a cluster horizontally? How are the subscribers and topics spread across containers and brokered between them?
>> 4. What is a good environment in the Google Cloud for that? I was eyeing Kubernetes. I have no experience with it as I come from a Google App Engine (Python) background, but I can learn it. The real question is: Will it scale?
>>
>> Thank you so much for your input! 🙏
>>
>>
>>
>>
>>
>> --
>>
>> Yanone
>> Dipl.-Designer,
>> Master of Design in Type & Media
>> https://yanone.de
>> Twitter/Instagram: @yanone
>> +49 176 24023268 (Germany)
>>
>> Type.World project:
>> https://type.world
>>
>> Crowd funding on:
>> https://www.patreon.com/typeWorld
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> 
> --
> 
> Yanone
> Dipl.-Designer,
> Master of Design in Type & Media
> https://yanone.de
> Twitter/Instagram: @yanone
> +49 176 24023268 (Germany)
> 
> Type.World project:
> https://type.world
> 
> Crowd funding on:
> https://www.patreon.com/typeWorld
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 