[zeromq-dev] Clustering xPub xSub Proxies

Justin Schwartz jschwartz at rcsworks.com
Thu May 21 16:03:36 CEST 2020


Thank you so much – this is really helpful. We’ve got a prototype working now thanks to your insights.

The trick was in separating the upstream and downstream proxies as their behavior is slightly different as you highlighted. We had been trying to get this working in a single code stream, but the xpub->connect->xsub is where the magic happens.

Thanks again!

From: zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> on behalf of James Harvey <JHarvey at factset.com>
Reply-To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Date: Thursday, May 21, 2020 at 4:20 AM
To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Subject: Re: [zeromq-dev] Clustering xPub xSub Proxies

Hi

The system relies on quite a lot of cloud concepts, below I will use the azure naming convention


When you say each VM contains 1 upstream and 1 downstream proxy - are these running as two separate processes/applications?

  *   Yes two separate processes, the upstream proxy is slightly different in that its PUB socket needs to actively connect to all the downstream proxies (rather than just bind)
 In your setup, which ZeroMQ sockets are you using for the upstream and downstream proxies, respectively?

  *   XPUB/XSUB
 Similarly, are your external connections coming to the Load Balancers also ZeroMQ sockets? If so, which ones are they?

  *   Yes XPUB/XSUB  and the LB is a network LB (not application/http).


More details (for diagram):


  *   Source (PUB) connects to the well know port on upstream LB
  *   LB knows which VM’s are in its scale set (dynamic) and routes the connection to one of the upstream proxies
  *   The upstream proxies have already connected to all the downstream proxies
     *   You could use dns from the LB (slow) to get IPs or
     *   Write some code to query the scale set IP addresses (this is cloud vendor specific)
  *   A Consumer (SUB) connects to a well known port on the downstream LB
  *   LB knows which VM’s are in its scale set (dynamic) and routes the connection to one of the downstream proxies
  *   At this point the subscription makes it all the way back to the source and the data flows.

If you are concerned about latency the VM’s in the scale set can be placed in a proximity group
If you are more concerned with reliability the VM’s in the scale set can be placed in different availability zones.

Note: the diagram is a simplification, in normal operation all the upstream proxy PUB ports would be connected to all the downstream proxy SUB ports

James
[cid:image001.png at 01D62F4E.B50497E0]

From: zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> On Behalf Of Justin Schwartz
Sent: 20 May 2020 18:02
To: zeromq-dev at lists.zeromq.org
Subject: Re: [zeromq-dev] Clustering xPub xSub Proxies


Hi -
 Thanks, I appreciate you taking the time to respond. I have a few follow-up questions:

When you say each VM contains 1 upstream and 1 downstream proxy - are these running as two separate processes/applications?
 In your setup, which ZeroMQ sockets are you using for the upstream and downstream proxies, respectively?
 Similarly, are your external connections coming to the Load Balancers also ZeroMQ sockets? If so, which ones are they?

The challenge we're facing is with xPub/xSub it's not clear how to relay messages between peers, if that's even possible.

Here's a diagram to illustrate where things break down:

[A screenshot of a cell phone  Description automatically generated]


In this simplified scenario, since the publisher and subscriber are connected to two separate brokers/proxies messages won’t flow unless the proxies are able to relay / forward messages to each other. Ultimately, we’re trying to understand if this is achievable and if so how do we do it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200521/2c3b581d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 80249 bytes
Desc: image001.png
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200521/2c3b581d/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 6674 bytes
Desc: image002.jpg
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200521/2c3b581d/attachment.jpg>


More information about the zeromq-dev mailing list