[zeromq-dev] max PUB/SUB topics

Wed May 15 11:58:01 CEST 2019

If I have another process that continually cycles through 0-1024 and anything greater than 1000 doesn’t come through… the issue is not a simply missing the original messages, it is that the subscriptions don’t seem to go through…

From: James Harvey <jamesdillonharvey at gmail.com>
Sent: Wednesday, May 15, 2019 12:54 PM
To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Cc: Bekritsky, Benjamin <Benjamin.Bekritsky at zebra.com>
Subject: Re: [zeromq-dev] max PUB/SUB topics

[External Email]
And regarding the c example where you see the issue if you start publishing before subscribing.
" However, if instead of alternating pub/sub, I do the pubs first and then the subs, I occasionally hit the 1000 limit. "

That is never going to be reliable as the publisher will throw away the data if there are no current subscriptions

Search up "late joiner" for possible solutions if that is a feature you need to support.

On Wed, May 15, 2019 at 10:39 AM James Harvey <jamesdillonharvey at gmail.com<mailto:jamesdillonharvey at gmail.com>> wrote:
Hi Benjy,

From your original example you are mixing binary for subscription subjects and strings for publish subjects. Use the same for both.

The issue is not with the number of subscriptions, its the fact the subjects you send do not match the subjects you subscribed to.

You std::string insert() is expecting a signed char, not unsigned

incoming_string.insert(0,1,i%256);

after 127 its going to turn negative and the std string is going to fail to print a unicode character, spitting out unprintable data

James

On Wed, May 15, 2019 at 9:53 AM Bekritsky, Benjamin via zeromq-dev <zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>> wrote:
Justin,

I rewrote your code in c and get the same results as you showed for Python. I’m on Ubuntu 16.04 and using the “latest_release” branch (4.3.1).

https://gist.github.com/bekritsk/c73af9c3080fb976328b053e0c454166<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_bekritsk_c73af9c3080fb976328b053e0c454166&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=EeNomsL8Of6KU6kxFDGILaAiG5URWw2vnonkRJA-39c&s=ZRtffhCaa7lqlH3Tc1MHsRtDiu0ISOVKdmY_wnrxQq4&e=>

However, if instead of alternating pub/sub, I do the pubs first and then the subs, I occasionally hit the 1000 limit.

I tried doing the same thing in Python, but Python seems to work whether the pub/subs are alternating or all the pubs are first followed by all of the subs.

Thanks,
Benjy

From: zeromq-dev <zeromq-dev-bounces at lists.zeromq.org<mailto:zeromq-dev-bounces at lists.zeromq.org>> On Behalf Of Justin Karneges
Sent: Tuesday, May 14, 2019 6:01 PM
To: Jamie Stewart via zeromq-dev <zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>>
Subject: Re: [zeromq-dev] max PUB/SUB topics

[External Email]
Works for me with 4.2.5 on Ubuntu 18.04.

https://gist.github.com/jkarneges/0b8f6e6d70c23807de0ad02c8ed72fcb<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_jkarneges_0b8f6e6d70c23807de0ad02c8ed72fcb&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=if5ZcgfMqCPKYZ553AP14LU19id8HIg6pMiaE2ez4Yw&s=9owZiXJwjq6awqaVthdwfEAbzrKv2cMkcA2YEHTGbZA&e=>

On Tue, May 14, 2019, at 12:59 AM, Bekritsky, Benjamin via zeromq-dev wrote:

Justin,

Thanks for the guidance, but I set the HWMs to 0 and neither application crashed. I also played with changing the HWMs to 2048 and am still encountering the same issue.

I then subscribed to “all” and received the messages from all the topics (0-1024);

It seems as though it is limiting the number of subscriptions to 1000. If I subscribed from 20 -1024, I get topics 20 – 1019.

I am using lib 4.2.3 and the publisher and subscriber on the same Ubuntu box.

Thanks,

Benjy
From: zeromq-dev <zeromq-dev-bounces at lists.zeromq.org<mailto:zeromq-dev-bounces at lists.zeromq.org>> On Behalf Of Justin Karneges
Sent: Tuesday, May 14, 2019 7:10 AM
To: Jamie Stewart via zeromq-dev <zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>>
Subject: Re: [zeromq-dev] max PUB/SUB topics

[External Email]

Hi,

Subscribing sends a message to the publisher. I believe you're hitting the default high water mark, and subscription messages are getting dropped.

If I'm expecting lots of subscriptions, my rule of thumb is to set SNDHWM=0 on the SUB socket and RCVHWM=0 on the PUB socket. This way the application will sooner die than lose a subscription message.

Justin

On Mon, May 13, 2019, at 10:06 AM, Bekritsky, Benjamin via zeromq-dev wrote:

I am using libzmq and setting up topics using binary data. In other words, my topic is the first 2 bytes of my message.

I subscribe by setting a and b to the values I’m interested in:

        for (uint16_t i = a; i <= b; i++)

        {

                d = zmq_setsockopt(sub_socket, ZMQ_SUBSCRIBE, &i, 2);

                if (d)

                {

                        printf ("E: subscription failed: %s\n", strerror (errno));

                        return -1;

                }

        }

My publisher publishes like this:

                string incoming_string = " this message";

                incoming_string.insert(0,1,i>>8);

                incoming_string.insert(0,1,i%256);

                int     rc = zmq_send(pub_socket, incoming_string.c_str(), incoming_string.size(), 0);

I can’t seem to subscribe to more than 1000 topics (values 0x0000 – 0x03e7). From what I understand, thousands of topics should be available.

Any guidance would be helpful.

Thanks,

Benjy

Benjamin Bekritsky

c/o Zebra Technologies

2 Negev St. (Motorola Building)

4th Floor

Airport City 7019900

Israel

________________________________

- CONFIDENTIAL-

This email and any files transmitted with it are confidential, and may also be legally privileged. If you are not the intended recipient, you may not review, use, copy, or distribute this message. If you receive this email in error, please notify the sender immediately by reply email and then delete this email.

________________________________

- CONFIDENTIAL-

This email and any files transmitted with it are confidential, and may also be legally privileged. If you are not the intended recipient, you may not review, use, copy, or distribute this message. If you receive this email in error, please notify the sender immediately by reply email and then delete this email.

_______________________________________________

zeromq-dev mailing list

zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>

https://lists.zeromq.org/mailman/listinfo/zeromq-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=if5ZcgfMqCPKYZ553AP14LU19id8HIg6pMiaE2ez4Yw&s=MFBykfTCfPngIL34eSeJ2V9NMDX85SYfQxo28ITv7RM&e=>

_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
https://lists.zeromq.org/mailman/listinfo/zeromq-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=EeNomsL8Of6KU6kxFDGILaAiG5URWw2vnonkRJA-39c&s=ml270hTRrev__Pw8FDFclDZaU-1DVTGAcU4KzR5bA_c&e=>

_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
https://lists.zeromq.org/mailman/listinfo/zeromq-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=EeNomsL8Of6KU6kxFDGILaAiG5URWw2vnonkRJA-39c&s=ml270hTRrev__Pw8FDFclDZaU-1DVTGAcU4KzR5bA_c&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20190515/17a41750/attachment.htm>