[zeromq-dev] NetMQ: is PUB/SUB filtering done at publisher side?

[Giacomo Tesio]

Thanks Doron.

I'll try this way: since I can't overload the process that recieve data
from the external source, I'll use it with a XSUB/XPUB proxy in the
subscribers process; XPUB will be inproc with SendWatermark to zero so that
the connected subscribers will receive all the messages. I can accept high
memory usage on the subscriber side, but not on the publisher one.

Looks reasonable?


Giacomo


On Thu, Jun 27, 2013 at 11:46 AM, Doron Somech <somdoron at gmail.com> wrote:

> Hi Giacomo,
>
> I'm pretty sure  subscribers cannot block the publisher, I'm not sure the
> breakpoint is a good way to test that.
>
> Anyway regarding your other questions, NetMQ can handle slow subscribers,
> you just need to set the SendWatermark to zero and NetMQ will queue all
> messages, you should however know that this can cause to a memory problem.
>
> I don't think you need any queues at any side, ZeroMQ and NetMQ are
> working with queues, when you send a message or receive a message you
> actually queue and dequeue, there is another thread that actually do the
> network stuff. So I think you should publish the message to NetMQ socket
> directly from the thread that listen to the source and on the subscriber
> side just consume the messages directly from the queue.
>
> Regarding the timeout, you can also do it with NetMQ using a poller or
> Poll method on a Socket.
>
>
>
>
> On Thu, Jun 27, 2013 at 11:45 AM, Giacomo Tesio <giacomo at tesio.it> wrote:
>
>> Hi Doron!
>>
>> Thanks for your help.
>> I have two (windows7) processes that communicate via netmq pub/sub (the
>> publisher runs as x86, the subscriber as x64). While debugging (in visual
>> studio), I've set a breakpoint just after the send() (in the publisher) and
>> just after the Recieve (in the subscriber) and I noted that the publisher
>> send the message but hang while the subscriber handle it.
>>
>> The publisher forwards data from another async source that is very fast
>> and I must ensure that:
>>  1) no message from the source is lost (thus I can't block the publisher
>> process, because the async source requires very low latency and do not
>> provide any cache: if no one is ready to read the message they send, the
>> message is lost)
>>  2) no message from the publisher is lost by the subscribers, even if
>> they have different processing speeds.
>>
>> To ensure 2 I'm using an unbounded BlockingCollection<T> as a queue (that
>> btw, enable me to TryTake messages with a timeout, that is useful to
>> control the consuming thread when the publisher don't send nothing)
>>
>> In case of slow subscribers, does NetMQ alone (without the unbounded
>> BlockingCollection) ensure that I won't neither loose any message nor block
>> the publisher or other subscribers?
>>
>>
>> Giacomo
>>
>>
>>
>>
>>
>>
>> On Thu, Jun 27, 2013 at 9:53 AM, Doron Somech <somdoron at gmail.com> wrote:
>>
>>> Hi Giacomo,
>>>
>>> I'm happy to hear you choose netmq, answers to your questions:
>>>
>>> 1. There are no functionality differences between NetMQ and
>>> CLRZMQ/ZeroMQ, there are some small API differences between NetMQ and
>>> CLRZMQ.
>>> 2. Same as ZeroMQ, so publisher side (it's based on ZeroMQ 3.2.2)
>>> 3. What to do you mean blocked? how did you noted that? I don't think
>>> you should queue because NetMQ / ZeroMQ is already doing exactly that. Just
>>> consume the messages from the socket.
>>>
>>> Doron
>>>
>>>
>>> On Thu, Jun 27, 2013 at 1:04 AM, Giacomo Tesio <giacomo at tesio.it> wrote:
>>>
>>>> Hello everybody, I'm working to my first ZeroMQ based real-world
>>>> application and I have a few questions:
>>>>
>>>> 1) I'm working on .NET and I choosed NetMQ instead of CLRZMQ just on
>>>> the base of github activity. I've noted there are some differences between
>>>> the two API (for example I had to Subscribe("") instead of SubscribeAll())
>>>> but they seem minor differences at a first look. Is there any difference in
>>>> functionality that I should be aware of?
>>>> 2) I have to use a topic based PUB tcp socket: is PUB/SUB filtering
>>>> done at publisher side?
>>>> 3) I noted (in debug) that the publisher blocks while the subscribers
>>>> handle each message: I've tried to minimize the impact of this with a queue
>>>> of messages of subscriber side, with the subscriber that fills the queue
>>>> and another thread consuming it. Is there a better approach?
>>>>
>>>> Thanks a lot!
>>>>
>>>>
>>>> Giacomo
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130627/eb95bf2f/attachment.htm>