[zeromq-dev] Fwd: ZMQ Closure - Change of Beginners Documentation
Josh Wo
josh at aggregateknowledge.com
Thu May 26 20:44:37 CEST 2011
I am agreeing with Mike's comments that the documentation should be simple but shouldn't be trivial.
I am using Java Binding and if following the documentation of the section, my application either get blocked or crashed due to access to a closed socket. I am using zmq in a multithread container (tomcat). Seems to me that java code doesn't need to take care of closure as the finalize will take care of itself? The example and the documentation is too trivial for real world use. I also hope there will be a different section for specific binding if there is a different handling for the binding.
Josh
___
_____________________________________
From: zeromq-dev-bounces at lists.zeromq.org [zeromq-dev-bounces at lists.zeromq.org] On Behalf Of zeromq-dev-request at lists.zeromq.org [zeromq-dev-request at lists.zeromq.org]
Sent: Thursday, May 26, 2011 11:22 AM
To: zeromq-dev at lists.zeromq.org
Subject: zeromq-dev Digest, Vol 41, Issue 104
Send zeromq-dev mailing list submissions to
zeromq-dev at lists.zeromq.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
or, via email, send a message with subject or body 'help' to
zeromq-dev-request at lists.zeromq.org
You can reach the person managing the list at
zeromq-dev-owner at lists.zeromq.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of zeromq-dev digest..."
Today's Topics:
1. Re: Fwd: ZMQ Closure - Change of Beginners Documentation
(Mike Pearce)
2. PUB doesn't discard msg when no SUB (Li, Jia)
3. Re: Fwd: ZMQ Closure - Change of Beginners Documentation
(Pieter Hintjens)
4. FastFlow queues (Pieter Hintjens)
5. Re: PUB doesn't discard msg when no SUB (Ian Barber)
6. Re: FastFlow queues (Martin Sustrik)
----------------------------------------------------------------------
Message: 1
Date: Thu, 26 May 2011 12:06:10 +0200
From: Mike Pearce <mike at kaew.be>
Subject: Re: [zeromq-dev] Fwd: ZMQ Closure - Change of Beginners
Documentation
To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Message-ID: <BANLkTimBpOkw9m3hqYJpiaiNQGjSJTaFdQ at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Pieter,
I completely disagree. The section is inaccurate and there is no more detail
to be found in later chapters. It states that an error in 0MQ exists that
requires that all sockets must be closed in advance of calling zmq_term.
This is incorrect for two reasons -
1. I do not believe that there is an error in ZMQ
2. zmq_term is designed to unblock socket commands in order for the socket
closure. To follow the beginners guide you must issue the zmq_close commands
from outside of the blocked call and the only way to do this is to ignore
the threading limitation of the commands.
A beginner to 0MQ is unlikely to be a beginner to computing and
multithreaded application programming is common place. You cannot write a
GUI without going multithreaded and so I doubt there are many single
threaded programs left. You also cannot avoid polling reads until you
consider a separate read thread making a blocking call.
I do agree that the guide should remain simple but I prefer that it gets
reworded. It is a great guide but this section has become inaccurate over
time.
I would like that this section of the guide have the following added -
1. It states that the socket functions are not thread safe and so anyone
implementing a multithreaded application solution must adhere to this.
2. It states that if the users application is doing a blocking read then
zmq_term is the means by which this blocked read will get unblocked. By
implication an application that implements a blocked read will need to be
multithreaded.
If this advice is not added then you will have missed a great opportunity to
warn new users about this potential pitfall and anyone that implements a
solution based upon the current advice will see a ZMQ that is unstable and
it will appear to them that the product is not good. This is a shame because
the product is good.
Please remember that this one little section in the beginners guide is the
only information offered regard 0MQ termination by this guide and there is
no alternative users guide offered from this part of your website.
Mike
On Thu, May 26, 2011 at 3:15 AM, Pieter Hintjens <ph at imatix.com> wrote:
> On Wed, May 25, 2011 at 8:22 AM, Mike Pearce <mike at kaew.be> wrote:
>
> > I can assure everyone that what I wrote is a lot better than the very
> > inaccurate section that currently exists and I am a bit sad that my
> > submission ended so abruptly.
>
> Well... the section you pointed to is in Ch1 and intended for people
> starting to use 0MQ. We don't discuss multithreaded apps anywhere in
> that section and it'd be inappropriate to start suddenly talking about
> that before people have even understood what 0MQ really is.
>
> There is lots of discussion about multiple threads later, including
> quite explicit (in bold) instructions about how to handle ETERM and
> how not to share sockets between threads:
>
> http://zguide.zeromq.org/page:all#toc30
> http://zguide.zeromq.org/page:all#toc38
>
> I'm happy to add a note earlier on saying, 'warning, if you're doing
> multiple threads', it's more complex than this', but for beginners the
> advice is exactly as I wrote it. Close your sockets, THEN do zmq_term.
> If you don't your app will block.
>
> This is only misleading if you don't read the rest of the text. But
> since good design takes into account how people actually work, not how
> we hope they'd work, I'll fix the text.
>
> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110526/44cc670f/attachment-0001.htm
------------------------------
Message: 2
Date: Thu, 26 May 2011 12:57:29 -0400
From: "Li, Jia" <jia.li at credit-suisse.com>
Subject: [zeromq-dev] PUB doesn't discard msg when no SUB
To: <zeromq-dev at lists.zeromq.org>
Message-ID:
<04CFED3E29349F47BEE980F4D2694C5F052EE19A at EPRI17P32001B.csfb.cs-group.com>
Content-Type: text/plain; charset="us-ascii"
Hi,
I have multiple PUB sockets and single SUB socket. The PUBs perform
connect() and the SUB performs bind().
The problem I see is that if the SUB starts after the PUBs, it still
receive all the messages sent before. This seems to contradict to what
the documentation said.
However, if I have only one PUB and one SUB, and I reverse the
connect/bind by having PUB bind() and SUB connect(), I get the
documented behavior in which SUB doesn't receive historical messages.
Here are the code:
SUB
-------
import zmq
context = zmq.Context()
sub = context.socket(zmq.SUB)
sub.setsockopt(zmq.SUBSCRIBE, "")
sub.bind("tcp://127.0.0.1:27888")
while(1):
msg = sub.recv()
print msg
PUB
-------
import zmq, time
from datetime import datetime
context = zmq.Context()
pub = context.socket(zmq.PUB)
pub.connect("tcp://127.0.0.1:27888")
while(1):
msg = datetime.now().strftime('%H:%M:%S.%f')
print msg
pub.send(msg)
time.sleep(1)
Regards,
Jia
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110526/b1dd5e28/attachment-0001.htm
------------------------------
Message: 3
Date: Thu, 26 May 2011 10:03:06 -0700
From: Pieter Hintjens <ph at imatix.com>
Subject: Re: [zeromq-dev] Fwd: ZMQ Closure - Change of Beginners
Documentation
To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Message-ID: <BANLkTi=Ra7K5aVZhqV2OxE70eVmGDaRy1w at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On Thu, May 26, 2011 at 3:06 AM, Mike Pearce <mike at kaew.be> wrote:
> I would like that this section of the guide have the following added -
> 1. It states that the socket functions are not thread safe and so anyone
> implementing a multithreaded application solution must adhere to this.
This is covered in the section on multithreading. Like I said, I'm
more than happy to repeat that earlier on, since people do seem to
keep making the assumption that threads can share sockets.
> 2. It states that if the users application is doing a blocking read then
> zmq_term is the means by which this blocked read will get unblocked. By
> implication an application that implements a blocked read will need to be
> multithreaded.
This isn't accurate, though. A classic way of exiting a blocked read
is to receive a message, or to be interrupted by a signal like SIGINT.
-Pieter
------------------------------
Message: 4
Date: Thu, 26 May 2011 11:05:58 -0700
From: Pieter Hintjens <ph at imatix.com>
Subject: [zeromq-dev] FastFlow queues
To: zeromq-dev at lists.zeromq.org
Message-ID: <BANLkTikBbnUuKpt8N=m2OyaRGxDoBN-SJg at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi Martin,
Someone pointed me to this:
http://www.1024cores.net/home/lock-free-algorithms/queues/fastflow
and asked how 0MQ's queues compare.
Any comments?
-Pieter
------------------------------
Message: 5
Date: Thu, 26 May 2011 19:13:11 +0100
From: Ian Barber <ian.barber at gmail.com>
Subject: Re: [zeromq-dev] PUB doesn't discard msg when no SUB
To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Message-ID: <BANLkTimEnHUEzzsV=G09bS4iv6n4iPZfoA at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
On Thu, May 26, 2011 at 5:57 PM, Li, Jia <jia.li at credit-suisse.com> wrote
>
> The problem I see is that if the SUB starts after the PUBs, it still
> receive all the messages sent before. This seems to contradict to what the
> documentation said.
>
> However, if I have only one PUB and one SUB, and I reverse the connect/bind
> by having PUB bind() and SUB connect(), I get the documented behavior in
> which SUB doesn't receive historical messages.
>
It's just as artifact of how the connection handling works - if you connect
the PUB it doesn't discard the messages as it is automatically trying to
establish the connection - the PUB that is binding knows it has no
connections so drops the messages.
Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110526/dec484e3/attachment-0001.htm
------------------------------
Message: 6
Date: Thu, 26 May 2011 20:22:56 +0200
From: Martin Sustrik <sustrik at 250bpm.com>
Subject: Re: [zeromq-dev] FastFlow queues
To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
Message-ID: <4DDE9A80.5050809 at 250bpm.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Pieter,
> Someone pointed me to this:
> http://www.1024cores.net/home/lock-free-algorithms/queues/fastflow
>
> and asked how 0MQ's queues compare.
It's very similar to what 0mq does internally.
There are some things that are not in 0MQ. Say the padding to avoid
cache-line pollution. We have a patch for that by Martin Pales and John
Dyte, but neither Martin Pales nor myself were able to show any speedup,
so the patch was not applied.
On the other hand, 0mq has several optimisations that the paper doesn't
account for afaics. First, we are using batching, ie. we move all the
queued messages to the other thread in a single step when the receiving
node asks for it. Second, there's some discussion in the article about
embedded data vs. passing pointers to data. In 0MQ we do both depending
on the message size. Embedded data is used for small messages, pointers
are used for large messages.
Martin
------------------------------
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
End of zeromq-dev Digest, Vol 41, Issue 104
*******************************************
More information about the zeromq-dev
mailing list