[zeromq-dev] zmq_send and zmq_rcv fail with unknown error

[Brian Knox]

This is the statement that concerns me in your requirement:

"Expecting ZeroMQ to implement some sort of reliable message shipping. ie.
I want messages to be maintaned on a queue if the connection is no
available or the remote machine crashed. Logstash is in a remote machine."

Two zeromq sockets connected to each other in and of themselves are not
what I would consider a reliable queuing system.  Zeromq sockets do queue
messages in memory, and how many messages they will queue in memory is
somewhat configurable - but even with high water mark settings, a PUSH /
PULL connection in and of itself is not going to give you reliable message
delivery (and PUB/SUB by definition will not).  The "queue" in zeromq
sockets should be thought of more as a buffer than a "message queue".

If what you really need is reliable message shipping, such a thing
certainly can be implemented with zeromq - but it's more complex than
simply connecting two sockets together and sending messages over them.

The ZeroMQ guide itself is a great place to get ideas for such an
architecture:

http://zguide.zeromq.org/page:all

To the zeromq core developers on the list - if I'm misrepresenting anything
here please feel free to chime in - but this is how I view zeromq from my
own experience with it.

Brian


On Thu, Jan 10, 2013 at 9:44 AM, Simone <xbirkettx at gmail.com> wrote:

> Brian Knox <briank <at> talksum.com> writes:
>
> >
> > "Moreover, I tried changing the zmq_connect to connect to a random IP
> > address that does not exist and I still get a successfull connection but
> > obviously nothing was connected..."
> >
> > This is normal / expected behavior with ZeroMQ.
> >
> > As for your logstash issue, I  haven't worked with logstash before so I
> > can't hazard a guess.  However, using a ZMQ_PAIR socket to connect to
> > another process is a bit strange and I'm uncertain why you would do
> > that.  It doesn't make sense to me that logstash would listen on a
> > ZMQ_PAIR as ZMQ_PAIR is mainly for use with inter-thread message passing
> > using the inproc transport ( see the documentation at
> > http://api.zeromq.org/3-2:zmq-socket ).
> >
> > On 1/10/2013 5:18 AM, Simone wrote:
> > > Hi Everyone
> > >
> > > I am using logstash to receive logs from my application. The
> communication
> is
> > > done using ZeroMQ. The application sends to the logstash server using
> ZMQ_PAIR.
> > >
> > > The connection and context creation on the application side return no
> error,
> > > however by invoking either zmq_send or zmq_recv I get -1 and errno set
> to 0
> and
> > > zmq_errno() set to 128.
> > >
> > > Unknown errors.....
> > >
> > > Moreover, I tried changing the zmq_connect to connect to a random IP
> address
> > > that
> > > does not exist and I still get a successfull connection but obviously
> nothing
> > > was
> > > connected...
> > >
> > > I am working on Windows x64 using ZeroMQ 2.2 on the client side (since
> this
> is
> > > the version supported by Logstash). Logstash does not seem to be the
> problem
> > > since I am quite sure the connection is not even established.
> > >
> > > I installed ZeroMQ from http://www.zeromq.org/distro:microsoft-windowsusing
> 2.2
> > > for Windows 7 and using the library libzmq-v100-mt.dll for running,
> precompiled,
> > > contained in the bin directory of the installation dir.
> > >
> > > Any idea?
> > >
> > > thanks a lot
> > >
> > > Simone Zandara
> > > Network Pogrammer - Massive Ubisoft Entertainment
> > >
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev <at> lists.zeromq.org
> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
>
> I see, well I dont know why but Logstash only listen on TCP connections
> over
> ZeroMQ.
>
> Moreover I tried following a simple example like this
>
> http://learning-0mq-with-
> pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pair.html
>
> Not implementing the server myself but using logstash as server.
> It creates a very dummy connection over TCP.
>
> Hwever the link you gave me, explicitly says that PAIR is indicated for
> interproc, so I cannot really expect it to work I guess.
>
> By the way what I am trying to do is
>
>
>  ------------                   ----------      -------------
> | APPLICATION | ---ZeroMQ----> | LOGSTASH | -> |ElasticSearch|
>  ------------                   ----------      -------------
>
> Expecting ZeroMQ to implement some sort of reliable message shipping. ie.
> I want
> messages to be maintaned on a queue if the connection is no available or
> the
> remote machine crashed. Logstash is in a remote machine.
>
> Logstash only supports three communication types
>
> PAIR
> PUSH/PULL
> PUB/SUB
>
> Shall I try another type or shall I just give up this way?
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130111/3537d122/attachment.htm>