[zeromq-dev] Pattern to use for distributed db

Noospheer Team team at noospheer.org
Fri Sep 16 08:28:33 CEST 2011


Hello 0mq ppl, I'm part of a project that's building a distributed db
from mostly existing parts, we intend on utilizing 0mq as the
communication mechanism amongst nodes running the software. Db objects
are key-value pairs where the hash function generates a key that
indicates its value's connections to other values. If some of those
connected values are stored on a separate node, it (theoretically)
knows the URI.

So, a server node needs to propagate a query to a known target, allow
the target node to process the query and return requested data to
originating node (which then forwards the data to the client who made
the query). What style of pattern seems suitable?

Warm regards,
Jordan Ash, Noospheer

On 9/16/11, zeromq-dev-request at lists.zeromq.org
<zeromq-dev-request at lists.zeromq.org> wrote:
> Send zeromq-dev mailing list submissions to
> 	zeromq-dev at lists.zeromq.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> or, via email, send a message with subject or body 'help' to
> 	zeromq-dev-request at lists.zeromq.org
>
> You can reach the person managing the list at
> 	zeromq-dev-owner at lists.zeromq.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of zeromq-dev digest..."
>
>
> Today's Topics:
>
>    1. Zero copy messages (Emmanuel TAUREL)
>    2. Re: Zero copy messages (Martin Sustrik)
>    3. Re: Zero copy messages (Martin Sustrik)
>    4. Invitation to connect on LinkedIn (Ben Kloosterman)
>    5. Subscription Forwarding Issue (Robert Walters)
>    6. Invitation to connect on LinkedIn (Qi Chen via LinkedIn)
>    7. Re: Subscription Forwarding Issue (Martin Sustrik)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 15 Sep 2011 17:18:08 +0200
> From: Emmanuel TAUREL <taurel at esrf.fr>
> Subject: [zeromq-dev] Zero copy messages
> To: zeromq-dev at mail.imatix.com
> Message-ID: <4E721730.60003 at esrf.fr>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hello all,
>
> I am using ZMQ 3.0.x  running on a Ubuntu 10.10 box (32 bits).
> I am using the PUB/SUB pattern. The publisher sends multipart messages.
> I am trying to use zero copy messages in the publisher side.
> The buffer I give to the zmq messages are statically allocated. They are
> data member of one object which has the process lifetime.
> Therefore, my free method is empty.
>
> Because it is zero copy, I assume that ZMQ does not copy the buffer
> contents.
> Is it true that I should keep these buffers unchanged until the free
> method is called?
>
> Thank's very much for your answers
>
> Emmanuel Taurel
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 15 Sep 2011 17:19:57 +0200
> From: Martin Sustrik <sustrik at 250bpm.com>
> Subject: Re: [zeromq-dev] Zero copy messages
> To: taurel at esrf.fr, ZeroMQ development list
> 	<zeromq-dev at lists.zeromq.org>
> Cc: zeromq-dev at mail.imatix.com
> Message-ID: <4E72179D.4030001 at 250bpm.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Emmanuel,
>
>> I am using ZMQ 3.0.x  running on a Ubuntu 10.10 box (32 bits).
>> I am using the PUB/SUB pattern. The publisher sends multipart messages.
>> I am trying to use zero copy messages in the publisher side.
>> The buffer I give to the zmq messages are statically allocated. They are
>> data member of one object which has the process lifetime.
>> Therefore, my free method is empty.
>>
>> Because it is zero copy, I assume that ZMQ does not copy the buffer
>> contents.
>> Is it true that I should keep these buffers unchanged until the free
>> method is called?
>
> Yes. Exactly.
>
> Martin
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 15 Sep 2011 17:19:57 +0200
> From: Martin Sustrik <sustrik at 250bpm.com>
> Subject: Re: [zeromq-dev] Zero copy messages
> To: taurel at esrf.fr, ZeroMQ development list
> 	<zeromq-dev at lists.zeromq.org>
> Cc: zeromq-dev at mail.imatix.com
> Message-ID: <4E72179D.4030001 at 250bpm.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Emmanuel,
>
>> I am using ZMQ 3.0.x  running on a Ubuntu 10.10 box (32 bits).
>> I am using the PUB/SUB pattern. The publisher sends multipart messages.
>> I am trying to use zero copy messages in the publisher side.
>> The buffer I give to the zmq messages are statically allocated. They are
>> data member of one object which has the process lifetime.
>> Therefore, my free method is empty.
>>
>> Because it is zero copy, I assume that ZMQ does not copy the buffer
>> contents.
>> Is it true that I should keep these buffers unchanged until the free
>> method is called?
>
> Yes. Exactly.
>
> Martin
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 16 Sep 2011 02:08:29 +0000 (UTC)
> From: Ben Kloosterman <bklooste at gmail.com>
> Subject: [zeromq-dev] Invitation to connect on LinkedIn
> To: ZeroMQ list <zeromq-dev at lists.zeromq.org>
> Message-ID:
> 	<622823233.4318013.1316138909250.JavaMail.app at ela4-app0134.prod>
> Content-Type: text/plain; charset="utf-8"
>
> ZeroMQ,
>
> I'd like to add you to my professional network on LinkedIn.
>
> - Ben
>
> Ben Kloosterman
> Information Technology and Services Professional
> China
>
> Confirm that you know Ben Kloosterman:
> https://www.linkedin.com/e/rcou07-gsmj4htr-66/isd/4230137651/UAP83XU_/?hs=false&tok=0vjD_dgVUwJAU1
>
> --
> You are receiving Invitation to Connect emails. Click to unsubscribe:
> http://www.linkedin.com/e/rcou07-gsmj4htr-66/uHS0kWuC-XSoHJRLrIS3nj7LVh3xbx9J9q3Zow6w0v/goo/zeromq-dev%40lists%2Ezeromq%2Eorg/20061/I1461585083_1/?hs=false&tok=0ulZcjuUEwJAU1
>
> (c) 2011 LinkedIn Corporation. 2029 Stierlin Ct, Mountain View, CA 94043,
> USA.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110916/3b457694/attachment-0001.htm
>
> ------------------------------
>
> Message: 5
> Date: Thu, 15 Sep 2011 22:59:06 -0400
> From: "Robert Walters" <walters.robert at mail.com>
> Subject: [zeromq-dev] Subscription Forwarding Issue
> To: zeromq-dev at lists.zeromq.org
> Message-ID: <20110916025907.219780 at gmx.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi All,
>
>  We are testing the subscription forwarding in 3.0.1. We have based our test
> code on the test_sub_forward.cpp file. We subscribe from the client, and see
> that the subscription is forwarded via the XPUB/XSUB mechanism. The client
> is filtering the packages based on our subscription but still receiving all
> the pacakages (looking at the tcpip traffic in wireshark). We assumed the
> filterting would be done at the publisher end. Are we doing something wrong?
>
>  Cheers,
>  Robert
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110915/3eced910/attachment-0001.htm
>
> ------------------------------
>
> Message: 6
> Date: Fri, 16 Sep 2011 03:56:20 +0000 (UTC)
> From: Qi Chen via LinkedIn <member at linkedin.com>
> Subject: [zeromq-dev] Invitation to connect on LinkedIn
> To: jinjing wu <zeromq-dev at lists.zeromq.org>
> Message-ID:
> 	<1926389683.4526561.1316145380263.JavaMail.app at ela4-app0131.prod>
> Content-Type: text/plain; charset="utf-8"
>
> LinkedIn
> ------------
>
>
>
>
>     Qi Chen requested to add you as a connection on LinkedIn:
>
> ------------------------------------------
>
> jinjing,
>
> I'd like to add you to my professional network on LinkedIn.
>
> - Qi
>
> Accept invitation from Qi Chen
> http://www.linkedin.com/e/rcou07-gsmmz6w3-b/uHS0kWuC-XSoHJRLrIS3nj7LVh3xbx9J9q3Zow6w0v/blk/I20245846_140/1BpC5vrmRLoRZcjkkZt5YCpnlOt3RApnhMpmdzgmhxrSNBszYMd35vdzgUdjgOc399bQJLnRF5rDhQbPoUcPcTej0PczgLrCBxbOYWrSlI/EML_comm_afe/?hs=false&tok=1Xqrf8zC-cJAU1
>
> View profile of Qi Chen
> http://www.linkedin.com/e/rcou07-gsmmz6w3-b/rsn/95178787/OXER/?hs=false&tok=0bvdQuapacJAU1
> ------------------------------------------
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110916/2f2ac034/attachment-0001.htm
>
> ------------------------------
>
> Message: 7
> Date: Fri, 16 Sep 2011 07:53:32 +0200
> From: Martin Sustrik <sustrik at 250bpm.com>
> Subject: Re: [zeromq-dev] Subscription Forwarding Issue
> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> Cc: Robert Walters <walters.robert at mail.com>
> Message-ID: <4E72E45C.1050609 at 250bpm.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> On 09/16/2011 04:59 AM, Robert Walters wrote:
>> Hi All,
>>
>> We are testing the subscription forwarding in 3.0.1. We have based our
>> test code on the test_sub_forward.cpp file. We subscribe from the
>> client, and see that the subscription is forwarded via the XPUB/XSUB
>> mechanism. The client is filtering the packages based on our
>> subscription but still receiving all the pacakages (looking at the tcpip
>> traffic in wireshark). We assumed the filterting would be done at the
>> publisher end. Are we doing something wrong?
>
> Pleas post the minimal test case that reproduces the issue.
>
> Thanks!
> Martin
>
>
> ------------------------------
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> End of zeromq-dev Digest, Vol 45, Issue 35
> ******************************************
>



More information about the zeromq-dev mailing list