[zeromq-dev] Communication between C++ zmq client and Python zmq Server

Thomas Rodgers rodgert at twrodgers.com
Thu Feb 1 05:18:49 CET 2018 

>
> Is this correct?


Yes


> 1. What advantages do I have when using something like protocol buffer to
> serialization data instead of
> send the data of a basic way such as I perform here turn the data to
> string and sending it?
>

If you want to send complex data structures, portably between different
programming language environments and processor architectures.
This is not a goal of ZeroMQ and is explicitly left to other libraries and
tools.

| 2.  Why use memcpy is bad or not suited?

http://en.cppreference.com/w/cpp/algorithm/copy
http://en.cppreference.com/w/cpp/algorithm/copy_n

are better alternatives in C++ and as, or more efficient than ::memcpy
(they in fact delegate, as inline code to ::memcpy if that is the fastest
way to perform the copy).

your example would read -

#include <algorithm>

...

std::copy_n(request.data(), result.size(), result.c_str());

OR

std::copy_n(request.data(), result.size(), result.begin());

On Wed, Jan 31, 2018 at 12:33 PM, Bernardo Augusto García Loaiza <
botibagl at gmail.com> wrote:

> Hi Thomas, thanks for your appreciations
>
> In the memcpy function in the second parameter named const void * source
> ...
>
> void * memcpy ( void * destination, const void * source, size_t num );
>
>
> the  c_strt() function is suited place here because this  return a pointer
> to the variable value
>
> Is this correct?
>
> [image: Inline image 1]
>
> I am C++ and ZMQ newbie, although is a great idea measure dynamically the
> size of the request
> in the moment of send the data via zmq such as you tell me.
>
> According to this, the data is sent to my server and I can visualize your
> content of a suited way.
>
> I have two questions:
>
>
> 1. What advantages do I have when using something like protocol buffer to
> serialization data instead of
> send the data of a basic way such as I perform here turn the data to
> string and sending it?
>
> I've been check inside the zmq documentations about it
> <http://zguide.zeromq.org/page:all#Serializing-Your-Data>, but I don't
> have clear if serialization
> in first instance is necessary or which the advantages and against
>
>
> 2.  Why use memcpy is bad or not suited?
> I've been read about it, and I found ...
>
> memcpy copies a block of raw memory  from one location to another ...
> Several of the inconvenient are (basically it's a copy/paste from some
> links):
>
>    - Overlapping source and destination buffers: The behaviour of memcpy
>    is undefined if the source and destination buffers overlap
>
> How to could overlap the source and destination buffers  in one zmq
> operation? Is right think in this question?
>
>
>
>    - Maintaining alignment
>
> Data types such as integers and floating point numbers are often subject
> to an alignment constraint that limits where they can be placed in memory.
> For example, 4-byte integers must often be placed at an address that is a
> multiple of 4 bytes. Normally the compiler would ensure that alignment
> constraints are respected, but using memcpy it is possible to copy an
> object to a misaligned address
>
> In other site, I found this about of bandwidth demand copying things:
>
> There is nothing wrong with memcpy, except that it takes a long time.
> Excessive use of memory copying is sometimes a sign of poor program
> design.
> In a real-time system, it may be outright unacceptable. For example, if
> you
> copy the entire packet data at each layer transition in a protocol stack,
> you
> may find that your stack cannot handle as high a bandwidth as it
> otherweise
> could, thereby failing to meet requirements.
>
> In some lectures talk about of use memmove() instead of memcpy, in
> relation to buffer overlapping ...
>
> In the context  of performance and data transport via zmq, wha is the
> recommended option to
> use instead of memcpy
>
> Because zeromq in their website documentation samples, use memcpy
> <http://zguide.zeromq.org/cpp:hwserver> ?
> Is this a old version to work witn zmq to connect to socket and send data?
>
> Thanks for all your orientation about it.
> Best Regards
>
>
>
>
>
>
>
>
> Bernardo Augusto García Loaiza
> Ingeniero de Sistemas
> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
> http://about.me/bgarcial
>
>
> On Tue, Jan 30, 2018 at 6:13 PM, Thomas Rodgers <rodgert at twrodgers.com>
> wrote:
>
>> You can’t memcpy a std::string, and in fact, you should t be using memcpy
>> in general in C++. I won’t go into the why of all of that here, but start
>> by changing -
>>
>> memcpy(request.data(), *&result*, 30);
>>
>> To
>>
>> memcpy(request.data(), *result.c_str()*, result.size() + 1);
>>
>>
>> On Tue, Jan 30, 2018 at 4:46 PM Bernardo Augusto García Loaiza <
>> botibagl at gmail.com> wrote:
>>
>>> Hi, I again.
>>>
>>> I've performed some small test in relation to the way on how to I
>>> sending the data member structure.
>>>
>>> I am turning the int data member structure to string before to be send
>>> it ..
>>> In the following code section  I am turn btnState in string data and
>>> assign it to string s variable and I contatenated it some additional string
>>> content and
>>> I sending this string concatenated result:
>>>
>>> void ZMQComponent::instrumentDataSend(instrumentData a){
>>>     /*  Initialize the data members structure instrumentData */
>>>     a.pos = sofa::defaulttype::Vec3d(1.0f, 1.0f, 1.0f);
>>>     a.quat = defaulttype::Quat(1.0f, 1.0f, 4.0f, 1.0f);
>>>     a.btnState = 5671;
>>>     a.openInst = 1.0f;
>>>     a.blnDataReady = false;
>>>
>>>
>>>
>>>    * string s, test, result, d;
>>>     s = to_string(a.btnState);
>>>     test = " is a number";
>>>     result = s + test;*
>>>
>>>
>>>
>>>     /*  We send  the btnState data  */
>>>
>>>
>>>     zmq::message_t request(30);
>>>
>>>
>>>
>>>
>>> /*  We ask for the memory address to ge the btnState content and send it. */
>>>
>>>
>>>     memcpy(request.data(), *&result*, 30);
>>>     socket.send(request);}
>>>
>>>
>>> The output or the message which arrive to python zmq server is the
>>> content of
>>> result variable (btnState turn to string in s content variable + string
>>> test concatenated)
>>> and some symbols characters of the :
>>>
>>> (cnvss_test) ➜  Python git:(ZMQCommunication) ✗ python server.py
>>> ZMQ Server listening ...
>>>
>>> *Received message from Sofa: b'\xb0\x1d\x19\xf4\xfd\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 is a number'*
>>>
>>> If I assign a value less than 30 in the request, by example zmq::message_t
>>> request(10) the output in my server is:
>>>
>>> Received message from Sofa: b'\x90\x94\xa1\x00\xfc\x7f\x00\x00\x0e\x00'
>>>
>>> If I assign a value greater than 10 in the request, by example zmq::message_t
>>> request(20) the output in my server is:
>>>
>>> Received message from Sofa: b'\x80$(\xc7\xfc\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x00*45 i*
>>>
>>> Then, the string or object which I receive in the server side, it has as
>>> long as the length or size assigned to zmq::message_t request variable
>>>
>>>
>>> Based in the above mentioned, is ZMQ whom add this strings in my message
>>> received?
>>>
>>> According to the previous process, my message is arrived to my server,
>>>
>>> then ....
>>>
>>>
>>>  is correct attempt what the serialization process with some entity like
>>> protocol buffer is necessary?
>>>
>>>
>>>  I understand that use something like google protocol buffer allow have
>>> some correlation more controlled in relation to objects sent and the
>>> objects received in relation of their real content ...
>>>
>>> In any case, how to can I remove the strings or characters symbols that
>>> are added in the message arrived to the server?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Bernardo Augusto García Loaiza
>>> Ingeniero de Sistemas
>>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>>> http://about.me/bgarcial
>>>
>>>
>>> On Sat, Jan 27, 2018 at 11:24 AM, Thomas Rodgers <rodgert at twrodgers.com>
>>> wrote:
>>>
>>>> “strat team” was supposed to be std::stringstream (Thanks for nothing
>>>> autocorrect!)
>>>>
>>>> http://en.cppreference.com/w/cpp/io/basic_stringstream
>>>>
>>>> On Sat, Jan 27, 2018 at 10:22 AM Thomas Rodgers <rodgert at twrodgers.com>
>>>> wrote:
>>>>
>>>>> Strings are the simplest. If you don’t need to parse the data on the
>>>>> C++ side, just format it as a JSON map on the C++ side, stream it into a
>>>>> strat team, send the resulting string on the socket and use Python’s
>>>>> support for parsing JSON on the received strong. For instance.
>>>>>
>>>>> You can also send structured binary to Python, as I mentioned; the
>>>>> Python CTypes module (included with Python) will let you define accessors
>>>>> for that data. As long as the sending and receiving side are the same CPU
>>>>> architecture you don’t have to worry about endianess conversions.
>>>>>
>>>>> On Sat, Jan 27, 2018 at 9:13 AM Bernardo Augusto García Loaiza <
>>>>> botibagl at gmail.com> wrote:
>>>>>
>>>>>> Hi Luca, this mean, then, with structured string or data is necessary
>>>>>> some middleware entity like GPB such as Thomas tell us ...
>>>>>>
>>>>>>
>>>>>> Bernardo Augusto García Loaiza
>>>>>> Ingeniero de Sistemas
>>>>>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>>>>>> http://about.me/bgarcial
>>>>>>
>>>>>>
>>>>>> On Sat, Jan 27, 2018 at 10:07 AM, Luca Boccassi <
>>>>>> luca.boccassi at gmail.com> wrote:
>>>>>>
>>>>>>> Yes you can just send unstructured binary data or strings - Python
>>>>>>> has
>>>>>>> native helpers for strings, and CZMQ as well
>>>>>>>
>>>>>>> On Fri, 2018-01-26 at 21:27 -0500, Bernardo Augusto García Loaiza
>>>>>>> wrote:
>>>>>>> > Thomas, I understand perfectly your explanation. But I want share
>>>>>>> > with you
>>>>>>> > another similar case, just for curiosity or to receive some
>>>>>>> > additional
>>>>>>> > orientation about it.
>>>>>>> >
>>>>>>> > I have a script client C++ ZEROMQ  as a documentation, and I have
>>>>>>> > script
>>>>>>> > server python, together based in the documentation zeromq website.
>>>>>>> > This mean, together are scripts basics
>>>>>>> >
>>>>>>> > From C++ client I am sending the string message
>>>>>>> > <https://github.com/bgarcial/zeroMQ_Client/blob/master/clien
>>>>>>> t.cpp#L27
>>>>>>> > >  and
>>>>>>> > I have a python server which receive this string message client
>>>>>>> > <https://github.com/bgarcial/zeroMQ_Client/blob/master/Pytho
>>>>>>> n/server.
>>>>>>> > py#L20>
>>>>>>> > of  a direct way and show me their content
>>>>>>> >
>>>>>>> > In this basic sample (documentation zmq based) I am not
>>>>>>> > using  structures
>>>>>>> > data  members. Is for this reason that in this case I receive and I
>>>>>>> > can see
>>>>>>> > the content of the client message in the zmq server side without
>>>>>>> have
>>>>>>> > use
>>>>>>> > something like Google Protocol Buffer, This mean, is because in
>>>>>>> this
>>>>>>> > case I
>>>>>>> > am not sending any structures data  members?
>>>>>>> >
>>>>>>> > I appreciate one more your orientation
>>>>>>> >
>>>>>>> >
>>>>>>> > Bernardo Augusto García Loaiza
>>>>>>> > Ingeniero de Sistemas
>>>>>>> > Estudiante de Maestría en Ingeniería Informática - Universidad
>>>>>>> EAFIT
>>>>>>> > http://about.me/bgarcial
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Jan 26, 2018 at 7:44 AM, Thomas Rodgers <
>>>>>>> rodgert at twrodgers.co
>>>>>>> > m>
>>>>>>> > wrote:
>>>>>>> >
>>>>>>> > > Short answer yes, it’s related to different languages. TCP and
>>>>>>> > > ZeroMQ only
>>>>>>> > > deal with transport and framing of messages (respectively) Python
>>>>>>> > > has no
>>>>>>> > > idea that that 10 bytes corresponds to a C/C++ struct of a given
>>>>>>> > > layout
>>>>>>> > > until you tell it. Different platforms (CPU architectures) have
>>>>>>> > > potentially
>>>>>>> > > different representations of fundamental types.
>>>>>>> > >
>>>>>>> > > On Thu, Jan 25, 2018 at 4:37 PM Bernardo Augusto García Loaiza <
>>>>>>> > > botibagl at gmail.com> wrote:
>>>>>>> > >
>>>>>>> > > > One more question. Why is necessary the serialization process?
>>>>>>> It
>>>>>>> > > > is
>>>>>>> > > > assumed that zeromq works via TCP and in the TCP protocol, the
>>>>>>> > > > data does
>>>>>>> > > > not undergo any transformation? This is related with the fact
>>>>>>> of
>>>>>>> > > > that I
>>>>>>> > > > have different platform/languages?
>>>>>>> > > >
>>>>>>> > > > On Thu, Jan 25, 2018 at 4:45 PM Bernardo Augusto García Loaiza
>>>>>>> <
>>>>>>> > > > botibagl at gmail.com> wrote:
>>>>>>> > > >
>>>>>>> > > > > Hi Thomas,
>>>>>>> > > > > Thanks for your illustrative response
>>>>>>> > > > >
>>>>>>> > > > > I'll look Google Protocol Buffers. My sender is from C++
>>>>>>> > > > > language and
>>>>>>> > > > > my reception is Python. What sort of installation recommend
>>>>>>> me
>>>>>>> > > > > you?
>>>>>>> > > > > Binaries or build protocol buffer along my C++ runtime? or o
>>>>>>> > > > > build protoc
>>>>>>> > > > > binary from source?
>>>>>>> > > > >
>>>>>>> > > > > On Wed, Jan 24, 2018 at 8:36 PM Thomas Rodgers
>>>>>>> <rodgert at twrodge
>>>>>>> > > > > rs.com>
>>>>>>> > > > > wrote:
>>>>>>> > > > >
>>>>>>> > > > > > You can have a look at Python’s ctypes module, which will
>>>>>>> let
>>>>>>> > > > > > you
>>>>>>> > > > > > define a ‘struct’ from Python with the same layout as your
>>>>>>> > > > > > C++ struct.
>>>>>>> > > > > >
>>>>>>> > > > > > You can also investigate any number of serialization
>>>>>>> > > > > > libraries that
>>>>>>> > > > > > have C++ and Python support, eg ProtoBufs or Thrift, or
>>>>>>> > > > > > MagPack or whatever.
>>>>>>> > > > > >
>>>>>>> > > > > > On Wed, Jan 24, 2018 at 5:26 PM Bernardo Augusto García
>>>>>>> > > > > > Loaiza <
>>>>>>> > > > > > botibagl at gmail.com> wrote:
>>>>>>> > > > > >
>>>>>>> > > > > > > Hi, ZMQ people.
>>>>>>> > > > > > > Greetings.
>>>>>>> > > > > > >
>>>>>>> > > > > > >
>>>>>>> > > > > > > I have a  C++ zeromq client process in which I am sending
>>>>>>> > > > > > > some data
>>>>>>> > > > > > > members structures
>>>>>>> > > > > > >
>>>>>>> > > > > > > *ZMQComponent.h* file
>>>>>>> > > > > > >
>>>>>>> > > > > > >
>>>>>>> > > > > > > #include <zmq.hpp>
>>>>>>> > > > > > > #include <sofa/defaulttype/VecTypes.h>
>>>>>>> > > > > > >
>>>>>>> > > > > > > // To Quat datatype
>>>>>>> > > > > > > #include <sofa/defaulttype/Quat.h>
>>>>>>> > > > > > > using sofa::defaulttype::Quat;
>>>>>>> > > > > > >
>>>>>>> > > > > > > using std::string;
>>>>>>> > > > > > >
>>>>>>> > > > > > > namespace sofa
>>>>>>> > > > > > > {
>>>>>>> > > > > > >
>>>>>>> > > > > > > namespace component
>>>>>>> > > > > > > {
>>>>>>> > > > > > >
>>>>>>> > > > > > > namespace controller
>>>>>>> > > > > > > {
>>>>>>> > > > > > >
>>>>>>> > > > > > > /* data structure which I want send data to python zmq
>>>>>>> > > > >
>>>>>>>
>>>>>> ...
>>
>> [Message clipped]
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180131/bce87f8c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 52201 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180131/bce87f8c/attachment.png>

More information about the zeromq-dev mailing list