[zeromq-dev] Communication between C++ zmq client and Python zmq Server

Bernardo Augusto García Loaiza botibagl at gmail.com
Wed Jan 31 19:33:58 CET 2018


Hi Thomas, thanks for your appreciations

In the memcpy function in the second parameter named const void * source ...

void * memcpy ( void * destination, const void * source, size_t num );


the  c_strt() function is suited place here because this  return a pointer
to the variable value

Is this correct?

[image: Inline image 1]

I am C++ and ZMQ newbie, although is a great idea measure dynamically the
size of the request
in the moment of send the data via zmq such as you tell me.

According to this, the data is sent to my server and I can visualize your
content of a suited way.

I have two questions:


1. What advantages do I have when using something like protocol buffer to
serialization data instead of
send the data of a basic way such as I perform here turn the data to string
and sending it?

I've been check inside the zmq documentations about it
<http://zguide.zeromq.org/page:all#Serializing-Your-Data>, but I don't have
clear if serialization
in first instance is necessary or which the advantages and against


2.  Why use memcpy is bad or not suited?
I've been read about it, and I found ...

memcpy copies a block of raw memory  from one location to another ...
Several of the inconvenient are (basically it's a copy/paste from some
links):

   - Overlapping source and destination buffers: The behaviour of memcpy is
   undefined if the source and destination buffers overlap

How to could overlap the source and destination buffers  in one zmq
operation? Is right think in this question?



   - Maintaining alignment

Data types such as integers and floating point numbers are often subject to
an alignment constraint that limits where they can be placed in memory. For
example, 4-byte integers must often be placed at an address that is a
multiple of 4 bytes. Normally the compiler would ensure that alignment
constraints are respected, but using memcpy it is possible to copy an
object to a misaligned address

In other site, I found this about of bandwidth demand copying things:

There is nothing wrong with memcpy, except that it takes a long time.
Excessive use of memory copying is sometimes a sign of poor program design.
In a real-time system, it may be outright unacceptable. For example, if you
copy the entire packet data at each layer transition in a protocol stack,
you
may find that your stack cannot handle as high a bandwidth as it otherweise
could, thereby failing to meet requirements.

In some lectures talk about of use memmove() instead of memcpy, in relation
to buffer overlapping ...

In the context  of performance and data transport via zmq, wha is the
recommended option to
use instead of memcpy

Because zeromq in their website documentation samples, use memcpy
<http://zguide.zeromq.org/cpp:hwserver> ?
Is this a old version to work witn zmq to connect to socket and send data?

Thanks for all your orientation about it.
Best Regards








Bernardo Augusto García Loaiza
Ingeniero de Sistemas
Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
http://about.me/bgarcial


On Tue, Jan 30, 2018 at 6:13 PM, Thomas Rodgers <rodgert at twrodgers.com>
wrote:

> You can’t memcpy a std::string, and in fact, you should t be using memcpy
> in general in C++. I won’t go into the why of all of that here, but start
> by changing -
>
> memcpy(request.data(), *&result*, 30);
>
> To
>
> memcpy(request.data(), *result.c_str()*, result.size() + 1);
>
>
> On Tue, Jan 30, 2018 at 4:46 PM Bernardo Augusto García Loaiza <
> botibagl at gmail.com> wrote:
>
>> Hi, I again.
>>
>> I've performed some small test in relation to the way on how to I sending
>> the data member structure.
>>
>> I am turning the int data member structure to string before to be send it
>> ..
>> In the following code section  I am turn btnState in string data and
>> assign it to string s variable and I contatenated it some additional string
>> content and
>> I sending this string concatenated result:
>>
>> void ZMQComponent::instrumentDataSend(instrumentData a){
>>     /*  Initialize the data members structure instrumentData */
>>     a.pos = sofa::defaulttype::Vec3d(1.0f, 1.0f, 1.0f);
>>     a.quat = defaulttype::Quat(1.0f, 1.0f, 4.0f, 1.0f);
>>     a.btnState = 5671;
>>     a.openInst = 1.0f;
>>     a.blnDataReady = false;
>>
>>
>>
>>    * string s, test, result, d;
>>     s = to_string(a.btnState);
>>     test = " is a number";
>>     result = s + test;*
>>
>>
>>
>>     /*  We send  the btnState data  */
>>
>>
>>     zmq::message_t request(30);
>>
>>
>>
>>
>> /*  We ask for the memory address to ge the btnState content and send it. */
>>
>>
>>     memcpy(request.data(), *&result*, 30);
>>     socket.send(request);}
>>
>>
>> The output or the message which arrive to python zmq server is the
>> content of
>> result variable (btnState turn to string in s content variable + string
>> test concatenated)
>> and some symbols characters of the :
>>
>> (cnvss_test) ➜  Python git:(ZMQCommunication) ✗ python server.py
>> ZMQ Server listening ...
>>
>> *Received message from Sofa: b'\xb0\x1d\x19\xf4\xfd\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 is a number'*
>>
>> If I assign a value less than 30 in the request, by example zmq::message_t
>> request(10) the output in my server is:
>>
>> Received message from Sofa: b'\x90\x94\xa1\x00\xfc\x7f\x00\x00\x0e\x00'
>>
>> If I assign a value greater than 10 in the request, by example zmq::message_t
>> request(20) the output in my server is:
>>
>> Received message from Sofa: b'\x80$(\xc7\xfc\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x00*45 i*
>>
>> Then, the string or object which I receive in the server side, it has as
>> long as the length or size assigned to zmq::message_t request variable
>>
>>
>> Based in the above mentioned, is ZMQ whom add this strings in my message
>> received?
>>
>> According to the previous process, my message is arrived to my server,
>>
>> then ....
>>
>>
>>  is correct attempt what the serialization process with some entity like
>> protocol buffer is necessary?
>>
>>
>>  I understand that use something like google protocol buffer allow have
>> some correlation more controlled in relation to objects sent and the
>> objects received in relation of their real content ...
>>
>> In any case, how to can I remove the strings or characters symbols that
>> are added in the message arrived to the server?
>>
>>
>>
>>
>>
>>
>>
>>
>> Bernardo Augusto García Loaiza
>> Ingeniero de Sistemas
>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>> http://about.me/bgarcial
>>
>>
>> On Sat, Jan 27, 2018 at 11:24 AM, Thomas Rodgers <rodgert at twrodgers.com>
>> wrote:
>>
>>> “strat team” was supposed to be std::stringstream (Thanks for nothing
>>> autocorrect!)
>>>
>>> http://en.cppreference.com/w/cpp/io/basic_stringstream
>>>
>>> On Sat, Jan 27, 2018 at 10:22 AM Thomas Rodgers <rodgert at twrodgers.com>
>>> wrote:
>>>
>>>> Strings are the simplest. If you don’t need to parse the data on the
>>>> C++ side, just format it as a JSON map on the C++ side, stream it into a
>>>> strat team, send the resulting string on the socket and use Python’s
>>>> support for parsing JSON on the received strong. For instance.
>>>>
>>>> You can also send structured binary to Python, as I mentioned; the
>>>> Python CTypes module (included with Python) will let you define accessors
>>>> for that data. As long as the sending and receiving side are the same CPU
>>>> architecture you don’t have to worry about endianess conversions.
>>>>
>>>> On Sat, Jan 27, 2018 at 9:13 AM Bernardo Augusto García Loaiza <
>>>> botibagl at gmail.com> wrote:
>>>>
>>>>> Hi Luca, this mean, then, with structured string or data is necessary
>>>>> some middleware entity like GPB such as Thomas tell us ...
>>>>>
>>>>>
>>>>> Bernardo Augusto García Loaiza
>>>>> Ingeniero de Sistemas
>>>>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>>>>> http://about.me/bgarcial
>>>>>
>>>>>
>>>>> On Sat, Jan 27, 2018 at 10:07 AM, Luca Boccassi <
>>>>> luca.boccassi at gmail.com> wrote:
>>>>>
>>>>>> Yes you can just send unstructured binary data or strings - Python has
>>>>>> native helpers for strings, and CZMQ as well
>>>>>>
>>>>>> On Fri, 2018-01-26 at 21:27 -0500, Bernardo Augusto García Loaiza
>>>>>> wrote:
>>>>>> > Thomas, I understand perfectly your explanation. But I want share
>>>>>> > with you
>>>>>> > another similar case, just for curiosity or to receive some
>>>>>> > additional
>>>>>> > orientation about it.
>>>>>> >
>>>>>> > I have a script client C++ ZEROMQ  as a documentation, and I have
>>>>>> > script
>>>>>> > server python, together based in the documentation zeromq website.
>>>>>> > This mean, together are scripts basics
>>>>>> >
>>>>>> > From C++ client I am sending the string message
>>>>>> > <https://github.com/bgarcial/zeroMQ_Client/blob/master/
>>>>>> client.cpp#L27
>>>>>> > >  and
>>>>>> > I have a python server which receive this string message client
>>>>>> > <https://github.com/bgarcial/zeroMQ_Client/blob/master/
>>>>>> Python/server.
>>>>>> > py#L20>
>>>>>> > of  a direct way and show me their content
>>>>>> >
>>>>>> > In this basic sample (documentation zmq based) I am not
>>>>>> > using  structures
>>>>>> > data  members. Is for this reason that in this case I receive and I
>>>>>> > can see
>>>>>> > the content of the client message in the zmq server side without
>>>>>> have
>>>>>> > use
>>>>>> > something like Google Protocol Buffer, This mean, is because in this
>>>>>> > case I
>>>>>> > am not sending any structures data  members?
>>>>>> >
>>>>>> > I appreciate one more your orientation
>>>>>> >
>>>>>> >
>>>>>> > Bernardo Augusto García Loaiza
>>>>>> > Ingeniero de Sistemas
>>>>>> > Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>>>>>> > http://about.me/bgarcial
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Jan 26, 2018 at 7:44 AM, Thomas Rodgers <
>>>>>> rodgert at twrodgers.co
>>>>>> > m>
>>>>>> > wrote:
>>>>>> >
>>>>>> > > Short answer yes, it’s related to different languages. TCP and
>>>>>> > > ZeroMQ only
>>>>>> > > deal with transport and framing of messages (respectively) Python
>>>>>> > > has no
>>>>>> > > idea that that 10 bytes corresponds to a C/C++ struct of a given
>>>>>> > > layout
>>>>>> > > until you tell it. Different platforms (CPU architectures) have
>>>>>> > > potentially
>>>>>> > > different representations of fundamental types.
>>>>>> > >
>>>>>> > > On Thu, Jan 25, 2018 at 4:37 PM Bernardo Augusto García Loaiza <
>>>>>> > > botibagl at gmail.com> wrote:
>>>>>> > >
>>>>>> > > > One more question. Why is necessary the serialization process?
>>>>>> It
>>>>>> > > > is
>>>>>> > > > assumed that zeromq works via TCP and in the TCP protocol, the
>>>>>> > > > data does
>>>>>> > > > not undergo any transformation? This is related with the fact of
>>>>>> > > > that I
>>>>>> > > > have different platform/languages?
>>>>>> > > >
>>>>>> > > > On Thu, Jan 25, 2018 at 4:45 PM Bernardo Augusto García Loaiza <
>>>>>> > > > botibagl at gmail.com> wrote:
>>>>>> > > >
>>>>>> > > > > Hi Thomas,
>>>>>> > > > > Thanks for your illustrative response
>>>>>> > > > >
>>>>>> > > > > I'll look Google Protocol Buffers. My sender is from C++
>>>>>> > > > > language and
>>>>>> > > > > my reception is Python. What sort of installation recommend me
>>>>>> > > > > you?
>>>>>> > > > > Binaries or build protocol buffer along my C++ runtime? or o
>>>>>> > > > > build protoc
>>>>>> > > > > binary from source?
>>>>>> > > > >
>>>>>> > > > > On Wed, Jan 24, 2018 at 8:36 PM Thomas Rodgers
>>>>>> <rodgert at twrodge
>>>>>> > > > > rs.com>
>>>>>> > > > > wrote:
>>>>>> > > > >
>>>>>> > > > > > You can have a look at Python’s ctypes module, which will
>>>>>> let
>>>>>> > > > > > you
>>>>>> > > > > > define a ‘struct’ from Python with the same layout as your
>>>>>> > > > > > C++ struct.
>>>>>> > > > > >
>>>>>> > > > > > You can also investigate any number of serialization
>>>>>> > > > > > libraries that
>>>>>> > > > > > have C++ and Python support, eg ProtoBufs or Thrift, or
>>>>>> > > > > > MagPack or whatever.
>>>>>> > > > > >
>>>>>> > > > > > On Wed, Jan 24, 2018 at 5:26 PM Bernardo Augusto García
>>>>>> > > > > > Loaiza <
>>>>>> > > > > > botibagl at gmail.com> wrote:
>>>>>> > > > > >
>>>>>> > > > > > > Hi, ZMQ people.
>>>>>> > > > > > > Greetings.
>>>>>> > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > > > I have a  C++ zeromq client process in which I am sending
>>>>>> > > > > > > some data
>>>>>> > > > > > > members structures
>>>>>> > > > > > >
>>>>>> > > > > > > *ZMQComponent.h* file
>>>>>> > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > > > #include <zmq.hpp>
>>>>>> > > > > > > #include <sofa/defaulttype/VecTypes.h>
>>>>>> > > > > > >
>>>>>> > > > > > > // To Quat datatype
>>>>>> > > > > > > #include <sofa/defaulttype/Quat.h>
>>>>>> > > > > > > using sofa::defaulttype::Quat;
>>>>>> > > > > > >
>>>>>> > > > > > > using std::string;
>>>>>> > > > > > >
>>>>>> > > > > > > namespace sofa
>>>>>> > > > > > > {
>>>>>> > > > > > >
>>>>>> > > > > > > namespace component
>>>>>> > > > > > > {
>>>>>> > > > > > >
>>>>>> > > > > > > namespace controller
>>>>>> > > > > > > {
>>>>>> > > > > > >
>>>>>> > > > > > > /* data structure which I want send data to python zmq
>>>>>> > > > >
>>>>>>
>>>>> ...
>
> [Message clipped]
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180131/cdfd48b0/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 52201 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180131/cdfd48b0/attachment.png>


More information about the zeromq-dev mailing list