[zeromq-dev] Communication between C++ zmq client and Python zmq Server
Thomas Rodgers
rodgert at twrodgers.com
Thu Feb 1 05:23:59 CET 2018
Correction! Should read -
std::copy_n(result.c_str(), result.size() + 1, request.data());
OR
std::copy_n(result.data(), result.size() + 1, request.data());
On Wed, Jan 31, 2018 at 10:18 PM, Thomas Rodgers <rodgert at twrodgers.com>
wrote:
> Is this correct?
>
>
> Yes
>
>
>> 1. What advantages do I have when using something like protocol buffer to
>> serialization data instead of
>> send the data of a basic way such as I perform here turn the data to
>> string and sending it?
>>
>
> If you want to send complex data structures, portably between different
> programming language environments and processor architectures.
> This is not a goal of ZeroMQ and is explicitly left to other libraries and
> tools.
>
> | 2. Why use memcpy is bad or not suited?
>
> http://en.cppreference.com/w/cpp/algorithm/copy
> http://en.cppreference.com/w/cpp/algorithm/copy_n
>
> are better alternatives in C++ and as, or more efficient than ::memcpy
> (they in fact delegate, as inline code to ::memcpy if that is the fastest
> way to perform the copy).
>
> your example would read -
>
> #include <algorithm>
>
> ...
>
> std::copy_n(request.data(), result.size(), result.c_str());
>
> OR
>
> std::copy_n(request.data(), result.size(), result.begin());
>
> On Wed, Jan 31, 2018 at 12:33 PM, Bernardo Augusto García Loaiza <
> botibagl at gmail.com> wrote:
>
>> Hi Thomas, thanks for your appreciations
>>
>> In the memcpy function in the second parameter named const void * source
>> ...
>>
>> void * memcpy ( void * destination, const void * source, size_t num );
>>
>>
>> the c_strt() function is suited place here because this return a
>> pointer to the variable value
>>
>> Is this correct?
>>
>> [image: Inline image 1]
>>
>> I am C++ and ZMQ newbie, although is a great idea measure dynamically the
>> size of the request
>> in the moment of send the data via zmq such as you tell me.
>>
>> According to this, the data is sent to my server and I can visualize your
>> content of a suited way.
>>
>> I have two questions:
>>
>>
>> 1. What advantages do I have when using something like protocol buffer to
>> serialization data instead of
>> send the data of a basic way such as I perform here turn the data to
>> string and sending it?
>>
>> I've been check inside the zmq documentations about it
>> <http://zguide.zeromq.org/page:all#Serializing-Your-Data>, but I don't
>> have clear if serialization
>> in first instance is necessary or which the advantages and against
>>
>>
>> 2. Why use memcpy is bad or not suited?
>> I've been read about it, and I found ...
>>
>> memcpy copies a block of raw memory from one location to another ...
>> Several of the inconvenient are (basically it's a copy/paste from some
>> links):
>>
>> - Overlapping source and destination buffers: The behaviour of memcpy
>> is undefined if the source and destination buffers overlap
>>
>> How to could overlap the source and destination buffers in one zmq
>> operation? Is right think in this question?
>>
>>
>>
>> - Maintaining alignment
>>
>> Data types such as integers and floating point numbers are often subject
>> to an alignment constraint that limits where they can be placed in memory.
>> For example, 4-byte integers must often be placed at an address that is a
>> multiple of 4 bytes. Normally the compiler would ensure that alignment
>> constraints are respected, but using memcpy it is possible to copy an
>> object to a misaligned address
>>
>> In other site, I found this about of bandwidth demand copying things:
>>
>> There is nothing wrong with memcpy, except that it takes a long time.
>> Excessive use of memory copying is sometimes a sign of poor program
>> design.
>> In a real-time system, it may be outright unacceptable. For example, if
>> you
>> copy the entire packet data at each layer transition in a protocol stack,
>> you
>> may find that your stack cannot handle as high a bandwidth as it
>> otherweise
>> could, thereby failing to meet requirements.
>>
>> In some lectures talk about of use memmove() instead of memcpy, in
>> relation to buffer overlapping ...
>>
>> In the context of performance and data transport via zmq, wha is the
>> recommended option to
>> use instead of memcpy
>>
>> Because zeromq in their website documentation samples, use memcpy
>> <http://zguide.zeromq.org/cpp:hwserver> ?
>> Is this a old version to work witn zmq to connect to socket and send data?
>>
>> Thanks for all your orientation about it.
>> Best Regards
>>
>>
>>
>>
>>
>>
>>
>>
>> Bernardo Augusto García Loaiza
>> Ingeniero de Sistemas
>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>> http://about.me/bgarcial
>>
>>
>> On Tue, Jan 30, 2018 at 6:13 PM, Thomas Rodgers <rodgert at twrodgers.com>
>> wrote:
>>
>>> You can’t memcpy a std::string, and in fact, you should t be using
>>> memcpy in general in C++. I won’t go into the why of all of that here, but
>>> start by changing -
>>>
>>> memcpy(request.data(), *&result*, 30);
>>>
>>> To
>>>
>>> memcpy(request.data(), *result.c_str()*, result.size() + 1);
>>>
>>>
>>> On Tue, Jan 30, 2018 at 4:46 PM Bernardo Augusto García Loaiza <
>>> botibagl at gmail.com> wrote:
>>>
>>>> Hi, I again.
>>>>
>>>> I've performed some small test in relation to the way on how to I
>>>> sending the data member structure.
>>>>
>>>> I am turning the int data member structure to string before to be send
>>>> it ..
>>>> In the following code section I am turn btnState in string data and
>>>> assign it to string s variable and I contatenated it some additional string
>>>> content and
>>>> I sending this string concatenated result:
>>>>
>>>> void ZMQComponent::instrumentDataSend(instrumentData a){
>>>> /* Initialize the data members structure instrumentData */
>>>> a.pos = sofa::defaulttype::Vec3d(1.0f, 1.0f, 1.0f);
>>>> a.quat = defaulttype::Quat(1.0f, 1.0f, 4.0f, 1.0f);
>>>> a.btnState = 5671;
>>>> a.openInst = 1.0f;
>>>> a.blnDataReady = false;
>>>>
>>>>
>>>>
>>>> * string s, test, result, d;
>>>> s = to_string(a.btnState);
>>>> test = " is a number";
>>>> result = s + test;*
>>>>
>>>>
>>>>
>>>> /* We send the btnState data */
>>>>
>>>>
>>>> zmq::message_t request(30);
>>>>
>>>>
>>>>
>>>>
>>>> /* We ask for the memory address to ge the btnState content and send it. */
>>>>
>>>>
>>>> memcpy(request.data(), *&result*, 30);
>>>> socket.send(request);}
>>>>
>>>>
>>>> The output or the message which arrive to python zmq server is the
>>>> content of
>>>> result variable (btnState turn to string in s content variable + string
>>>> test concatenated)
>>>> and some symbols characters of the :
>>>>
>>>> (cnvss_test) ➜ Python git:(ZMQCommunication) ✗ python server.py
>>>> ZMQ Server listening ...
>>>>
>>>> *Received message from Sofa: b'\xb0\x1d\x19\xf4\xfd\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x0045 is a number'*
>>>>
>>>> If I assign a value less than 30 in the request, by example zmq::message_t
>>>> request(10) the output in my server is:
>>>>
>>>> Received message from Sofa: b'\x90\x94\xa1\x00\xfc\x7f\x00\x00\x0e\x00'
>>>>
>>>> If I assign a value greater than 10 in the request, by example zmq::message_t
>>>> request(20) the output in my server is:
>>>>
>>>> Received message from Sofa: b'\x80$(\xc7\xfc\x7f\x00\x00\x0e\x00\x00\x00\x00\x00\x00\x00*45 i*
>>>>
>>>> Then, the string or object which I receive in the server side, it has
>>>> as long as the length or size assigned to zmq::message_t request
>>>> variable
>>>>
>>>>
>>>> Based in the above mentioned, is ZMQ whom add this strings in my
>>>> message received?
>>>>
>>>> According to the previous process, my message is arrived to my server,
>>>>
>>>> then ....
>>>>
>>>>
>>>> is correct attempt what the serialization process with some entity
>>>> like protocol buffer is necessary?
>>>>
>>>>
>>>> I understand that use something like google protocol buffer allow have
>>>> some correlation more controlled in relation to objects sent and the
>>>> objects received in relation of their real content ...
>>>>
>>>> In any case, how to can I remove the strings or characters symbols that
>>>> are added in the message arrived to the server?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Bernardo Augusto García Loaiza
>>>> Ingeniero de Sistemas
>>>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>>>> http://about.me/bgarcial
>>>>
>>>>
>>>> On Sat, Jan 27, 2018 at 11:24 AM, Thomas Rodgers <rodgert at twrodgers.com
>>>> > wrote:
>>>>
>>>>> “strat team” was supposed to be std::stringstream (Thanks for nothing
>>>>> autocorrect!)
>>>>>
>>>>> http://en.cppreference.com/w/cpp/io/basic_stringstream
>>>>>
>>>>> On Sat, Jan 27, 2018 at 10:22 AM Thomas Rodgers <rodgert at twrodgers.com>
>>>>> wrote:
>>>>>
>>>>>> Strings are the simplest. If you don’t need to parse the data on the
>>>>>> C++ side, just format it as a JSON map on the C++ side, stream it into a
>>>>>> strat team, send the resulting string on the socket and use Python’s
>>>>>> support for parsing JSON on the received strong. For instance.
>>>>>>
>>>>>> You can also send structured binary to Python, as I mentioned; the
>>>>>> Python CTypes module (included with Python) will let you define accessors
>>>>>> for that data. As long as the sending and receiving side are the same CPU
>>>>>> architecture you don’t have to worry about endianess conversions.
>>>>>>
>>>>>> On Sat, Jan 27, 2018 at 9:13 AM Bernardo Augusto García Loaiza <
>>>>>> botibagl at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Luca, this mean, then, with structured string or data is
>>>>>>> necessary some middleware entity like GPB such as Thomas tell us ...
>>>>>>>
>>>>>>>
>>>>>>> Bernardo Augusto García Loaiza
>>>>>>> Ingeniero de Sistemas
>>>>>>> Estudiante de Maestría en Ingeniería Informática - Universidad EAFIT
>>>>>>> http://about.me/bgarcial
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Jan 27, 2018 at 10:07 AM, Luca Boccassi <
>>>>>>> luca.boccassi at gmail.com> wrote:
>>>>>>>
>>>>>>>> Yes you can just send unstructured binary data or strings - Python
>>>>>>>> has
>>>>>>>> native helpers for strings, and CZMQ as well
>>>>>>>>
>>>>>>>> On Fri, 2018-01-26 at 21:27 -0500, Bernardo Augusto García Loaiza
>>>>>>>> wrote:
>>>>>>>> > Thomas, I understand perfectly your explanation. But I want share
>>>>>>>> > with you
>>>>>>>> > another similar case, just for curiosity or to receive some
>>>>>>>> > additional
>>>>>>>> > orientation about it.
>>>>>>>> >
>>>>>>>> > I have a script client C++ ZEROMQ as a documentation, and I have
>>>>>>>> > script
>>>>>>>> > server python, together based in the documentation zeromq website.
>>>>>>>> > This mean, together are scripts basics
>>>>>>>> >
>>>>>>>> > From C++ client I am sending the string message
>>>>>>>> > <https://github.com/bgarcial/zeroMQ_Client/blob/master/clien
>>>>>>>> t.cpp#L27
>>>>>>>> > > and
>>>>>>>> > I have a python server which receive this string message client
>>>>>>>> > <https://github.com/bgarcial/zeroMQ_Client/blob/master/Pytho
>>>>>>>> n/server.
>>>>>>>> > py#L20>
>>>>>>>> > of a direct way and show me their content
>>>>>>>> >
>>>>>>>> > In this basic sample (documentation zmq based) I am not
>>>>>>>> > using structures
>>>>>>>> > data members. Is for this reason that in this case I receive and
>>>>>>>> I
>>>>>>>> > can see
>>>>>>>> > the content of the client message in the zmq server side without
>>>>>>>> have
>>>>>>>> > use
>>>>>>>> > something like Google Protocol Buffer, This mean, is because in
>>>>>>>> this
>>>>>>>> > case I
>>>>>>>> > am not sending any structures data members?
>>>>>>>> >
>>>>>>>> > I appreciate one more your orientation
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Bernardo Augusto García Loaiza
>>>>>>>> > Ingeniero de Sistemas
>>>>>>>> > Estudiante de Maestría en Ingeniería Informática - Universidad
>>>>>>>> EAFIT
>>>>>>>> > http://about.me/bgarcial
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Fri, Jan 26, 2018 at 7:44 AM, Thomas Rodgers <
>>>>>>>> rodgert at twrodgers.co
>>>>>>>> > m>
>>>>>>>> > wrote:
>>>>>>>> >
>>>>>>>> > > Short answer yes, it’s related to different languages. TCP and
>>>>>>>> > > ZeroMQ only
>>>>>>>> > > deal with transport and framing of messages (respectively)
>>>>>>>> Python
>>>>>>>> > > has no
>>>>>>>> > > idea that that 10 bytes corresponds to a C/C++ struct of a given
>>>>>>>> > > layout
>>>>>>>> > > until you tell it. Different platforms (CPU architectures) have
>>>>>>>> > > potentially
>>>>>>>> > > different representations of fundamental types.
>>>>>>>> > >
>>>>>>>> > > On Thu, Jan 25, 2018 at 4:37 PM Bernardo Augusto García Loaiza <
>>>>>>>> > > botibagl at gmail.com> wrote:
>>>>>>>> > >
>>>>>>>> > > > One more question. Why is necessary the serialization
>>>>>>>> process? It
>>>>>>>> > > > is
>>>>>>>> > > > assumed that zeromq works via TCP and in the TCP protocol, the
>>>>>>>> > > > data does
>>>>>>>> > > > not undergo any transformation? This is related with the fact
>>>>>>>> of
>>>>>>>> > > > that I
>>>>>>>> > > > have different platform/languages?
>>>>>>>> > > >
>>>>>>>> > > > On Thu, Jan 25, 2018 at 4:45 PM Bernardo Augusto García
>>>>>>>> Loaiza <
>>>>>>>> > > > botibagl at gmail.com> wrote:
>>>>>>>> > > >
>>>>>>>> > > > > Hi Thomas,
>>>>>>>> > > > > Thanks for your illustrative response
>>>>>>>> > > > >
>>>>>>>> > > > > I'll look Google Protocol Buffers. My sender is from C++
>>>>>>>> > > > > language and
>>>>>>>> > > > > my reception is Python. What sort of installation recommend
>>>>>>>> me
>>>>>>>> > > > > you?
>>>>>>>> > > > > Binaries or build protocol buffer along my C++ runtime? or o
>>>>>>>> > > > > build protoc
>>>>>>>> > > > > binary from source?
>>>>>>>> > > > >
>>>>>>>> > > > > On Wed, Jan 24, 2018 at 8:36 PM Thomas Rodgers
>>>>>>>> <rodgert at twrodge
>>>>>>>> > > > > rs.com>
>>>>>>>> > > > > wrote:
>>>>>>>> > > > >
>>>>>>>> > > > > > You can have a look at Python’s ctypes module, which will
>>>>>>>> let
>>>>>>>> > > > > > you
>>>>>>>> > > > > > define a ‘struct’ from Python with the same layout as your
>>>>>>>> > > > > > C++ struct.
>>>>>>>> > > > > >
>>>>>>>> > > > > > You can also investigate any number of serialization
>>>>>>>> > > > > > libraries that
>>>>>>>> > > > > > have C++ and Python support, eg ProtoBufs or Thrift, or
>>>>>>>> > > > > > MagPack or whatever.
>>>>>>>> > > > > >
>>>>>>>> > > > > > On Wed, Jan 24, 2018 at 5:26 PM Bernardo Augusto García
>>>>>>>> > > > > > Loaiza <
>>>>>>>> > > > > > botibagl at gmail.com> wrote:
>>>>>>>> > > > > >
>>>>>>>> > > > > > > Hi, ZMQ people.
>>>>>>>> > > > > > > Greetings.
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > I have a C++ zeromq client process in which I am
>>>>>>>> sending
>>>>>>>> > > > > > > some data
>>>>>>>> > > > > > > members structures
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > *ZMQComponent.h* file
>>>>>>>> > > > > > >
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > #include <zmq.hpp>
>>>>>>>> > > > > > > #include <sofa/defaulttype/VecTypes.h>
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > // To Quat datatype
>>>>>>>> > > > > > > #include <sofa/defaulttype/Quat.h>
>>>>>>>> > > > > > > using sofa::defaulttype::Quat;
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > using std::string;
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > namespace sofa
>>>>>>>> > > > > > > {
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > namespace component
>>>>>>>> > > > > > > {
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > namespace controller
>>>>>>>> > > > > > > {
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > /* data structure which I want send data to python zmq
>>>>>>>> > > > >
>>>>>>>>
>>>>>>> ...
>>>
>>> [Message clipped]
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180131/6b9ebb5d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 52201 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180131/6b9ebb5d/attachment.png>
More information about the zeromq-dev
mailing list