[zeromq-dev] EOF on file transfer

[André Caron]

Hi there,

Maybe it's just me, but you code seems quite complicated for the task at
hand.  I don't understand why you need to seek at all since successive
reads on the same file handle remember the file pointer.

I got this to work locally with exactlythe following code:

    def sendfile(path, chunk_size, socket):
        checksum = hashlib.sha1()
        with open(path, 'rb') as stream:
            chunk = stream.read(chunk_size)
            while chunk:
                checksum.update(chunk)
                socket.send(chunk)
                chunk = stream.read(chunk_size)
            socket.send('')
        print checksum.hexdigest()

    def recvfile(path, socket):
        checksum = hashlib.sha1()
        with open(path, 'wb') as stream:
            chunk = socket.recv()
            while chunk:
                checksum.update(chunk)
                stream.write(chunk)
                chunk = socket.recv()
        print checksum.hexdigest()

The checksum will allow you to confirm that you get exactly the same
content as you sent (at least manually, since you're not sending the
checksum to the peer beforehand).

In addition to that, PAIR is probably not the right choice of socket type
for this application because it's not guaranteed to work correctly over
transports other than "inproc".  Relevant quote from the 4.1 docs:

> ZMQ_PAIR sockets are designed for inter-thread communication across the
zmq_inproc(7) transport and do not implement functionality such as
auto-reconnection. ZMQ_PAIR sockets are considered experimental and may
have other missing or broken aspects.

One notable case where it fails (and which I actually hit while testing
this) is that "socket.recv()" call will not unblock until the reader gets a
message.  If the TCP connection is lost, the receiver will block forever
unless the sender magically becomes aware of the disconnection and
explicitly reconnects.

Cheers,

André

On Tue, Dec 9, 2014 at 9:03 PM, Tiago Hillebrandt <
tiagohillebrandt at ubuntu.com> wrote:

> Hello,
>
> The +1 is needed, since the range() starts on 0. Otherwise it will ignore
> the last chunk.
>
> Just found another solution using SEEK_CUR:
>
> -filename.seek(offset)
> +filename.seek(0, 1)
>
> Thus, it will always continue from current (last) position (
> https://docs.python.org/2/library/os.html#os.lseek ).
>
> Thank you all for the suggestions.
>
> Tiago
>
>
>
> 2014-12-09 16:02 GMT-06:00 KIU Shueng Chuan <nixchuan at gmail.com>:
>
> "range (offsets+1)" should just be "range (offsets)".
>> "filename.seek (offset)" should be "filename.seek (offset * chunksize)"
>>
>> Try doing a local file copy with your loop first. The problem in your
>> code is not with zeromq.
>> On 10 Dec 2014 00:47, "Tiago Hillebrandt" <tiagohillebrandt at ubuntu.com>
>> wrote:
>>
>>> Hey guys,
>>>
>>> I am using the below Python code to transfer big files between a server
>>> and a client.
>>>
>>> *Implementation to send file, server*
>>>
>>>     CHUNK_SIZE = 250000
>>>
>>>     message = pair.recv() # message is the path to the file
>>>
>>>     filename = open(message, 'rb')
>>>     filesize = os.path.getsize(message)
>>>
>>>     offsets = (int(ceil(filesize / CHUNK_SIZE)), 0)[filesize <= CHUNK_SIZE]
>>>
>>>     for offset in range(offsets + 1):
>>>         filename.seek(offset)
>>>
>>>         chunksize = CHUNK_SIZE
>>>
>>>         if offset == offsets:
>>>             chunksize = filesize - (CHUNK_SIZE * (offset - 1)) # calculate the size of the last chunk
>>>
>>>         data = filename.read(chunksize)
>>>
>>>         pair.send(data)
>>>
>>>     pair.send(b'')
>>>
>>> *Implementation to receive file, client*
>>>
>>> while True:
>>>     data = pairs.recv()
>>>
>>>     if data is not '':
>>>         target.write(data)
>>>     else:
>>>         break
>>>
>>> However, after transfer a big file using this implementation, for some
>>> reason an extra data is being added at end of the file:
>>>
>>> *File server side*
>>>
>>> $ stat file.zip
>>>   File: `file.zip'
>>>   Size: 1503656416  Blocks: 2936840    IO Block: 4096   regular file
>>>
>>> *Client side*
>>>
>>> $ stat file.zip
>>>   File: `file.zip'
>>>   Size: 1503906416  Blocks: 2937328    IO Block: 4096   regular file
>>>
>>> The size and blocks are different between them.
>>>
>>> That said, do you have any suggestions to calculate/send the end of file
>>> properly?
>>>
>>> Thanks!
>>> --
>>> *Tiago Hillebrandt*
>>> Ubuntu Member
>>> Ubuntu Brazilian Community Council Member
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
>
> --
> *Tiago Hillebrandt*
> Ubuntu Member
> Ubuntu Brazilian Community Council Member
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20141210/97fd2c0f/attachment.htm>