[zeromq-dev] EOF on file transfer

[Tiago Hillebrandt]

Hello,

The +1 is needed, since the range() starts on 0. Otherwise it will ignore
the last chunk.

Just found another solution using SEEK_CUR:

-filename.seek(offset)
+filename.seek(0, 1)

Thus, it will always continue from current (last) position (
https://docs.python.org/2/library/os.html#os.lseek ).

Thank you all for the suggestions.

Tiago



2014-12-09 16:02 GMT-06:00 KIU Shueng Chuan <nixchuan at gmail.com>:

> "range (offsets+1)" should just be "range (offsets)".
> "filename.seek (offset)" should be "filename.seek (offset * chunksize)"
>
> Try doing a local file copy with your loop first. The problem in your code
> is not with zeromq.
> On 10 Dec 2014 00:47, "Tiago Hillebrandt" <tiagohillebrandt at ubuntu.com>
> wrote:
>
>> Hey guys,
>>
>> I am using the below Python code to transfer big files between a server
>> and a client.
>>
>> *Implementation to send file, server*
>>
>>     CHUNK_SIZE = 250000
>>
>>     message = pair.recv() # message is the path to the file
>>
>>     filename = open(message, 'rb')
>>     filesize = os.path.getsize(message)
>>
>>     offsets = (int(ceil(filesize / CHUNK_SIZE)), 0)[filesize <= CHUNK_SIZE]
>>
>>     for offset in range(offsets + 1):
>>         filename.seek(offset)
>>
>>         chunksize = CHUNK_SIZE
>>
>>         if offset == offsets:
>>             chunksize = filesize - (CHUNK_SIZE * (offset - 1)) # calculate the size of the last chunk
>>
>>         data = filename.read(chunksize)
>>
>>         pair.send(data)
>>
>>     pair.send(b'')
>>
>> *Implementation to receive file, client*
>>
>> while True:
>>     data = pairs.recv()
>>
>>     if data is not '':
>>         target.write(data)
>>     else:
>>         break
>>
>> However, after transfer a big file using this implementation, for some
>> reason an extra data is being added at end of the file:
>>
>> *File server side*
>>
>> $ stat file.zip
>>   File: `file.zip'
>>   Size: 1503656416  Blocks: 2936840    IO Block: 4096   regular file
>>
>> *Client side*
>>
>> $ stat file.zip
>>   File: `file.zip'
>>   Size: 1503906416  Blocks: 2937328    IO Block: 4096   regular file
>>
>> The size and blocks are different between them.
>>
>> That said, do you have any suggestions to calculate/send the end of file
>> properly?
>>
>> Thanks!
>> --
>> *Tiago Hillebrandt*
>> Ubuntu Member
>> Ubuntu Brazilian Community Council Member
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>


-- 
*Tiago Hillebrandt*
Ubuntu Member
Ubuntu Brazilian Community Council Member
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20141209/39ad6aa3/attachment.htm>