[zeromq-dev] PUB/SUB in realtime, aka real radio broadcast

Samuele Carli carlisamuele at csspace.net
Sun Nov 10 15:36:46 CET 2013


Hello,

I would like to realize something similar to a real radio broadcast:

- a publisher distributes packets of information
- subscribers receive and process information

but in my case timing is very important; to keep the radio analogy:

- all subscribers receive the same information pretty much at the same time
- if a subscriber stops listening for a few seconds, it loses some words but does not lose synchronization with others (when he starts listening again, he's where every one else is in the story)

I thought that setting a very low HWM (1) in a publisher-subscriber socket would pretty much solve my problems, but instead I find that HWM is somewhat ignored.

A very very ingenious and terribly coded example would have, on the server side, a radio broadcasting a sequential number and a timestamp, in a somewhat verbose way to increase the package size to some credible value:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.setsockopt(zmq.SNDHWM, 1)
socket.setsockopt(zmq.RCVHWM, 1)

socket.bind("tcp://*:9999")

counter = 0
while True:
	msg = "%d-%s " % (counter,time.time())*100
	socket.send(msg)
	print 'sent ', msg[0:20].split(' ')[0]
	counter += 1
	time.sleep(0.005)	


And the listener, instead, hears a message, and wonders how long it took to get the message:

import zmq
import time

#  Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.SUB)

socket.setsockopt(zmq.SNDHWM, 1)
socket.setsockopt(zmq.RCVHWM, 1)
socket.setsockopt(zmq.DELAY_ATTACH_ON_CONNECT, 1)
socket.setsockopt(zmq.SUBSCRIBE, '')
socket.connect ("tcp://localhost:9999")

while True:
		msg = socket.recv()
		t = time.time()
		count, mt = msg[0:25].split(' ')[0].split('-')
		print 'got: ', str(msg[0:25].split(' ')[0]), ' at: ', t, '  len:  ', len(msg) , ' delay: ', t - float(mt)
		time.sleep(0.5)



What I would expect in this case, is indeed to see a lot of packet loss on the receiver side, but very small delay.

Instead, what happens is that somehow there is no packet loss and the delay increases to very unreasonable values, an output example from the client is:
[...]
got:  325-1384093263.38  at:  1384093264.38   len:   1800  delay:  0.997805833817
got:  326-1384093263.39  at:  1384093265.38   len:   1800  delay:  1.98881793022
got:  327-1384093263.39  at:  1384093266.38   len:   1800  delay:  2.98981881142
got:  328-1384093263.4  at:  1384093267.38   len:   1700  delay:  3.98081588745
got:  329-1384093263.4  at:  1384093268.38   len:   1700  delay:  4.98182487488
got:  330-1384093263.41  at:  1384093269.38   len:   1800  delay:  5.97282195091
got:  331-1384093263.41  at:  1384093270.38   len:   1800  delay:  6.97381591797
got:  332-1384093263.42  at:  1384093271.38   len:   1800  delay:  7.96481204033
got:  333-1384093263.42  at:  1384093272.39   len:   1800  delay:  8.96581697464
got:  334-1384093263.43  at:  1384093273.39   len:   1800  delay:  9.95681405067
got:  335-1384093263.43  at:  1384093274.39   len:   1800  delay:  10.9578130245
got:  336-1384093263.44  at:  1384093275.39   len:   1800  delay:  11.9488120079
got:  337-1384093263.44  at:  1384093276.39   len:   1800  delay:  12.9498069286
got:  338-1384093263.45  at:  1384093277.39   len:   1800  delay:  13.9408349991
got:  339-1384093263.45  at:  1384093278.39   len:   1800  delay:  14.9418139458
[...]

meanwhile the server already printed:

[...]
sent  3263-1384093278.39
sent  3264-1384093278.4
sent  3265-1384093278.4
sent  3266-1384093278.41
sent  3267-1384093278.41
sent  3268-1384093278.42
sent  3269-1384093278.42
[...]

Changing the network buffer size even to unreasonably small settings, for example:

socket.setsockopt(zmq.SNDBUF, 256)
socket.setsockopt(zmq.RCVBUF, 256)

on both ends, does have very little effect unless the packets you send are much bigger then the buffer (try changing the multiplier to 9999 or more): in that case the receiver barely receives any packages, but still with somewhat big delay.


This said, anybody would like to help me to understand what's going on here?
Maybe there is a way to eat all broadcasted packets in one go and keep only the last one?
Is this configuration of any practical interest to anyone?

I'm on debian sid, python 2.7.5 and zmq version 3.2.4.

Thank you very much!

Cheers,
Samuele

-- 
|--
| Samuele Carli
|--
| Contacts:
|
|	Home page   : www.csspace.net
|       E-mail      : carlisamuele _at_ csspace.net
|       Icq         : 60401601
|	Skype	    : wohthan
|	jabber/gtalk: wohthan at gmail.com (no emails here!)
|--



More information about the zeromq-dev mailing list