[zeromq-dev] programmable message routing

Martin Sustrik sustrik at fastmq.com
Thu Jan 22 13:55:26 CET 2009


Hi Ferenc,

>> I am not sure what exactly does your scenario look like, but I suspect 
>> you may want load-balancing (i.e. fair distribution of messages 
>> between several queues). If that's the case have a look at 0.3.3 
>> branch in Subversion. There, you cas specify a 'load-balance' flag 
>> when creating the exchange that will cause exchange to distribute 
>> messages between the queues in round-robin manner.
> 
> To be honest it was just a toy scenario :) My original idea was to 
> implement Amazon SQS like message routing:
> 
> http://docs.amazonwebservices.com/AWSSimpleQueueService/2008-01-01/SQSGettingStartedGuide/

I've never used SQS myslef. I'll give it a read later on.

>> As for more complex routing, yes, it's on our roadmap. If you want to 
>> contribute here we'll be happy to collaborate with you.
>>
>> Basically, the idea of generalised routing is that there'll be another 
>> layer on top of existing 0MQ implementation that will define special 
>> section of the message (a.k.a. message header) containing data that 
>> can be used by routing algorithm. The layer should expose an API that 
>> would allow 0MQ users to specify the message header (this can be a 
>> simple integer, string or even a table of name-value pairs).
> 
> Will this API a part of the message class? Something like this:
> 
> message_t::add_header(void *header_data_, size_t header_size_);
> void *message_t::get_header_data();
> size_t message_t::get_header_size();

As for the API, there are 3 problems here IMO:

1. The above solution is not backward-compatible.
2. I am pretty sure users would hate filling in the headers in binary 
format.
3. How do we ensure that messages created with say string header will 
work with exchange implementation that routes messages on string but 
won't work with one that routes on integers?

The problem is complex and multi-faceted, but we can proceed to the 
solution in a gradual manner.

I would say creating several classes for different types of messages 
would help. Thus we'll have unmodified backward-compatible message_t + 
several derived message types, like this:

message_t msg1 (100);
integer_message_t msg2 (1, 120);
string_message_t msg3 ("abc", 120);
etc.

> How do you think the raw_message should modify to handle this?

There's no need to change raw_message_t, it handles the message as a 
transparent BLOB. The only thing to do is to fill in the header when 
contructing the message (in one of the specialised classes).

>> Once you have that you are able to write an exchange implementation 
>> that will check the message header and dispatch the message to 
>> appropriate destinations.
> 
> Currently in the api_thread the exchange type is std::pair <std::string, 
> out_engine_t*>. Which means that writing own exchange implementation 
> means writing own out_engine which is a simple wrapper around demux.
> How the users will be able to write own exchange without dealing 
> raw_message? Do you have any plan yet?

I would say the implemented of the exchange would simply choose the 
message type that it requires. Say, you want to write a topic exchange, 
so you expect that the message has string as the header.

Martin





More information about the zeromq-dev mailing list