[zeromq-dev] Erlang style messaging

Pieter Hintjens ph at imatix.com
Fri Aug 29 11:01:27 CEST 2014


Steve,

Like I suggested, look at the CZMQ zactor class and the several
examples of actor classes in that library. You'll find most if not all
the answers you're looking for. It turns out to be surprisingly simple
to do proper multithreading in C, all you need is ZeroMQ inproc:// and
some patterns for building APIs on top of that.

-Pieter

On Fri, Aug 29, 2014 at 2:13 AM, Steve Murphy <murf at parsetree.com> wrote:
> Trevor,
>
> True, if the actor model can cover all multithreaded situations, then
> that's the way to go. The question then becomes, does the actor
> model cover all multithreaded situations, and if apparently not, can
> the situation be adapted to turn it back into an actor compatible
> situation?
>
> Been thinking, based on my Asterisk experience, what "lessons"
> I/we have learned.
>
>    Asterisk has over 140 different lock init calls. Of all the structs
> locked,
>    channels are the biggest and busiest, and most central.
>    trylock() is called from around 100 different spots in the code.
>    mutex_lock is called from around 900 different spots in the code.
>    mutex_unlock from over 1000 places.
>    To cut down the number of deadlocks, they have 3 different
> DEADLOCK_AVOIDANCE()
>    macros, which are called in 36 places in the code, mostly channel
> drivers. They
>    briefly drop the lock, and pick it up again.
>
> I see obvious ways to organize things for concurrency in a non-concurrent
> language like C:
> a. Thread per object instance, with command queue. That way, no instance
>     field can be modified without "serializing" the requests in its command
> queue.
>     What's an object? anywhere you see a malloc for a struct, or a "new"
> operator
>     being performed, and that struct/class/whatever could have mutliple
> threads want
>     to read or write it, then that's an "object instance". It could be a
> list, a table,
>     or an element in such, ...anything.
> b. Sub-object instance: (trying to learn lessons from my Asterisk
> experiences)
>     Just FYI, Asterisk has a channel struct that includes 23 pointers to
> structs,
>                   4 string pointers, 11 stringfield pointers (to get rid of
> unused space),
>                   25 sub-structs, 13 sub-struct pointers, maybe to lists,
>                   an array of ints for fd's, and another for a pair of pipe
> fd's,
>                   20 individual int fields,  6 fixed length string
>                   arrays, and I may have under-counted some of these.
>                   In Asterisk, sometimes, you have to lock the channel
> struct before
>                   you get the subfield lock.  At any time, someone might
> hang up,
>                   and several channels may blink out of existence.  To keep
> these
>                   opaque, there are over 200 get/set functions.  I also see
> that
>                   channels are outfitted with ref counts, so all you have to
> do, to  insure
>                   your channel pointer doesn't get destroyed while you hold
> it, you bump
>                   its refcount, and it will remain alive until you decrement
> the count.
>             Soooo,
>    Objects can be hierarchical... So, maybe, you need
>    to NOT allow different threads to directly access the sub-objects. So,
> You
>    ask the Parent object to change the sub-objects, be it to modify an
> element
>    in a list, or add or remove elements from a list. If you have an object,
>    like an Asterisk Channel structure, instead of handing out pointers to
> the subojbects,
>    and locking on everything, you make update requests to the channel object
>    instance instead. Hmmm. That would mean a pretty big set of add/mod/del
>    requests, and a fairly huge variety of data objects being passed in and
> fetched out.
>    It looks like Asterisk has been working to make channels opaque over the
> years.
> c. TRY TO LIVE WITHOUT WAITING FOR A RESPONSE after a making a mod.
>     With locks, you wait around until the change is made, so you can remove
> the lock.
>     How often do you actually need to have the change requested,
> accomplished, before
>     continuing? Hardly ever, or almost always?
> d. That's it. I'm still looking for the magic bullets to help architect
> large and complex
>     multithreaded projects. Does anyone have any other guidelines?
>
> murf
>
>
>
> On Thu, Aug 28, 2014 at 9:30 AM, Trevor Bernard <trevor.bernard at gmail.com>
> wrote:
>>
>> Murf,
>>
>> It depends on how you phrase the problem -- If you only ever process
>> the messages you receive on a single thread and in order, you will
>> never deadlock because you don't have the necessary conditions to
>> deadlock, which are shared state and more than one thread
>> accessing/modifying it concurrently. This is how "actors" work. The
>> code you write to handle the messages is thread safe. You achieve
>> parallelism by distributing the work out to multiple worker processes
>> via message passing.
>>
>>
>> On Thu, Aug 28, 2014 at 11:58 AM, Steve Murphy <murf at parsetree.com> wrote:
>> > Pieter--
>> >
>> > Last year, I read the book, Programming Erlang, by Joe Armstrong, and I
>> > was
>> > fascinated by the ideology behind the general thread-per-object
>> > approach,
>> > where
>> > each "object" is managed by its own thread, via message passing.
>> >
>> > Erlang has a really "involved" message passing scheme, involving pattern
>> > matching, a "mailbox", recieve timer, a "save queue". Needless to say,
>> > all
>> > this makes a very powerful way of prioritizing messages, so a busy
>> > object
>> > manager can pull high-priority requests from the mailbox and act on them
>> > immediately, saving lower priority requests for later.
>> >
>> > I see in a paper at http://zeromq.org/blog:multithreading-magic, the
>> > same
>> > sort of admiration for Erlang's methodology.
>> >
>> > But...
>> >
>> > I'm not seeing the cure-all, end-all, solution to concurrency problems,
>> > and
>> > it bothers me, because I'm probably missing something fundamental,
>> > something
>> > I should have picked up on, but didn't.
>> >
>> > Erlang allows other processes/threads to drop requests in an object's
>> > mailbox, but
>> > it also has a mechanism for waiting until the action is complete, as the
>> > "object"
>> > can send a response.
>> >
>> > It's this response wait that is the killer. Now, I've done a lot of work
>> > on/with Asterisk,
>> > and it is heavily mulithreaded, in the old-school way, and has a ton of
>> > critical
>> > sections, and locks, and mutiple locks for a single action. They have
>> > evolved
>> > some fairly involved strategies to avoid deadlocks, including putting a
>> > timer
>> > on the lock, and if it times out, giving up the lock they already have,
>> > and
>> > starting
>> > over, allowing the contending party to obtain the lock they need, finish
>> > their
>> > "thing", which allows you to continue and obtain the lock you need to do
>> > your
>> > "thing". And on and on it goes.
>> >
>> > Now, I didn't go and look up the particulars about "N-party dance",
>> > etc.,
>> > but
>> > the classic resource deadlock situations still seem
>> > in play when you have to wait for completion. A asks B to
>> > complete a task, and waits for him to respond. In order to get that
>> > done, B
>> > requests
>> > A for something, and waits forever for A to finish. And so on. Perhaps C
>> > or
>> > even D
>> > are also involved.
>> > I keep thinking that such basic situations
>> > aren't solved by
>> > switching to the Erlang methods. There must be some architectural,
>> > perhaps
>> > hierarchical organizing,
>> > some sort of general design practice, that can
>> > overcome these kinds of problems, I'm just blind to it at the moment.
>> >
>> > Situations like 'atomic' changes on two or more objects at once, etc.
>> > and I
>> > don't
>> > see in the fog, how Erlang solves these problems in general. Can someone
>> > point me to some literature that might make things clear?
>> >
>> > murf
>> >
>> > --
>> >
>> > Steve Murphy
>> > ParseTree Corporation
>> > 57 Lane 17
>> > Cody, WY 82414
>> > ✉  murf at parsetree dot com
>> > ☎ 307-899-5535
>> >
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
>
> --
>
> Steve Murphy
> ParseTree Corporation
> 57 Lane 17
> Cody, WY 82414
> ✉  murf at parsetree dot com
> ☎ 307-899-5535
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list