[zeromq-dev] Erlang style messaging

Steve Murphy murf at parsetree.com
Fri Aug 29 02:13:45 CEST 2014


True, if the actor model can cover all multithreaded situations, then
that's the way to go. The question then becomes, does the actor
model cover all multithreaded situations, and if apparently not, can
the situation be adapted to turn it back into an actor compatible

Been thinking, based on my Asterisk experience, what "lessons"
I/we have learned.

   Asterisk has over 140 different lock init calls. Of all the structs
   channels are the biggest and busiest, and most central.
   trylock() is called from around 100 different spots in the code.
   mutex_lock is called from around 900 different spots in the code.
   mutex_unlock from over 1000 places.
   To cut down the number of deadlocks, they have 3 different
   macros, which are called in 36 places in the code, mostly channel
drivers. They
   briefly drop the lock, and pick it up again.

I see obvious ways to organize things for concurrency in a non-concurrent
language like C:
a. Thread per object instance, with command queue. That way, no instance
    field can be modified without "serializing" the requests in its command
    What's an object? anywhere you see a malloc for a struct, or a "new"
    being performed, and that struct/class/whatever could have mutliple
threads want
    to read or write it, then that's an "object instance". It could be a
list, a table,
    or an element in such, ...anything.
b. Sub-object instance: (trying to learn lessons from my Asterisk
    Just FYI, Asterisk has a channel struct that includes 23 pointers to
                  4 string pointers, 11 stringfield pointers (to get rid of
unused space),
                  25 sub-structs, 13 sub-struct pointers, maybe to lists,
                  an array of ints for fd's, and another for a pair of pipe
                  20 individual int fields,  6 fixed length string
                  arrays, and I may have under-counted some of these.
                  In Asterisk, sometimes, you have to lock the channel
struct before
                  you get the subfield lock.  At any time, someone might
hang up,
                  and several channels may blink out of existence.  To keep
                  opaque, there are over 200 get/set functions.  I also see
                  channels are outfitted with ref counts, so all you have
to do, to  insure
                  your channel pointer doesn't get destroyed while you hold
it, you bump
                  its refcount, and it will remain alive until you
decrement the count.
   Objects can be hierarchical... So, maybe, you need
   to NOT allow different threads to directly access the sub-objects. So,
   ask the Parent object to change the sub-objects, be it to modify an
   in a list, or add or remove elements from a list. If you have an object,
   like an Asterisk Channel structure, instead of handing out pointers to
the subojbects,
   and locking on everything, you make update requests to the channel object
   instance instead. Hmmm. That would mean a pretty big set of add/mod/del
   requests, and a fairly huge variety of data objects being passed in and
fetched out.
   It looks like Asterisk has been working to make channels opaque over the
    With locks, you wait around until the change is made, so you can remove
the lock.
    How often do you actually need to have the change requested,
accomplished, before
    continuing? Hardly ever, or almost always?
d. That's it. I'm still looking for the magic bullets to help architect
large and complex
    multithreaded projects. Does anyone have any other guidelines?


On Thu, Aug 28, 2014 at 9:30 AM, Trevor Bernard <trevor.bernard at gmail.com>

> Murf,
> It depends on how you phrase the problem -- If you only ever process
> the messages you receive on a single thread and in order, you will
> never deadlock because you don't have the necessary conditions to
> deadlock, which are shared state and more than one thread
> accessing/modifying it concurrently. This is how "actors" work. The
> code you write to handle the messages is thread safe. You achieve
> parallelism by distributing the work out to multiple worker processes
> via message passing.
> On Thu, Aug 28, 2014 at 11:58 AM, Steve Murphy <murf at parsetree.com> wrote:
> > Pieter--
> >
> > Last year, I read the book, Programming Erlang, by Joe Armstrong, and I
> was
> > fascinated by the ideology behind the general thread-per-object approach,
> > where
> > each "object" is managed by its own thread, via message passing.
> >
> > Erlang has a really "involved" message passing scheme, involving pattern
> > matching, a "mailbox", recieve timer, a "save queue". Needless to say,
> all
> > this makes a very powerful way of prioritizing messages, so a busy object
> > manager can pull high-priority requests from the mailbox and act on them
> > immediately, saving lower priority requests for later.
> >
> > I see in a paper at http://zeromq.org/blog:multithreading-magic, the
> same
> > sort of admiration for Erlang's methodology.
> >
> > But...
> >
> > I'm not seeing the cure-all, end-all, solution to concurrency problems,
> and
> > it bothers me, because I'm probably missing something fundamental,
> something
> > I should have picked up on, but didn't.
> >
> > Erlang allows other processes/threads to drop requests in an object's
> > mailbox, but
> > it also has a mechanism for waiting until the action is complete, as the
> > "object"
> > can send a response.
> >
> > It's this response wait that is the killer. Now, I've done a lot of work
> > on/with Asterisk,
> > and it is heavily mulithreaded, in the old-school way, and has a ton of
> > critical
> > sections, and locks, and mutiple locks for a single action. They have
> > evolved
> > some fairly involved strategies to avoid deadlocks, including putting a
> > timer
> > on the lock, and if it times out, giving up the lock they already have,
> and
> > starting
> > over, allowing the contending party to obtain the lock they need, finish
> > their
> > "thing", which allows you to continue and obtain the lock you need to do
> > your
> > "thing". And on and on it goes.
> >
> > Now, I didn't go and look up the particulars about "N-party dance", etc.,
> > but
> > the classic resource deadlock situations still seem
> > in play when you have to wait for completion. A asks B to
> > complete a task, and waits for him to respond. In order to get that
> done, B
> > requests
> > A for something, and waits forever for A to finish. And so on. Perhaps C
> or
> > even D
> > are also involved.
> > I keep thinking that such basic situations
> > aren't solved by
> > switching to the Erlang methods. There must be some architectural,
> perhaps
> > hierarchical organizing,
> > some sort of general design practice, that can
> > overcome these kinds of problems, I'm just blind to it at the moment.
> >
> > Situations like 'atomic' changes on two or more objects at once, etc.
> and I
> > don't
> > see in the fog, how Erlang solves these problems in general. Can someone
> > point me to some literature that might make things clear?
> >
> > murf
> >
> > --
> >
> > Steve Murphy
> > ParseTree Corporation
> > 57 Lane 17
> > Cody, WY 82414
> > ✉  murf at parsetree dot com
> > ☎ 307-899-5535
> >
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Steve Murphy
ParseTree Corporation
57 Lane 17
Cody, WY 82414
✉  murf at parsetree dot com
☎ 307-899-5535
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140828/b7c498eb/attachment.html>

More information about the zeromq-dev mailing list