Streamlined Erlang RPC Server
While working on cloudant-dbcore, we noticed varying performance characteristics of code that used distributed Erlang to spawn processes on remote nodes. Some of our code was pretty snappy, while profiling revealed that others were orders of magnitude slower while essentially attempting the same basic function. So we took a look and found some optimizations that led to impressive speedups for remote operations and a new open-source project we’d like to share.
Upon closer examination, part of the problem was the use of rpc:call/4 or /5. ‘rex’ is the RPC server that ships with Erlang/OTP. When looking at its source code, we found that rpc:call does a spawn_monitor for another middleman process which makes a gen_server:call to {rex, Node}, then receives the Result, and exit()s with it. On the remote node, the rex server spawn_monitors a local process to do the MFA and report the result to rex.
So, why did we write our own RPC server, ‘rexi’? The advantage of not using rex as the RPC server is that we get to customize the flow more. For example, we can send async messages starting workers without spawning a local process, but we still get notifications if those processes crash. Also, minimizing the number of processes involved helped us with CPU utilization, since we are not copying data from process to process all over the place.
Here’s an example of how to use rexi.
Check out the rexi repo on Github and take it for a spin. Let us know what you think.
