Recently I watched a fantastic talk from Saša Jurić, which goes through a lot of the internals of the how the BEAM VM works. If you haven’t watched it, I definitely recommend checking it out.

Anyway, in this talk, he shows a method for quickly finding resource-hogging processes in a live environment, all without bringing the system down.

Let’s say you have a function that involves some sort of recursion, and somehow has ended up in an infinite loop of sorts.

In a live system, this is obviously bad and can result in the process crashing.

A simple way to:

a) Figure which process is causing this b) Figure out what the hell is going on with the process

1iex> Process.list
2|> Enum.map(&{Process.info(&1, :reductions), &1})
3|> Enum.take(5)
4# => {{:reductions, 301203128421}, #PID<0.82.0>}
5# => {{:reductions, 292424}, #PID<0.32.0>}
6# => {{:reductions, 2345}, #PID<0.24.0>}
7# => {{:reductions, 232}, #PID<0.62.0>}
8# => {{:reductions, 123}, #PID<0.58.0>}

Here we can clearly tell that #PID<0.82.0> has gone through quite a few reductions (method calls). This is pretty obviously our culprit process. Let’s do some more digging and see what’s going on here:

1# pid = #PID<0.82.0>
2iex> :dbg.tracer
3iex> :dbg.p(pid, [:call]) # trace calls
4iex> :dbg.tpl(:_, []); :timer.sleep(1000); :dbg.stop();
5
6# => call 'Elixir.Module.Function' :method(arg1, arg2..)
7# => call 'Elixir.Module.Function' :method(arg1+2, arg2..)

Now we can see exactly what is going on in this process and which functions are being called (and with what arguments).

Hopefully this is enough information for us to go away and fix the bug - but unfortunately, this process is still alive and recursing, so we need to take care of that first.

1iex> Process.exit(pid, :kill) # similar to kill -9

Now ideally you wouldn’t actually want to have to do this in a live environment however, it’s nice to know the ability is there!