Benchmarking in Elixir

Appending to a list in Elixir ([1] ++ [2]) is slower than prepending and reversing [ 2 | [1] ] |> Enum.reverse but how bad is it?

Start by creating a new project, mix new benchmarking and add benchfella as a dependency in your mix.exs file

defp deps do
  [{:benchfella, "~> 0.3.2"}]
end

and run mix deps.get

Benchfella benchmarks work similarly to tests. Create a directory named bench and then create a file ending in _bench.exs. Benchfella will find these files and run them.

Create a file bench/list_append_bench.exs We will write our functions in the bench file but you can reference functions in another module to benchmark your project code.

This benchmark will test three different ways to build a list, (1) append each element to the list using ++, (2) build up the list using a recursive tail where the element is added to the head but the tail is built up recursively, and (3) prepending the element to a list accumulator and then reversing the list at the end.

defmodule ListAppendBench do
  use Benchfella

  @length 1_000

  # First bench mark
  bench "list1 ++ list2" do
    build_list_append(1, @length)
  end

  # Second bench mark
  bench "[head | recurse ]" do
    build_list_recursive_tail(1, @length)
  end

  # Third bench mark
  bench "[head | tail] + Enum.reverse" do
    build_list_prepend(1, @length)
  end

  @doc """
  Build a list of numbers from `num` to `total` by appending each item
  to the end of the list
  """
  def build_list_append(num, total, acc \\ [])
  def build_list_append(total, total, acc), do: acc
  def build_list_append(num, total, acc) do
    acc = acc ++ [num]
    next_num = num + 1
    build_list_append(next_num, total, acc)
  end

  @doc """
  Build a list of numbers from `num` to `total` by building
  the list with a recursive tail instead of using an accumulator
  """
  def build_list_recursive_tail(total, total), do: []
  def build_list_recursive_tail(num, total) do
    [ num | build_list_recursive_tail(num + 1, total) ]
  end

  @doc """
  Build a list of numbers from `num` to `total` by prepending each item
  and reversing the list at the end
  """
  def build_list_prepend(num, total, acc \\ [])
  def build_list_prepend(total, total, acc), do: Enum.reverse(acc)
  def build_list_prepend(num, total, acc) do
    acc = [num | acc]
    next_num = num + 1
    build_list_prepend(next_num, total, acc)
  end
end

Run the benchmark with mix bench and you see the results,

Settings:
  duration:      1.0 s

## ListAppendBench
[10:15:32] 1/3: list1 ++ list2
[10:15:34] 2/3: [head | tail] + Enum.reverse
[10:15:37] 3/3: [head | recurse ]

Finished in 6.66 seconds

## ListAppendBench
[head | tail] + Enum.reverse      100000   20.87 µs/op
[head | recurse ]                 100000   21.25 µs/op
list1 ++ list2                       500   3228.16 µs/op

The results: prepending to a list and reversing it is 200 times faster than appending and only fractionally faster than building the tail recursively.

For more complex benchmarks, Benchfella has various hooks for test setup and teardown. It also has ability to compare benchmark runs with mix bench.cmp and graph the results with mix bench.graph.