Andrew Timberlake Andrew Timberlake

Hi, I’m Andrew, a programer and entrepreneur from South Africa, founder of Sitesure for monitoring websites, APIs, and background jobs.
Thanks for visiting and reading.


Pause tests in Ember

Simply await on a promise which resolves after a timeout.

test("my test", async function(assert) {
  // setup…
  await new Promise(resolve => setTimeout(resolve, 30000));
  // …assert
});
28 May 2018

Repo.count in Ecto

I have often wanted to just do the following but Ecto’s Repo module doesn’t have a count method.

iex> MyApp.Repo.count(MyApp.Account)
42

It is not too difficult to create a count function that will allow you to count the results of any query.

defmodule MyApp.DBUtils do
  import Ecto.Query, only: [from: 2]

  @doc "Generate a select count(id) on any query"
  def count(query),
    do: from t in clean_query_for_count(query), select: count(t.id)

  # Remove the select field from the query if it exists
  defp clean_query_for_count(query),
    do: Ecto.Query.exclude(query, :select)
end

This will provide a shortcut for counting any query

MyApp.DBUtils.count(MyApp.Account) |> Repo.one!

Now, to enable Repo.count we can modify the repo module usually found in lib/my_app/repo.ex

defmodule MyApp.Repo do
  use Ecto.Repo, otp_app: :my_app

  def count(query),
    do: MyApp.DBUtils.count(query) |> __MODULE__.one!
end

That’s it. This will enable a count on any query including complicated queries and those that have a select expression set.

19 Sep 2016

Benchmarking in Elixir

Appending to a list in Elixir ([1] ++ [2]) is slower than prepending and reversing [ 2 | [1] ] |> Enum.reverse but how bad is it?

Start by creating a new project, mix new benchmarking and add benchfella as a dependency in your mix.exs file

defp deps do
  [{:benchfella, "~> 0.3.2"}]
end

and run mix deps.get

Benchfella benchmarks work similarly to tests. Create a directory named bench and then create a file ending in _bench.exs. Benchfella will find these files and run them.

Create a file bench/list_append_bench.exs We will write our functions in the bench file but you can reference functions in another module to benchmark your project code.

This benchmark will test three different ways to build a list, (1) append each element to the list using ++, (2) build up the list using a recursive tail where the element is added to the head but the tail is built up recursively, and (3) prepending the element to a list accumulator and then reversing the list at the end.

defmodule ListAppendBench do
  use Benchfella

  @length 1_000

  # First bench mark
  bench "list1 ++ list2" do
    build_list_append(1, @length)
  end

  # Second bench mark
  bench "[head | recurse ]" do
    build_list_recursive_tail(1, @length)
  end

  # Third bench mark
  bench "[head | tail] + Enum.reverse" do
    build_list_prepend(1, @length)
  end

  @doc """
  Build a list of numbers from `num` to `total` by appending each item
  to the end of the list
  """
  def build_list_append(num, total, acc \\ [])
  def build_list_append(total, total, acc), do: acc
  def build_list_append(num, total, acc) do
    acc = acc ++ [num]
    next_num = num + 1
    build_list_append(next_num, total, acc)
  end

  @doc """
  Build a list of numbers from `num` to `total` by building
  the list with a recursive tail instead of using an accumulator
  """
  def build_list_recursive_tail(total, total), do: []
  def build_list_recursive_tail(num, total) do
    [ num | build_list_recursive_tail(num + 1, total) ]
  end

  @doc """
  Build a list of numbers from `num` to `total` by prepending each item
  and reversing the list at the end
  """
  def build_list_prepend(num, total, acc \\ [])
  def build_list_prepend(total, total, acc), do: Enum.reverse(acc)
  def build_list_prepend(num, total, acc) do
    acc = [num | acc]
    next_num = num + 1
    build_list_prepend(next_num, total, acc)
  end
end

Run the benchmark with mix bench and you see the results,

Settings:
  duration:      1.0 s

## ListAppendBench
[10:15:32] 1/3: list1 ++ list2
[10:15:34] 2/3: [head | tail] + Enum.reverse
[10:15:37] 3/3: [head | recurse ]

Finished in 6.66 seconds

## ListAppendBench
[head | tail] + Enum.reverse      100000   20.87 µs/op
[head | recurse ]                 100000   21.25 µs/op
list1 ++ list2                       500   3228.16 µs/op

The results: prepending to a list and reversing it is 200 times faster than appending and only fractionally faster than building the tail recursively.

For more complex benchmarks, Benchfella has various hooks for test setup and teardown. It also has ability to compare benchmark runs with mix bench.cmp and graph the results with mix bench.graph.

28 Mar 2016

Install from source using Ansible

TL;DR, All the code can be found here

Sometimes, when you want complete control, you want to be able to install packages from source and still use an automated tool like Ansible to do that.

A simple set of tasks can check for the existence of files to eliminate the need for running tasks that are already complete but that doesn’t help us with making sure we have the correct version installed.

I’m going to walk through creating a play that will build ruby from source. It will not do any work if ruby is already installed and is already the correct version. If not correct, it will:

A first pass can be found in this gist If repeated, this build will re-download the archive, extract it, configure it and make it. It won’t install the binary again because it checks for the existence of the file /usr/local/bin/ruby but other than that, all tasks will re-run.

The first step is to create a task that will determine the installed ruby version if present.

- name: Get installed ruby version
  command: ruby --version  # Run this command
  ignore_errors: true  # We don’t want and error in this command to cause the task to fail
  changed_when: false
  failed_when: false
  register: ruby_installed_version  # Register a variable with the result of the command

This task will run ruby --version but will silently fail if ruby is not installed. If ruby is installed, then it registers the version string in a variable named ruby_installed_version.

The next step is to create a variable we can use to test whether to build ruby or not. This is set in our global_vars to a default of false. Then add a task that will set that variable to true if the version string doesn’t match.

- name: Force install if the version numbers do not match
  set_fact:
    ruby_reinstall_from_source: true
  when: '(ruby_installed_version|success and (ruby_installed_version.stdout | regex_replace("^.*?([0-9\.]+).*$", "\\1") | version_compare(ruby_version, operator="!=")))'

Now we can add a when clause to all our other tasks. This will skip the task if ruby is correctly installed. That can be seen in this gist

The when clause checks for two things, (1) the task which checked the ruby version failed (i.e. there is no ruby installed) or (2) the ruby_reinstall_from_source variable is true (i.e. the versions don’t match).

An example task with the when clause:

- name: Download Ruby
  when: ruby_installed_version|failed or ruby_reinstall_from_source
  get_url:
    url: "https://cache.ruby-lang.org/pub/ruby/2.3/ruby-{{ruby_version}}.tar.gz"
    dest: "/tmp/ruby-{{ruby_version}}.tar.gz"
    sha256sum: "{{ruby_sha256sum}}"

  # …

We now have a conditional on every test. That seems a bit redundant. This can be improved by using the block syntax. By using a block we can check the condition once, and then run or skip the whole installation in one move.

- when: ruby_installed_version|failed or ruby_reinstall_from_source
  block:
    - name: Download Ruby
      when: ruby_installed_version|failed or ruby_reinstall_from_source
      get_url:
        url: "https://cache.ruby-lang.org/pub/ruby/2.3/ruby-{{ruby_version}}.tar.gz"
        dest: "/tmp/ruby-{{ruby_version}}.tar.gz"
        sha256sum: "{{ruby_sha256sum}}"

    # …

The final code can be found in this gist, https://gist.github.com/andrewtimberlake/802bd8d285b3e18c5ebe, where you can walk through the three revisions as outlined in the article.

21 Mar 2016

Using Dead Man's Snitch with Whenever

A quick tip to make it easier to use Dead Man's Snitch with the whenever gem

Whenever is a great gem for managing cron jobs. Dead Man’s Snitch is a fantastic and useful tool for making sure those cron jobs actually run when they should.

Whenever includes a number of predefined job types which can be overridden to include snitch support.

The job_type command allows you to register a job type. It takes a name and a string representing the command. Within the command string, anything that begins with : is replaced with the value from the jobs options hash. Sounds complicated but is in fact quite easy.

Include the whenever gem in your Gemfile and then run

$ bundle exec wheneverize

This will create a file, config/schedule.rb. Insert these lines at the top of your config file, I have mine just below set :output.

These lines add && curl https://nosnch.in/:snitch to each job type just before :output.

job_type :command,   "cd :path && :task && curl https://nosnch.in/:snitch :output"
job_type :rake,      "cd :path && :environment_variable=:environment bin/rake :task --silent && curl https://nosnch.in/:snitch :output"
job_type :runner,    "cd :path && bin/rails runner -e :environment ':task' && curl https://nosnch.in/:snitch :output"
job_type :script,    "cd :path && :environment_variable=:environment bundle exec script/:task && curl https://nosnch.in/:snitch :output"

Now add your job to the schedule. A simple rake task would like this:

every 1.day, roles: [:app] do
  rake "log:clear"
end

Now it’s time to create the snitch. You can grab a free account at deadmanssnitch.com and add a new snitch.

New Snitch

Then, once that’s saved, you’ll see a screen with your snitch URL. All you need to do is copy the hex code at the end.

Snitch URL

Use that hex code in your whenever job as follows:

every 1.day, roles: [:app] do
  rake "log:clear", snitch: "06ebef375f"
end

Now deploy and update your whenverized cron job. DMS will let you know as soon as your job runs for the first time so you know it has begun to work. After that, they’ll only let you know if it fails to check in.

Tip: For best tracking, you want your DMS job to check in just before the end of the period you’re monitoring (in the above example 1 day). To do that, I revert to cron syntax in whenever and set my job up as:

# Assuming your server time zone is set to UTC
every "59 23 * * *", roles: [:app] do
  rake "log:clear", snitch: "06ebef375f"
end

See Does it matter when I ping a snitch?. Remember to allow time for the job to run and complete. For more information, read through the full DMS FAQ

6 Sep 2015

Cleaning up a Ruby hash

I’ve found a number of times where I have needed to iterate over a hash and modify the values. The most recent was stripping excess spaces from the values of a Rails params hash.

The only way I know of doing this is:

hash = {one: "  one  ", two: "two  "}
hash.each do |key, value|
  hash[key] = value.strip!
end
#=> {:one=>“one”, :two=>“two”}

This is a lot less elegant than using map on an Array

["  one  ", "two  "].map(&:strip!)
#=> ["one", "two"]

I wanted something like #map for a Hash

So I came up with Hash#clean (this is a monkey patch so exercise with caution)

class Hash
  def clean(&block)
    each { |key, value|
      self[key] = yield(value)
    }
  end
end

Now it’s as easy as,

{one: "  one  ", two: "two  "}.clean(&:strip!)
#=> {:one=>"one", :two=>"two"}

Now I can easily sanitise Rails parameter hashes

def model_params
  params.require(:model).permit(:name, :email, :phone).clean(&:strip!)
end
30 Aug 2015

Video: Product strategy is about saying no

I quickly drew out the graph from the video on determining great feature fit. What you're looking for is features that will be used by all your users all of the time.

30 Jul 2015

Watch YouTube videos at full window (not full screen)

I use a large 27" iMac which I divide up windows with a browser in the top right of the screen. One thing that often frustrated me is that I could not maximise a video to fill the window completely. I had to fill my entire screen or watch it in the embedded size.

It turns out this is not too hard, change the URL in the browser from https://www.youtube.com/watch?v=oHg5SJYRHA0 to https://www.youtube.com/embed/oHg5SJYRHA0

16 Jul 2015

Skipping blank lines in ruby CSV parsing

I recently had an import job failing because it took too long. When I had a look at the file I saw that there were 74 useful lines but a total of 1,044,618 lines in the file (My guess is MS Excel having a little fun with us).

Most of the lines were simply rows of commas:

Row,Of,Headers
some,valid,data
,,
,,
,,
,,
,,

The CSV library has an option named skip_blanks but the documentation says “Note that this setting will not skip rows that contain column separators, even if the rows contain no actual data”, so that’s not actually helpful in this case.

What is needed is skip_lines with a regular expression that will match any lines with just column separators (/^(?:,\s*)+$/). The resulting code looks like this:

require 'csv'
CSV.foreach('/tmp/tmp.csv',
            headers: true,
            skip_blanks: true,
            skip_lines: /^(?:,\s*)+$/) do |row|
  puts row.inspect
end

#<CSV::Row "Row":"some" "Of":"valid" "Headers":"data">
#=> nil
12 Jul 2015

Append items to a sorted collection in Backbone.js

I won’t cover all the boiler plate code but you can view that at JSFiddle The project is a ListItem model and a corresponding ListCollection. There is a ListItemView which is compiled into a ListView to create an ordered list. There is a FormView used for adding items to the collection.

The first component of our code is the comparator in the collection which keeps the list sorted by name.

var ListCollection = Backbone.Collection.extend({
  model: ListItem,
  comparator: function(item) {
    return item.get('name').toLowerCase();
  }
});

With this a simple render method will always have the list in order but it needs to redraw the list every time the collection is updated. Simply bind the add event to this.render and you’re done.

//...
  initialize: function() {
    this.listenTo(this.collection, 'add', this.render);
  },
  render: function() {
    var items = [];
    this.collection.each(function(item) {
      items.push((new ListItemView({model: item})).render().el);
    });
    this.$el.html(items);
    return this;
  }
//...

What if we have a list that is more complicated or we want to display the item being added. For this we need a couple of things.

  1. Split the creation of the item view out into its own factory method
  2. Call the factory method when building the initial list within render
  3. Create a new addItem method which will append the item to the list
  4. Change our event binding to this.addItem
//...
  initialize: function() {
    this.listenTo(this.collection, 'add', this.addItem);
  },
  render: function() {
    var self = this;
    var items = [];
    this.collection.each(function(item) {
      items.push(self.buildItemView(item).render().el);
    });
    this.$el.html(items);
    return this;
  },
  addItem: function(item) {
    var $view = this.buildItemView(item).render().$el;
    this.$el.append($view.hide().fadeIn());
  },
  buildItemView: function(item) {
    return new ListItemView({model: item});
  }
//...

The problem now is that we’re using jQuery’s append which adds the item view to the end of the list negating the work of the comparator in our Backbone collection. What we need now is a way to insert the new item into the list at the correct index. For that we’ll need at add an insertAt method to jQuery. This new method will take an index and an element and it will place it into the childNodes collection at the correct index.

$.fn.extend({
  insertAt: function(index, element) {
    var lastIndex = this.children().size();
    if(index < lastIndex) {
      this.children().eq(index).before(element);
    } else {
      this.append(element);
    }
    return this;
  }
});

Now we can update our addItem method to calculate the index of the new item and then add it into the list at that index.

//...
  addItem: function(item) {
    // Get the index of the newly added item
    var index = this.collection.indexOf(item);
    // Build a view for the item
    var $view = this.buildItemView(item).render().$el;
    // Insert the view at the same index in the list
    this.$el.insertAt(index, $view.hide().fadeIn());
  }
//...

The final working product is embedded here:

29 Jun 2015

Next page