Me (Andrew Timberlake)

Andrew Timberlake

Recent Posts

Extending the timeout for Rails processes through Passenger on Nginx

I recently had to allow for a long running reporting task in a Rails app which is running under Passenger (mod_rails) on Nginx The default setting is 60 seconds and setting send_timeout to 300 in the Nginx config wasn't working. After much searching, I found the answer. Edit /ext/nginx/Conguration.c and change the lines:
ngx_conf_merge_msec_value(conf->upstream.send_timeout, prev->upstream.send_timeout, 60000);
ngx_conf_merge_msec_value(conf->upstream.read_timeout, prev->upstream.read_timeout, 60000);
to (for a 5 min timeout)
ngx_conf_merge_msec_value(conf->upstream.send_timeout, prev->upstream.send_timeout, 300000);
ngx_conf_merge_msec_value(conf->upstream.read_timeout, prev->upstream.read_timeout, 300000);
You will then need to recompile nginx and restart it (sending the HUP signal won't work in this case) Hope this helps someone else out there.

Rails plugin that provides rake tasks for identifying missing specs

I have a bad habbit of generating my views while specing the controller and then never generating a view spec. I wrote a rake task to tell me which views have no specs, kind of like a rcov for views. I then realised that it could be useful to find all files with missing specs so now we have unspec'd, a Rails plugin that provides rake tasks for identifying missing specs

How to select an array of values in Rails

I recently wanted to get a list of IDs from rails, but wanted them in an array. First try is to use :select => 'id' in:
Model.find(:all, :select => 'id')
but that creates a model for each id In order to get an array of values, you can use select_values:
Model.connection.select_values('select id from models')
This is great, but what if you have a more complex query and you still want to use the query building in Rails finder methods. You can use construct_finder_sql. This is a private method so it needs to be called via send and you should be cautious about it's availability in future versions of Rails. The final query then is:
sql = Model.send(:construct_finder_sql, :select => 'id', :conditions => [...])
Model.connection.select_values(sql)
This will be quicker and use less memory because it doesn't create a model object for each id. I would only use this when pulling fairly large arrays of values, stick to the normal way of doing things for smaller collections. If you want to get two or three values per 'row', you can use select_rows which will return an array of arrays.

Accessing the Rails logger inside of library files

A quick tip, if you're developing a library which lives inside of RAILS_ROOT/lib, you can use the Rails logger via DEFAULT_RAILS_LOGGER
class MyLib
  def my_method
    RAILS_DEFAULT_LOGGER.info "my log message" if RAILS_DEFAULT_LOGGER
  end
end
I added if RAILS_DEFAULT_LOGGER so that it doesn't trip up if used outside of RAILS

Download a large amount of data in CSV from Rails

I've just had to re-write an export script because the application now needed to be able to download 14000 records in CSV format. Originally I was reading all the records (with multiple joins) and then creating a csv string and then writing it to the response using send_data.

The original code

This is a simplified version of my code
accounts = Account.find(:all)
csv_string = FasterCSV.generate do |csv|
  accounts.each do |account|
    csv << [account.id, account.name]
  end
end
send_data csv_string, :filename => 'accounts.csv', 
                      :type => 'text/csv', 
                      :disposition => 'attachment'
My first thought was to stream the data using
send_file_headers!  :filename => 'accounts.csv', 
                    :type => 'text/csv', 
                    :disposition => 'attachment'
render :text => Proc.new {|response, output|
  ...
}
The problem with this is that send_file_headers! requires that the :length option be set. I also was running into the problem of loading 14000 records into memory before iterating over them. For the memory problem, I considered using the new Model.find_in_batches but I needed to join tables and use some complicated conditions.

The solution

I finally settled on chunked reading of the data from the database and a more explicit use of render :text => Proc...
require 'fastercsv'
def download
  filename = 'accounts.csv'
  headers.merge!(
    'Content-Type' => 'text/csv',
    'Content-Disposition' => "attachment; filename=\"#{filename}\"",
    'Content-Transfer-Encoding' => 'binary'
  )
  @performed_render = false

  render :status => 200, :text => Proc.new { |response, output|
    headings = ["ID", "Name"]
    output.write FasterCSV.generate_line(headings)

    last_account_id = 0
    while last_account_id do
      accounts = Account.find(:all,
                              :conditions => ["accounts.id > ?", last_account_id],
                              :order => 'accounts.id',
                              :limit => 1000)

      last_account_id = accounts.size > 0 ? accounts[-1].id : nil

      accounts.each { |account|
        data = [account.id, account.name]
        output.write FasterCSV.generate_line(data)
      }
    end
  }
end
I hope this useful for you, if you have any questions, please ask in the comments.

Map Fields - A Rails plugin to ease the importing of CSV files

I've just released a plugin, map-fieilds, that eases the importing of CSV files. When importing CSV files for a project, I wanted to add flexibility for the users so that they could import their CSV files in a looser format and then map their format to the format I needed. map-fieilds will intercept calls to a method and show an intermediate screen where the user can map their columns to the expected columns.

How to install

sudo gem install map-fields
In your environment.rb file:
config.gem 'map-fields', :version => '~> 0.1.0', :lib => 'map_fields'
If you prefer, it can be installed as a plugin:
script/plugin install git://github.com/internuity/map-fields.git

Using it in your controller

lists_controller.rb:
class ListsController < AppliactionController
  map_fields :create, 
                    ['Title', 'First name', 'Last name'], 
                    :file_field => :file, 
                    :params => [:list]

  def index
    @lists = List.find(:all)
  end

  def new
    @list = List.new
  end

  def create
    @list = List.new(params[:list])
    if fields_mapped?
      mapped_fields.each do |row|
        @list.contact.create(:title => row[0],
                             :first_name => row[1],
                             :last_name => row[2])
      end
      flash[:notice] = 'Contact list created'
      redirect_to :action => :index
    else
      render
    end
  rescue MapFields::InconsistentStateError
    flash[:error] = 'Please try again'
    redirect_to :action => :new
  rescue MapFields::MissingFileContentsError
    flash[:error] = 'Please upload a file'
    redirect_to :action => :new
  end
end

Explanation

Setup map-fields

Setup map-fields at the top of the controller. map_fields accepts three parameters
  1. The first is the method to intercept, in this case :create
  2. The second is an array of expected CSV fields. The order of the fields in this array is the order they will be available in the row object when finally reading the CSV file.
  3. Lastly, a hash of options.
    :file_field is the field that contains the import file. This is :file by default
    :params is an array of parameters you want preserved. If you have a form based around a model, you just need to put the model name here and all the sub-fields will be preserved.
    So, if you have a form with list[:name] etc, use :params => [:list]

Create your new view with a file field

You can now setup your new view as normal with an included file field map-fields_new

Handle the mapping in your create action

The create action now has to perform two functions, the mapping and then the final creating. You can call the fields_mapped? method to see if the mapping has been performed and if not, render the mapping view. There is a mapping partial which you can use so your view is as easy as:
#create.html.erb
<%= render :partial => 'map_fields/map_fields' %>
and it produces the following: map_fields-create When the fields have been mapped, you can iterate through them with:
mapped_fields.each do |row|
  # row.number returns the number of the row in the original CSV file
  # row[0] is the first mapped field, in this case Title
  # row[1] is the second mapped field, in this case First name
  # row[2] is the third mapped field etc...
end

Two errors can be raised:

  1. MapFields::InconsistentStateError is raised when map-fields is unable to determine whether a file is being uploaded or mapped. It can be experienced through a combination of using the back button and refreshes but is seldom experienced.
  2. MapFields::MissingFileContentsError is raised when no file has been uploaded
Please feel free to ask any questions in the comments and raise issues on the GitHub page.

Conditionally include development gems

I was playing with metric_fu and after having to install multiple dependency gems I realised that it was going to be a pain to put the project into production. The metric_fu instructions suggest using config.gem in your environment file but that will mean that when you push the app to production, you need to install the metric_fu gem and all it's dependencies. Considering metric_fu is only used for development, I needed a way around this. I decided to wrap the Rakefile entry in an environment check
if ['test', 'development'].include?(RAILS_ENV)
  require 'metric_fu'

  MetricFu::Configuration.run do |config|
    #define which metrics you want to use
    config.metrics  = [:churn, :saikuro, :stats, :flog, :flay, :reek, :roodi, :rcov]
    config.flay     = { :dirs_to_flay => ['app', 'lib']  }
    config.flog     = { :dirs_to_flog => ['app', 'lib']  }
    config.reek     = { :dirs_to_reek => ['app', 'lib']  }
    config.roodi    = { :dirs_to_roodi => ['app', 'lib'] }
    config.saikuro  = { :output_directory => 'scratch_directory/saikuro',
                        :input_directory => ['app', 'lib'],
                        :cyclo => "",
                        :filter_cyclo => "0",
                        :warn_cyclo => "5",
                        :error_cyclo => "7",
                        :formater => "text"} #this needs to be set to "text"
    config.churn    = { :start_date => "1 year ago", :minimum_churn_count => 10}
    config.rcov     = { :test_files => ['test/**/*_test.rb',
                                        'spec/**/*_spec.rb'],
                        :rcov_opts => ["--sort coverage",
                                       "--no-html",
                                       "--text-coverage",
                                       "--no-color",
                                       "--profile",
                                       "--rails",
                                       "--exclude /gems/,/Library/,spec"]}
  end
end
This allows metric_fu to run in either test or development and won't even try to load the gem in production.

Easily compare a variable with multiple values

I often forget this simple little trick for comparing multiple values against a single variable. Instead of
var = 2
if var == 1 || var == 2 || var == 3
  puts "yes"
end
#=> "yes"
You can do the following
var = 2
if [1,2,3].include?(var)
  puts "yes"
end
#=> "yes"

Setting the session base domain in Rails 2.3

Prior to Rails 2.3, you could set the session base domain as follows:
ActionController::Base.session_options[:session_domain] = '.example.com'
In Rails 2.3, this needs to change to:
config.action_controller.session[:domain] = '.example.com'
I usually set this in my config/environments/.rb configuration files but it could be set in config/environment.rb if you want it to apply across configurations.

Background

This setting will allow for a shared session across multiple subdomains. This is useful where you want a user to login at www.example.com and then be able to remain logged in when accessing user.example.com

Using Webrat's save_and_open_page on Linux

Mmmm, this year has flown and I have done hardly any blogging, despite my intention to blog at least once a week. Oh, well. My excuse is working extra hard to cover a family holiday and the upcoming trip to Scottland on Rails. The best way to get going on something is to just do it so here is a quick post on how to view the page you're working on in Webrat with it's save page feature when on Linux (It is currently only works on Windows and Mac) A quick monkey patch in your test helper or Cucumber env file:
module Webrat::SaveAndOpenPage
  alias_method :old_open_in_browser, :open_in_browser
  def open_in_browser(path)
    if ruby_platform =~ /linux/i
      `firefox #{path}`
    else
      old_open_in_browser path
    end
  end
end
This is not perfect because it is specific to using Firefox, but I know most of you use Firefox anyway, and at least it will continue to work for those using Windows or Mac Now let's hope I can get my work load under control and post more frequently.