I recently wanted to get a list of IDs from rails, but wanted them in an array.
First try is to use :select => 'id' in:
Model.find(:all, :select => 'id')
but that creates a model for each id
In order to get an array of values, you can use select_values:
Model.connection.select_values('select id from models')
This is great, but what if you have a more complex query and you still want to use the query building in Rails finder methods.
You can use construct_finder_sql. This is a private method so it needs to be called via send and you should be cautious about it's availability in future versions of Rails.
The final query then is:
sql = Model.send(:construct_finder_sql, :select => 'id', :conditions => [...])
Model.connection.select_values(sql)
This will be quicker and use less memory because it doesn't create a model object for each id.
I would only use this when pulling fairly large arrays of values, stick to the normal way of doing things for smaller collections.
If you want to get two or three values per 'row', you can use select_rows which will return an array of arrays.
A quick tip, if you're developing a library which lives inside of RAILS_ROOT/lib, you can use the Rails logger via DEFAULT_RAILS_LOGGER
class MyLib
def my_method
RAILS_DEFAULT_LOGGER.info "my log message" if RAILS_DEFAULT_LOGGER
end
end
I added if RAILS_DEFAULT_LOGGER so that it doesn't trip up if used outside of RAILS
I've just had to re-write an export script because the application now needed to be able to download 14000 records in CSV format.
Originally I was reading all the records (with multiple joins) and then creating a csv string and then writing it to the response using send_data.
The original code
This is a simplified version of my code
accounts = Account.find(:all)
csv_string = FasterCSV.generate do |csv|
accounts.each do |account|
csv << [account.id, account.name]
end
end
send_data csv_string, :filename => 'accounts.csv',
:type => 'text/csv',
:disposition => 'attachment'
My first thought was to stream the data using
send_file_headers! :filename => 'accounts.csv',
:type => 'text/csv',
:disposition => 'attachment'
render :text => Proc.new {|response, output|
...
}
The problem with this is that send_file_headers! requires that the :length option be set. I also was running into the problem of loading 14000 records into memory before iterating over them.
For the memory problem, I considered using the new Model.find_in_batches but I needed to join tables and use some complicated conditions.
The solution
I finally settled on chunked reading of the data from the database and a more explicit use of render :text => Proc...
require 'fastercsv'
def download
filename = 'accounts.csv'
headers.merge!(
'Content-Type' => 'text/csv',
'Content-Disposition' => "attachment; filename=\"#{filename}\"",
'Content-Transfer-Encoding' => 'binary'
)
@performed_render = false
render :status => 200, :text => Proc.new { |response, output|
headings = ["ID", "Name"]
output.write FasterCSV.generate_line(headings)
last_account_id = 0
while last_account_id do
accounts = Account.find(:all,
:conditions => ["accounts.id > ?", last_account_id],
:order => 'accounts.id',
:limit => 1000)
last_account_id = accounts.size > 0 ? accounts[-1].id : nil
accounts.each { |account|
data = [account.id, account.name]
output.write FasterCSV.generate_line(data)
}
end
}
end
I hope this useful for you, if you have any questions, please ask in the comments.
I've just released a plugin, map-fieilds, that eases the importing of CSV files.
When importing CSV files for a project, I wanted to add flexibility for the users so that they could import their CSV files in a looser format and then map their format to the format I needed.
map-fieilds will intercept calls to a method and show an intermediate screen where the user can map their columns to the expected columns.
How to install
sudo gem install map-fields
In your environment.rb file:
config.gem 'map-fields', :version => '~> 0.1.0', :lib => 'map_fields'
If you prefer, it can be installed as a plugin:
script/plugin install git://github.com/internuity/map-fields.git
Using it in your controller
lists_controller.rb:
class ListsController < AppliactionController
map_fields :create,
['Title', 'First name', 'Last name'],
:file_field => :file,
:params => [:list]
def index
@lists = List.find(:all)
end
def new
@list = List.new
end
def create
@list = List.new(params[:list])
if fields_mapped?
mapped_fields.each do |row|
@list.contact.create(:title => row[0],
:first_name => row[1],
:last_name => row[2])
end
flash[:notice] = 'Contact list created'
redirect_to :action => :index
else
render
end
rescue MapFields::InconsistentStateError
flash[:error] = 'Please try again'
redirect_to :action => :new
rescue MapFields::MissingFileContentsError
flash[:error] = 'Please upload a file'
redirect_to :action => :new
end
end
Explanation
Setup map-fields
Setup map-fields at the top of the controller.
map_fields accepts three parameters
- The first is the method to intercept, in this case :create
- The second is an array of expected CSV fields. The order of the fields in this array is the order they will be available in the row object when finally reading the CSV file.
- Lastly, a hash of options.
:file_field is the field that contains the import file. This is :file by default
:params is an array of parameters you want preserved. If you have a form based around a model, you just need to put the model name here and all the sub-fields will be preserved.
So, if you have a form with list[:name] etc, use :params => [:list]
Create your new view with a file field
You can now setup your new view as normal with an included file field
Handle the mapping in your create action
The create action now has to perform two functions, the mapping and then the final creating.
You can call the fields_mapped? method to see if the mapping has been performed and if not, render the mapping view.
There is a mapping partial which you can use so your view is as easy as:
#create.html.erb
<%= render :partial => 'map_fields/map_fields' %>
and it produces the following:
When the fields have been mapped, you can iterate through them with:
mapped_fields.each do |row|
# row.number returns the number of the row in the original CSV file
# row[0] is the first mapped field, in this case Title
# row[1] is the second mapped field, in this case First name
# row[2] is the third mapped field etc...
end
Two errors can be raised:
- MapFields::InconsistentStateError is raised when map-fields is unable to determine whether a file is being uploaded or mapped. It can be experienced through a combination of using the back button and refreshes but is seldom experienced.
- MapFields::MissingFileContentsError is raised when no file has been uploaded
Please feel free to ask any questions in the comments and raise issues on the GitHub page.
I was playing with metric_fu and after having to install multiple dependency gems I realised that it was going to be a pain to put the project into production.
The metric_fu instructions suggest using config.gem in your environment file but that will mean that when you push the app to production, you need to install the metric_fu gem and all it's dependencies. Considering metric_fu is only used for development, I needed a way around this.
I decided to wrap the Rakefile entry in an environment check
if ['test', 'development'].include?(RAILS_ENV)
require 'metric_fu'
MetricFu::Configuration.run do |config|
#define which metrics you want to use
config.metrics = [:churn, :saikuro, :stats, :flog, :flay, :reek, :roodi, :rcov]
config.flay = { :dirs_to_flay => ['app', 'lib'] }
config.flog = { :dirs_to_flog => ['app', 'lib'] }
config.reek = { :dirs_to_reek => ['app', 'lib'] }
config.roodi = { :dirs_to_roodi => ['app', 'lib'] }
config.saikuro = { :output_directory => 'scratch_directory/saikuro',
:input_directory => ['app', 'lib'],
:cyclo => "",
:filter_cyclo => "0",
:warn_cyclo => "5",
:error_cyclo => "7",
:formater => "text"} #this needs to be set to "text"
config.churn = { :start_date => "1 year ago", :minimum_churn_count => 10}
config.rcov = { :test_files => ['test/**/*_test.rb',
'spec/**/*_spec.rb'],
:rcov_opts => ["--sort coverage",
"--no-html",
"--text-coverage",
"--no-color",
"--profile",
"--rails",
"--exclude /gems/,/Library/,spec"]}
end
end
This allows metric_fu to run in either test or development and won't even try to load the gem in production.
I often forget this simple little trick for comparing multiple values against a single variable.
Instead of
var = 2
if var == 1 || var == 2 || var == 3
puts "yes"
end
#=> "yes"
You can do the following
var = 2
if [1,2,3].include?(var)
puts "yes"
end
#=> "yes"
Prior to Rails 2.3, you could set the session base domain as follows:
ActionController::Base.session_options[:session_domain] = '.example.com'
In Rails 2.3, this needs to change to:
config.action_controller.session[:domain] = '.example.com'
I usually set this in my config/environments/.rb configuration files but it could be set in config/environment.rb if you want it to apply across configurations.
Background
This setting will allow for a shared session across multiple subdomains.
This is useful where you want a user to login at www.example.com and then be able to remain logged in when accessing user.example.com
Mmmm, this year has flown and I have done hardly any blogging, despite my intention to blog at least once a week. Oh, well. My excuse is working extra hard to cover a family holiday and the upcoming trip to Scottland on Rails.
The best way to get going on something is to just do it so here is a quick post on how to view the page you're working on in Webrat with it's save page feature when on Linux (It is currently only works on Windows and Mac)
A quick monkey patch in your test helper or Cucumber env file:
module Webrat::SaveAndOpenPage
alias_method :old_open_in_browser, :open_in_browser
def open_in_browser(path)
if ruby_platform =~ /linux/i
`firefox #{path}`
else
old_open_in_browser path
end
end
end
This is not perfect because it is specific to using Firefox, but I know most of you use Firefox anyway, and at least it will continue to work for those using Windows or Mac
Now let's hope I can get my work load under control and post more frequently.
