flouri.sh - Blog

QueryReviewer is now rails 2.1 safe

2008-06-25T03:50:00Z

Thanks to a nice suggestion by the community which I pretty much used verbatim. Still works with rails 2.0 and 1.2.3. Check out the project homepage.

http://code.google.com/p/query-reviewer/

I'm working at Pivotal

2008-04-30T16:05:00Z

After a 4 month interlude at SpongeFish, I'm now working at Pivotal Labs. Pivotal's a really fun place to do agile rails consulting and I'm loving it so far...

MySQL Query Reviewer - now with AJAX and Profiling

2008-04-10T21:05:00Z

Kevin Hall and I have released a new version of the query_reviewer plugin. You should start by looking at my first post, to see what the basic premise is before reading this article. The single largest improvment is the ability to analyze the database requests of AJAX requests, which is accomplished by piggy backing javascript or HTML into the ends of AJAX responses. Here's what it currently looks like, analyzing my project on a page that does lots of database requests:

The improvements are:

View query analysis for AJAX requests
Take into account the duration of a query (if production data)
Show PROFILE information from mysql
Warnings for long key lengths (which can be bad even when you hit an index)

Lots of information is now available for a single slow query:

I'm really excited that this plugin is getting attention, and welcome feedback, suggestions, and bug reports on the Google Code project homepage.

To install it, simply visit the project homepage.

Flourish is back from the dead

2008-03-30T18:07:00Z

Things started going very badly for flouri.sh when my home server wouldn't boot up. I removed almost all the hardware, but could not even get a POST screen or any beeps. I even reseated the RAM and CPU, but no luck. Finally, I grabbed an old media PC shuttle cube and attempted the craziest idea:

The 4 x RAID5 disks are actually being powered by my old server, but the PCI raid card is inserted into my old media PC. The shuttle doesn't have enough power or room to run all those disks so I'm using the old server as a lifeboat with 4 IDE cables running between them!

Writing data migrations in rails

2008-02-18T19:11:00Z

Using ActiveRecord can be tricky, here's how

Data migrations are a major headache in every rails project I've worked on. Developers typically write straight SQL migrations which take much longer to create and test, or they use ActiveRecord and run into problems. I recent wrote a data migration which took 7 existing tables and compressed them into 4. Some of the new 4 had the same names as the existing models, so I finally figured out how to do this safely with ActiveRecord. Here's my advice:

When changing the schema of an existing set of tables, create new tables and then rename them.
Include active record fragments inside the migration class at the top of your migrations.
Put your data migrations inside transactions.
Make your data migrations completely reversible (which is a lot easier when you follow the first rule)

Here's an example of an ActiveRecord class fragment at the top of a migration:

class MarkPrimaryBits < ActiveRecord::Migration
  class LessonVersion < ActiveRecord::Base
    belongs_to :lesson
  end

  class Lesson < ActiveRecord::Base
    has_many :lesson_versions
  end
 
  def self.up
    ...

By putting the Lesson and LessonVersion classes at the top of the migration, I'm allowing myself to use those classes inside my migration and be completely independent of any changes made to the real model from then on (including the deletion of the class itself). Furthermore, I can have another migration which uses those same class names with completely different meanings and they won't conflict with each other.

Extending has_many associations correctly

2008-02-15T18:11:00Z

How to build custom methods on associations that aren't slow

The "has_many do" syntax has been widely adopted in rails, but I often see it going wrong. Consider the following association extension:

has_many :versions do
  def primary
    find(:first, :conditions => {:primary => true})
  end
end

What's wrong with this code is that the finder has to execute a database request every time it's invoked. True, rails has a query cache, but it results in more database requests then one should need. We can make it better by caching the result in an instance variable attached to the association:

has_many :versions do
  def primary
    @primary ||= find(:first, :conditions => {:primary => true})
  end
end

But what if you already have the association loaded? Inside the block, you can access various methods of the AssociationCollection and AssociationProxy classes. Notable methods are the following:

proxy_owner - the module that contains the association
proxy_reflection - the Reflection object that contains the association options (FK, :dependent, etc)
proxy_target - the cached association data, if the association has been loaded
loaded? - returns true if the association has been loaded
reset - delete the cached association data and forget it has been loaded

Using these methods, we can rewrite our method to be the most efficient possible, by making use of loaded association data when present:

has_many :versions do
  def primary
    if loaded?
      @primary ||= proxy_target.detect { |ver| ver.primary? }
    else
      @primary ||= find(:first, :conditions => {:primary => true})
    end
  end
end

Now our method is guarenteed to make only 1 database call, no matter how many times invoked. It also will make zero database calls if the entire association has already been loaded. The only downside is that "how to determine if primary" logic has to be written once in rails_sql and once in pure ruby.

MySQL query_reviewer plugin

2008-02-13T18:46:00Z

query_analyzer on steriods!

Features

QueryReviewer is an advanced SQL query analyzer. It accomplishes the following goals:

View all EXPLAIN output for all SELECT queries to generate a page
Rate a page's SQL usage into one of three categories: OK, WARNING, CRITICAL
Attach meaningful warnings to individual queries
Find out where the query was executed with a stack trace
Display advanced interactive summary on page

It accomplishes this by injecting an absolutely positioned div into your HTML markup before it's sent out of rails. View injection can be turned off, if necessary and replaced with <%= query_review_output %> somewhere in your view.

Screenshot

View larger

Project homepage

http://code.google.com/p/query-reviewer/

Installing

svn export http://query-reviewer.googlecode.com/svn/trunk/ vendor/plugins/query_reviewer

Then optionally run "rake query_reviewer:setup" and edit config/query_reviewer.yml to change the default settings.

mac.com "Web Gallery" Photos in Mephisto

2008-01-02T21:24:00Z

All the flickr functionality with better iPhoto integration!

I just finished a conversion of my existing flickr ajax mephisto plugin to use mac.com web galleries! I finally ended up paying for mac.com, because of the awesome integration with everything.

Features:

Pull photos from a specific album or your entire gallery
Uses AJAX to pull feed data, so it won't bottleneck your main page rendering
Randomly chooses X photos to display every time
A variety of supported images sizes: square, small, medium, and large
Images automatically link to Apple's awesome web gallery interface

Issues (depending on size of gallery, mine has 400 images at the time):

The RSS feed takes between 2 and 10 seconds to deliver from apple.
The downloaded XML file takes 2 to 5 seconds process.

Usage

How I used it in my site:

<p>Pictures:</p>
{% macgallery feed: http://gallery.mac.com/dsboulder/?webdav-method=truthget&feedfmt=photocastrss  count: 4  format: square  width: 75  height: 75 %} 
{% endmacgallery %}

Supported options to the "macgallery" liquid tag:

feed: REQUIRED. The url of the RSS feed to your gallery, or to a specific album.
count: OPTIONAL (default: 6). The number of images to display.
format: OPTIONAL (default: square). Also valid are "small", "medium", and "large".
width/height: OPTIONAL (default: none). If specified, will add width and height attributes to the IMG tag. Useful for resizing the square images from their 160x160 default size.

Where to get the plugin


svn checkout http://mephisto-mac-gallery-ajax.googlecode.com/svn/trunk/ vendor/plugins/mephisto_mac_gallery_ajax

Mephisto UV Syntax Highlighting Plugin

2007-12-03T16:05:00Z

After I wrote the article on hacking UV into mephisto, I made it an actual plugin. The only trick was inserting a fake codepress.rb file since mephisto requires "codepress" in 2 places.

To get up and running:

Delete the existing filtered_column_code_macro plugin.
./script/plugin install http://wush.net/svn/public/plugins/mephisto/uv_code_macro
Copy over UV's xhtml theme CSS files. See using UV as a library.
You'll have to run something like Uv.copy_files "xhtml", "public/stylesheets" from the console
Include one or more of the new stylesheets in your layout.
Use <filter:code lang="XXXX" theme="XXXX"> tags to highlight your code!

Mephisto Flickr AJAX Loader

2007-12-01T16:00:00Z

I'm tired of waiting for my blog to load because of a slow flickr feed request! I designed this new plugin by copying the flickr photostream plugin, but making it actually load the pictures and the feed through an immediate AJAX request. This tremendously increased the performance of my mephisto pages!

The plugin defines a flickr controller, which accepts the AJAX request. Unfortunately, you can't change how the photos are laid out anymore (I'm not setting up a liquid context for this ajax response). Of course, if you don't like the result of the plugin, just change it to match what you want:

class FlickrController < ActionController::Base
  def index
    result = ""
    pics = find_pictures
    pics.each do |pic|
      result << "<a href='#{pic.link}'><img src='#{pic.send(params[:format].to_sym)}' alt='#{pic.title}'></a>"
    end

    render :inline => result
  end
end

How to install and use

Uninstall the flickr photostream plugin
./script/plugin install http://wush.net/svn/public/plugins/mephisto/mephisto_flickr_ajax
Use the following tag in your liquid template:
{{flickrajaxphotostream feed:<YOUR_FEED_URL> count:<NUMBER_IMAGES> format:<[square, small, etc]>}} {{endflickrajaxphotostream}}

Originally posted on ELC's blog

Rendering views without a web request in rails

2007-11-14T04:01:00Z

Why the heck do you want to do that?

Views are very well integrated into the rails framework, but they're only typically rendered when an http request comes in. ActionMailer is the main exception, but what if you want to render a view for use in another backend application? These days, document fragments are being used everywhere, and often times I'll need rendered HTML for a use other than just sending it back to the requesting user.

A solution: instantiate a controller and view

Controllers are just objects, and so are views. We can instantiate a controller, instantiate a view, then point the view to the controller and we're ready to go. The only think we're missing is the session and the request objects, but not every view needs those. I used this once for updating a facebook profile using a backgroundrb worker:

class FakeView < ActionView::Base
  include SomeHelper
  include SomeOtherHelper
end

class FakeController < ActionController::Base
  def render_some_view
    action_view = FakeView.new(File.join(RAILS_ROOT, "app", "views"), {})
    action_view.instance_variable_set("@controller", self)
    markup = action_view.render(:partial => 'facebook/your_profile')    
  end
end

By subclasses ActionView::Base, you can mix helpers into the view class, making their methods available.

Another solution: use the test framework!

The rails TestProcess is the only place where views are rendered. If you really want to simulate a real experience when rendering a view, use the test process. First, you'll need an actual controller with an actual action you want to render in it, like this one:

class FacebookController
  before_filter :login_required
  
  def your_profile    
  end
end

Then, we create and instante a test as follows:

class FacebookTest
  include ActionController::TestProcess
  attr_reader :response
  
  def initialize
    require_dependency 'application' unless defined?(ApplicationController)
    @controller = UserScheduleEntryController.new
    @request    = ActionController::TestRequest.new
    @response   = ActionController::TestResponse.new
  end
  
  def render_your_profile(user)
    @controller.instance_variable_set("@user", user) # bypass login required
    get :your_profile
    @response
  end  
end

test = FacebookTest.new
test.render_your_profile(user)
markup = test.response.body

The require_dependency was something I threw in because backgroundrb didn't have some of the required classes loaded at that point, it may not be something you need in your application.

I originally posted this article on ELC's blog

Tremble at my fast internet speeds

2007-11-07T21:48:00Z

Ever since moving to San Francisco, I can't believe how fast internet access can be here! Check it out:

The sample above was taken at my house... middle of the day... using a cable modem with Comcast over 802.11n wireless!

Duplicate Migrations in Rails

2007-11-07T04:37:00Z

Why we need duplicate migrations

Have you ever been working on a large project, and had people check in migrations with the same numbers? It's happened to me probably no less than 10 times in the last year. In each case, the situation is recoverable, but sometimes requires a lot of manual rolling back of specific migrations on possibly several machines. Then you have to renumber all the migrations after the conflict, of course.

An even worse situation is when a project is branched and remerged. For example, you might want to branch out several complicated features from trunk for a few weeks, then bring them back when complete. Assuming you create 2 feature branches (for adding profiles and friends to your users), you could end up with something like this:



  036_modify_users_to_include_first_name.rb
  037_create_profiles.rb
  037_create_friendships.rb
  037_fix_a_bug.rb
  038_add_timestamps_to_friendships.rb
  038_modify_accounts_to_limit_length.rb
  039_modify_users_to_include_gender.rb

In the above situation, the person merging the two branches has a very difficult situation ahead. Everyone working on the project is probably on revision 37 (profiles branch), 38 (friends branch), or 39 (trunk). The safe way to proceed with traditional rails migrations is to force all machines be migrated down to 36. No new migrations can be added while the migrations are then renumbered so they range from 036 to 042. Finally, all users can update from trunk and run rake db:migrate. Of course, people often forget to migrate down, and end up stuck in the middle of a sequence of migrations that has been renumbered (I am so tired of reversing migrations by hand).

Solution: Allowing duplicate migration version numbers

In the above example, the 3 migrations numbered 37 are not dependent in any way. Because they had to be developed independently, duplicate version numbers are very rarely dependent. For this reason, we beleive that it is usually safe to create a "partial ordering" of migrations rather than an exact ordering. In this partial ordering (which can be represented as a lattice), migrations with the same version number will be run in an arbitrary order:

Since all of the dependencies in the above lattice flow downward, we can satisfy the partial ordering by running the migrations alphabetically by filename, alphabetizing them first by version number and then by class name. This will only work if we can make the assumption that when new migrations are added, they can only be dependent on those with smaller version numbers.

How the plugin works

Traditional rails schema_info table cannot hold enough information to keep track of which migrations have been run, so we need to adopt a new schema format, which we place in a new schema_infos table:

In this new schema, every record represents a migration that has been run. By traversing this table, we can get an accurate picture of the state of the system, and decide which migration to run next.

If we want to migrate to version 10, for example, we create an alphabetical listing of migrations up to and including version 10(s). Then we traverse that list in order, running "up" on migrations which have not been previously run, and inserting a record into schema_infos. Finally, we create a list of migrations with version numbers larger than 10, and run "down" on those in reverse alphabetical order, removing the entries in schema_info.

A little under the hood

Below is the main migrate function. It does exactly what is discussed in the previous section:

def migrate_with_duplicates
  migration_classes_before(@target_version).each do |(version, migration_class)|    
    next if schema_information_contains?(migration_class)
    ActiveRecord::Base.logger.info "Migrating up #{migration_class} (#{version})"
    migration_class.migrate(:up)
    insert_schema_information(migration_class)
  end
  
  migration_classes_after(@target_version).each do |(version, migration_class)|              
    next if !schema_information_contains?(migration_class)
    ActiveRecord::Base.logger.info "Migrating down #{migration_class} (#{version})"
    migration_class.migrate(:down)
    remove_schema_information(migration_class)
  end
end

What would be even better...

I've always wanted to write a migration system based on partial orderings where dependencies are explicit, and version numbers are history. Such a system would work nicely on top of the new schema_infos table format. The tricky part would be how to state the dependencies without forcing the migration author to work too hard.

Download

From the ELC plugin repository: http://wush.net/svn/public/plugins/duplicate_migrations

To install:
./script/plugin install -x http://wush.net/svn/public/plugins/duplicate_migrations
(installing automatically creates the schema_infos table and populates it, but does NOT delete your old schema_info table... don't panic!)

Originally posted on ELC's blog

Ruby gems and plugins

2007-11-05T03:30:00Z

This is complete list with link to every open source ruby gem and rails plugin I've written (updated every time I write something new).

A complete listing of ruby gems and rails plugins I've written

acts_as_solr_system
query_reviewer
Mephisto: Flickr AJAX photo feed
Mephisto: Ultraviolet code syntax highlighting
duplicate_migrations
acts_as_runnable_code_on_rails
acts_as_runnable_code
acts_as_wrapped_class
wizard_form_for
mongrel_upload_progress_db
preload_fixtures
use_db

Syntax highlighting Mephistro admin side

2007-11-04T22:44:00Z

Using CodePress to make writing blog entries easier

After I got Metphisto working with Ultraviolet syntax highlighting, I decided to make writing blog entries easier! I already had some experience with my TicTacToe project using CodePress for real-time syntax highlighting. From there, I was able to modify Mephisto to use syntax highlighting while I was writing my articles... the goal being to make the HTML easier to read and easier to write:

The first step was to install CodePress from codepress.org into the mephistro public directory. I ended up putting the css in /stylesheets, and the javascript in /javascripts... big mistake! Codepress likes to be kept all together (probably best in /public), and you shouldn't have to rewrite any of the paths in codepress.html or codepress.js, like I did.

The second step was to include codepress.js in your layout.

Then I made the following modifications to /app/views/admin/articles/_form.rhtml replacing only the dd tag that defines the "body" textarea:

<dd><%= form.text_area :body, :class => 'fat codepress html linenumbers-off', :rows => 25  %>
  <br/><%= check_box_tag "enable_codepress", "1", true, :onchange => "article_body.toggleEditor();" %>
&nbsp;  Use codepress for syntax highlighting 
  <%= hidden_field_tag "article[body_hidden]", @article.body, :id => "article_body_hidden" %>
</dd>

Just changing the class of the text_area is enough to active codepress... pretty cool huh? The checkbox is cool too, allowing you to turn codepress on/off on the fly. Unfortunately, I had to do some craziness with a hidden field. Codepress does not submit the edited data with the form, like you'd expect. Because it creates an design-mode iframe, the browser does not submit your code the way you want it to. However, it provides an object on which you can call a method to get the unformatted code. We can copy the code into the hidden field with an onsubmit in /app/views/admin/articles/new.rhtml and /app/views/admin/articles/edit.rhtml by adding an :onsubmit option to the form like so:

{:id => 'article-form', :multipart => true, 
  :onsubmit => "document.getElementById('article_body_hidden').value = article_body.getCode();"}

Of course, we've now got a hidden field named article[body_hidden] which contains the actual body, rather than the regular field article[body], so we need to deal with that somewhere. I chose /app/models/article.rb, adding the following public method:

def body_hidden=(newval)
  self.body = newval
end

This is still mostly untested, but I'm really enjoying it so far. Uploading an asset inline might not submit the body of the article anymore, so you'll have to a similar piece of javascript to copy the value from the codepress object or lose your changes when you upload an image.