You can speed up development load time and get rid of rspec in development if you simply not require it in :development and just load the tasks in the Rakefile where they are needed.

# Gemfile
group :development, :test do
  gem 'rspec-rails', :require => false
end

# Rakefile
require File.expand_path('../config/application', __FILE__)
require 'rspec-rails' if ['test', 'development'].include?(Rails.env)
...

Repeat for any other testing lib for even more speedup 🙂