Switching User Agents while Grabbing with Mechanize

When grabbing pages, your user-agent could compromise your efforts of ‘not getting sued‘, its easy to see that the so called
‘WWW-Mechanize/0.8.5 (http://rubyforge.org/projects/mechanize/)’ browser (mechanize default user agent) may not be ‘browsing’…

So in order to stay undetected, we change our user_agent…

require 'activesupport' #for the rand part...

class MyGrabber
  def grab
    doc = Hpricot(agent.get('www.lawyer-rich-company.com').body)
    puts doc.search('#secret_info span').inner_html
  end

  def agent
    return @agent if @agent
    @agent = WWW::Mechanize.new
    @agent.user_agent_alias = WWW::Mechanize::AGENT_ALIASES.keys.rand
    #login or what not...
    @agent
  end
end
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s