Data Noise

OK, this is totally insane. First Jamis Buck hit us with Inspecting a live Ruby process article where he demonstrated a number of very interesting technics you can use in order to poke around your live Ruby process.

But then Mauricio Fernandez showed us how masters do their katas ( 型 ) in "Inspecting a live Ruby process", easier if you cheat.

I didn't know that gdb can be that powerful! I need to read more about it.

On a side note, I've just finished reading Scalable Internet Architectures by Theo Schlossnagle and I very enjoyed it. If you followed this interesting exchange of opinions, you should definitely read this book. The author make a clear point that the raw speed of your language of choice does not matter in creating scalable solutions. Even if your application is powered by C, you still can reach a point when your single server is not able to serve all incoming requests and you need to scale your application horizontally. This book explains how.

Next on my "to read" list is Building Scalable Web Sites by Cal Henderson.

One of my projects required a bittorrent integration. I've tried two available in Ruby options:

1. rubytorrent is a pure Ruby bittorrent client.
2. libtorrent-ruby is an inteface for libtorrent library.

Unfortunately, the first was not that stable and the latter I wasn't able to build at all. I am pretty sure that I was doing something wrong, but the time was pretty tight and I had to come up with a stable solution.

I remember there is a pretty decent open source application called Transmission. It didn't take me more than several hours to create a binding for Ruby. So here we go:

Transmission is a simple native bittorrent client for Ruby based on libtransmission library.

Installation

The only prerequisite for this library is that your Ruby must be compiled with pthread support enabled:

    $ ./configure --enable-pthread
    $ make
    $ sudo make install

Having done that, download gem and you should be able to install this gem without problems:

    $ sudo gem install transmission.gem

Synopsis

Assuming that you have progressbar gem installed:

    $ sudo gem install progressbar

You can use this script to download with a torrent file:

    require "rubygems"
    require "transmission"
    require "progressbar"

    # create a new session
    session = Transmission.new

    # open a torrent file
    torrent = session.open(ARGV[0])

    # set destination directory
    torrent.download_to File.expand_path('~/tmp')

    # starts a new native thread in background
    torrent.start

    progress = ProgressBar.new(" Downloading", 1.0)
    trap("INT") do
      torrent.stop
      progress.finish
      exit
    end

    # display progress bar
    until torrent.just_finished?
      progress.set(torrent.stat.progress)
      sleep 0.5
    end
    progress.finish

Known issues

1. I have tested it only on Linux and Mac OSX. It definitely is not going to work on Windows.
2. API is quite limited. For example, you can't create a new torrent file with it.

Enjoy.

I've been using Lucene for many years on many different projects. Thanks to Dave Balmain, this library is ported to Ruby and it seems to be even faster than original. It's called ferret. I've just started playing with it and so far it looks very promising.

Sometime ago, Zenspider posted a neat trick that allows searching ri database. Since Ruby version 1.8.5, this database is expanded significantly in size, not to mention that every installed gem extends it with its own documentation. It might take some time if you just go through all these yaml files and grep for an interesting information.

This particular task nicely fits with what ferret can offer.

Install ferret

    $ sudo gem install ferret

Build index using the following script (ri_indexer):

    $ cat ri_indexer
    #!/usr/bin/env ruby

    require "rdoc/ri/ri_driver"
    require "rubygems"
    require "ferret"
    require "find"
    require "yaml"
    include Ferret

    INDEX_FILE = File.expand_path('~/.ri_index')

    fis = Index::FieldInfos.new
    fis.add_field :name, :term_vector => :no
    fis.add_field :content, :store => :no
    fis.create_index(INDEX_FILE)

    index = I.new(:path => INDEX_FILE, :create => true)

    dirs = RI::Paths::PATH
    dirs.each do |dir|
      Find.find(dir) do |fn|
        next unless File.file?(fn)
        doc = YAML.load(File.read(fn))
        next unless doc.respond_to?(:comment)
        next unless doc.comment
        index << {
          :name => doc.full_name, 
          :content => doc.comment.map{|f|f.body if f.respond_to?(:body)}.join("\n")
        }
      end
    end
    index.optimize
    index.close
    $ ./ri_indexer

Now you can use this script for searching (ri_search):

    $ cat ri_search
    #!/usr/bin/env ruby

    require "rubygems"
    require "ferret"
    require "find"
    require "rdoc/ri/ri_driver"
    include Ferret

    INDEX_FILE = File.expand_path('~/.ri_index')

    query = ARGV.join(' ')
    ARGV.clear

    RI::Options.instance.use_stdout = true
    ri = RiDriver.new
    index = I.new(:path => INDEX_FILE)
    index.search_each(query) do |id, score|
      puts
      begin
        ri.get_info_for(index[id][:name])
      rescue Exception
        puts $!.message
      end
    end
    $ ./ri_search kill

This is way more faster than the original script. Also this script accepts quite sophisticated query expressions. For example,

    $ ri_search rescue AND public
    $ ri_search +split -String

Refer to ferret's trac web site, where you can find more information about this wonderful library.