Data Noise

Lately I've been playing with MacFuse project and all I could say is that this is a fantastic effort of bringing the well-known user-space(fuse) file system to the OSX land. The latest versions of fuse work well with FreeBSD and MacFuse extends this functionality to OSX platform.

My goal was to make Ruby's fusefs extension play along with it. The major stumbling point was that MacFuse requires using the new fuse API, so I had to make several modifications to fusefs in order to make it possible at least to compile it.

OK. If you are a braveheart type and not afraid that it might damage your computer, these are the steps you need to do in order to install this extension:

1. Download the latest version of MacFuse and use the installer that it comes with.

2. Check out my version of fusefs from http://svn.datanoise.com/fusefs-osx and use the standard procedure of installing Ruby extensions:

   $ svn co http://svn.datanoise.com/fusefs-osx
   $ cd fusefs-osx
   $ make
   $ sudo make install
   

That's about it. Now you can start playing with examples located in <fusefs-osx>/sample directory. When you are at it, I'd like you to take a look at one of my personal toys - the remote DRb-based file system. It is located in <fusefs-osx>/sample/drb directory and that's how you use it:

1. On the remote side (a Windows box will do just fine) start the drb server:

   $ ruby ./sample/drb/drbfs_server.rb ~/shared
   druby://myhost:7777
   

   The script takes the name of a directory you want to share. When the script is started, it'll print the URL you have to use when you mount your DRb filesystem on the client side.

2. On the client side, mount DRb file system:

   $ ruby ./sample/drb/drbfs.rb mount/ druby://myhost:7777
   $ ls mount
   

   The client script takes the name of a directory you want to mount to and the URL of the server from the step one.

Enjoy and don't forget to read <fusefs-osx>/OSX.txt for the important API changes.

So many times while watching log files I wanted to find a command that combines tail with grep filtering functionally. The Unix command repository is huge and I'm pretty sure you can do it without resorting to a scripting language. But having spent several minutes googling for this functionality to no avail, I decided to bring Ruby to the rescue.

At first I thought that I had to port File::Tail Perl module to Ruby, but I found that it has been done already by Florian Frank. He has even provided a convenient way of installing it:

    $ sudo gem install file-tail

OK, the rest was as obvious as this:

    $ cat tailg
    #!/usr/bin/env ruby

    require 'rubygems'
    gem 'file-tail'
    require 'file/tail'


    if ARGV.size != 2
      $stderr.puts "Usage #$0 <regexp> <log-file>"
    end

    begin
      File.open(ARGV[1]) do |log|
        log.extend(File::Tail)
        log.interval = 2
        log.backward(10)
        log.tail { |line| puts line if line =~ /#{ARGV[0]}/ }
      end 
    rescue Exception
    end

    $./tailg 'products' /var/log/searchd_query.log
    [Sat Mar  3 13:39:40 2007] 0.000 sec [all/0/rel 3 (0,9)] [products] NOA
    [Sat Mar  3 13:39:44 2007] 0.000 sec [all/0/rel 22 (0,9)] [products] lauren
    [Sat Mar  3 13:40:09 2007] 0.000 sec [all/0/rel 22 (9,9)] [products] lauren

tailg command accepts two parameters: a regexp to filter for interesting lines and a log file itself.

ruby-debug 0.5.1 has been released! The installation as usual:

    $ sudo gem install ruby-debug

Below is the list of changes:

Post-mortem debugging

First of all, what the heck is post-mortem debugging? Let's say you run a script and instead of the expected result you get an exception trace. Wouldn't it be great if we can roll back to the point where this exception is raised and explore the state of your program (possibly, by moving up and down the frame stack)?

Note that it is different than setting a catch point. By setting a catch point you activate the debugger when an exception (of a specific class) is about to be raised and it doesn't matter whether it's going to be handled later in the code or not. In the post-mortem case you know that an exception's been raised and not handled as expected.

Now I'm going to demonstrate several ways of using this feature, starting from the simplest:

Google calculator is a very powerful tool. Besides basic arithmetic, you can use it in many other interesting ways. But I always wanted to use it from the command line without opening a new Safari window.

Ruby to the rescue:

    $ cat gcal 
    #!/usr/bin/env ruby
    %w(rubygems open-uri hpricot erb).each {|lib| require lib }
    doc = Hpricot(open("http://www.google.com/search?q=#{ERB::Util.u(ARGV*' ')}"))
    puts (doc/'/html/body/p/table/tr/td[3]/font/b').inner_text

    $ ./gcal 2^20
    2^20 = 1 048 576

    $ ./gcal 'sqrt(-1)'
    sqrt(-1) = i

    $ ./gcal 30 rubles in dollars
    30 Russian rubles = 1.1379844 U.S. dollars

    $ ./gcal 2 gallons in liters
    2 US gallons = 7.5708236 liters

It's not too bad for three-liner like this. By the way, did I say that Hpricot is awesome?

OK, this is totally insane. First Jamis Buck hit us with Inspecting a live Ruby process article where he demonstrated a number of very interesting technics you can use in order to poke around your live Ruby process.

But then Mauricio Fernandez showed us how masters do their katas ( 型 ) in "Inspecting a live Ruby process", easier if you cheat.

I didn't know that gdb can be that powerful! I need to read more about it.

On a side note, I've just finished reading Scalable Internet Architectures by Theo Schlossnagle and I very enjoyed it. If you followed this interesting exchange of opinions, you should definitely read this book. The author make a clear point that the raw speed of your language of choice does not matter in creating scalable solutions. Even if your application is powered by C, you still can reach a point when your single server is not able to serve all incoming requests and you need to scale your application horizontally. This book explains how.

Next on my "to read" list is Building Scalable Web Sites by Cal Henderson.

One of my projects required a bittorrent integration. I've tried two available in Ruby options:

1. rubytorrent is a pure Ruby bittorrent client.
2. libtorrent-ruby is an inteface for libtorrent library.

Unfortunately, the first was not that stable and the latter I wasn't able to build at all. I am pretty sure that I was doing something wrong, but the time was pretty tight and I had to come up with a stable solution.

I remember there is a pretty decent open source application called Transmission. It didn't take me more than several hours to create a binding for Ruby. So here we go:

Transmission is a simple native bittorrent client for Ruby based on libtransmission library.

Installation

The only prerequisite for this library is that your Ruby must be compiled with pthread support enabled:

    $ ./configure --enable-pthread
    $ make
    $ sudo make install

Having done that, download gem and you should be able to install this gem without problems:

    $ sudo gem install transmission.gem

Synopsis

Assuming that you have progressbar gem installed:

    $ sudo gem install progressbar

You can use this script to download with a torrent file:

    require "rubygems"
    require "transmission"
    require "progressbar"

    # create a new session
    session = Transmission.new

    # open a torrent file
    torrent = session.open(ARGV[0])

    # set destination directory
    torrent.download_to File.expand_path('~/tmp')

    # starts a new native thread in background
    torrent.start

    progress = ProgressBar.new(" Downloading", 1.0)
    trap("INT") do
      torrent.stop
      progress.finish
      exit
    end

    # display progress bar
    until torrent.just_finished?
      progress.set(torrent.stat.progress)
      sleep 0.5
    end
    progress.finish

Known issues

1. I have tested it only on Linux and Mac OSX. It definitely is not going to work on Windows.
2. API is quite limited. For example, you can't create a new torrent file with it.

Enjoy.

I've been using Lucene for many years on many different projects. Thanks to Dave Balmain, this library is ported to Ruby and it seems to be even faster than original. It's called ferret. I've just started playing with it and so far it looks very promising.

Sometime ago, Zenspider posted a neat trick that allows searching ri database. Since Ruby version 1.8.5, this database is expanded significantly in size, not to mention that every installed gem extends it with its own documentation. It might take some time if you just go through all these yaml files and grep for an interesting information.

This particular task nicely fits with what ferret can offer.

Install ferret

    $ sudo gem install ferret

Build index using the following script (ri_indexer):

    $ cat ri_indexer
    #!/usr/bin/env ruby

    require "rdoc/ri/ri_driver"
    require "rubygems"
    require "ferret"
    require "find"
    require "yaml"
    include Ferret

    INDEX_FILE = File.expand_path('~/.ri_index')

    fis = Index::FieldInfos.new
    fis.add_field :name, :term_vector => :no
    fis.add_field :content, :store => :no
    fis.create_index(INDEX_FILE)

    index = I.new(:path => INDEX_FILE, :create => true)

    dirs = RI::Paths::PATH
    dirs.each do |dir|
      Find.find(dir) do |fn|
        next unless File.file?(fn)
        doc = YAML.load(File.read(fn))
        next unless doc.respond_to?(:comment)
        next unless doc.comment
        index << {
          :name => doc.full_name, 
          :content => doc.comment.map{|f|f.body if f.respond_to?(:body)}.join("\n")
        }
      end
    end
    index.optimize
    index.close
    $ ./ri_indexer

Now you can use this script for searching (ri_search):

    $ cat ri_search
    #!/usr/bin/env ruby

    require "rubygems"
    require "ferret"
    require "find"
    require "rdoc/ri/ri_driver"
    include Ferret

    INDEX_FILE = File.expand_path('~/.ri_index')

    query = ARGV.join(' ')
    ARGV.clear

    RI::Options.instance.use_stdout = true
    ri = RiDriver.new
    index = I.new(:path => INDEX_FILE)
    index.search_each(query) do |id, score|
      puts
      begin
        ri.get_info_for(index[id][:name])
      rescue Exception
        puts $!.message
      end
    end
    $ ./ri_search kill

This is way more faster than the original script. Also this script accepts quite sophisticated query expressions. For example,

    $ ri_search rescue AND public
    $ ri_search +split -String

Refer to ferret's trac web site, where you can find more information about this wonderful library.

How often did you write this kind of code?

    def initialize(arg1, arg2, arg3)
      @arg1 = arg1
      @arg2 = arg2
      @arg3 = arg3
    end

All this constructor does it propagates parameters to initialize method to instance variables. In Ruby, there is a better way:

    class Binding
      def local_to_instance
        eval("local_variables").each do |name|
          eval("self").instance_variable_set("@#{name}", eval(name))
        end
      end

      alias :kernel_eval :eval
      def eval(code)
        kernel_eval(code, self)
      end
    end

    class Test
      attr_accessor :arg1, :arg2, :arg3

      def initialize(arg1, arg2, arg3)
        binding.local_to_instance
      end
    end

    t = Test.new(1,2,3)
    puts t.arg1
    # => 1
    puts t.arg3
    # => 3
    p t
    # => #<Test:0x1c6600 @arg1=1, @arg3=3, @arg2=2>

Note that Ruby 1.9 already provides Binding#eval method, so you don't need this kernel_eval hack.

I know this kind of silly and, I guess, inefficient. But still.

Many Rails users enjoyed this little gem developed by Florian Groß. But as Mauricio Fernandez noted in his blog, this extension is broken with the brand new Ruby 1.8.5. In order to make its magic possible, breakpoint library relied on the implementation of Binding.of_caller, which in turn relied on the bug in Ruby's implementation of trace calls. With new release, this bug has been patched.

Mauricio has started working of fixing this problem and proposed a new method binding_n(n), which would return a binding for any frame in the call stack. But you don't have to wait for his work to complete. ruby-debug has this functionality already implemented for you.

Consider this small snippet:

    require "rubygems"
    require 'ruby-debug'

    module Kernel
      def binding_n(n = 0)
        frame = Debugger.current_context.frames[n+1]
        raise "Unknown frame #{n}" unless frame
        frame.binding 
      end
    end

    def test
      puts eval("var", binding_n(1))
    end

    Debugger.start do
      var = 'Hello'
      test
    end

And most likely, I will add this method to the next version of ruby-debug. It can be handy sometimes.

Rinda framework is part of the standard Ruby library. But what is it and how can it be useful? This question I will try to address in this article.

In the center of Rinda is a concept called tuple spaces. The idea of tuple spaces was initially introduced in the programming language Linda in 1982 which was designed to ease the implementation of distributed applications. With Rinda your application consists of many simultaneously running programs located anywhere on your network. All these programs are coordinated by a single process, tuple space. In essence, tuple space is a shared repository of objects accessed via network. What is interesting is that TupleSpace itself doesn't coordinate programs, but rather programs use TupleSpace to coordinate their workflow with each other. TupleSpace service provides a handful set of operations that I am going to describe next. What is really interesting is that just by using this small number of methods you can implement very powerful concepts.

What kind of objects TupleSpace can hold?

In essence, Rinda tuple space can hold Arrays or Hashs. Actually, it can hold any object that resembles (via ducktyping):

• array, by implementing size and [] methods.
• hash, by implementing each {|k,v| ...} and keys methods.

I remember a long time ago when I was programming C++ and using CORBA/COM(+) most of the time, frameworks spent reams of code dedicated to make one simple thing possible: asynchronous remote calls. You start a remote call, then go about your business and return for a result.

I remember I spent huge amount of time trying to make this thing work. Let's see how Ruby can handle this task. I'm going to use XMLRPC framework for demonstration purposes. Let's say you want to fetch the latest post from your Typo blog and you want to do it asynchronously of course.

    require "xmlrpc/client"

    client = XMLRPC::Client.new2("http://www.yourblog.com/backend/xmlrpc")

    post = Thread.start {
      client.call_async(*%w|metaWeblog.getRecentPosts blog user pwd 1|).first
    }

    # do your stuff here
    $stdout.sync = true
    5.times do print '.'; sleep 1; end
    puts
    # done

    puts post.value['url']

In this code I rely on the fact that Thread#value method returns a result of the last statement of the thread's block and it joins on the thread if it's still running.

Not too bad for one-liner!

OK, here's a new version of ruby-debug. Most changes are related to the remote debugging and improvements targeted for a possible GUI integration:

• Wait for a client connection. With this option on, the debugger will wait for a connection, before returning control to the script. You can enable it with -w option of rdebug script or by calling Debugger.wait_connection = true.

• Stop on connect. With this option on, the debugger will stop when a remote client establishes a connection. rdebug script has this parameter enabled by default, but you can disable it with -n option. In your own script you can use Debugger.stop_on_connect = true to activate it.

• Controlling thread. When activating a remote debugger, a control thread is activated as well. It listens on the socket (port 8990, by default) and accepts a number of commands that you can use to control a debugger, such as add/delete breakpoints, interrupt an application, etc. This is quite helpful for a GUI frontend, where you can set breakpoints at any time while your application is running.

There are also a couple of additions to the debugger commands:

• l[ist] = - displays the current exception point.

• f[rame] n - switches to nth frame. You can see the list of frames and their numbers with frame or where command.

• tm[ate] - if you are running Mac OSX and addicted to TextMate, this command opens the current file using this editor.

That's all for now. Enjoy.

Scott Bronson has published a very cool article about the current offerings when it comes to debugging Ruby applications. First off, I appreciate the kind words regarding ruby-debug extension. Second off, he made a very interesting observation:

    I hardly ever need a debugger when developing new Ruby on Rails applications.  
    However, when trying to understand an existing codebase, I find that few things 
    help more in understanding exactly what is going on than stepping through 
    the code using a debugger.

I fully subscribe to this idea. I've been developing with Ruby since 2002 and I hardly ever needed a debugger. But there were a couple of times when I really wanted to have a debugger. It was when I ran into a bug in an existing library and I didn't have an intimate knowledge of its internals (no pun intended). It's just much easier to step through the code and see how things are meshed together.

Also, I'd like to make some comments on the current debugger offerings:

• There is an official debug.rb library bundled with the standard Ruby distribution. AFAIK, all other debuggers, except maybe Arachno Ruby's one, are based on this small and nifty library, including my ruby-debug. The major drawback of it is the speed of the execution.

• Debugger from ZenHack tries to fix this problem by re-implementing trace function in C using RubyInline extension. Unfortunately, it is still quite slow to my taste.

• Debugger from Arachno Ruby IDE is very nice, but the last time I checked it used a patched version of Ruby interpreter. Which means that you have to ask for a new patch with each subsequent Ruby release. (But don't quote me on that though :-)

• There is another GUI debugger called MrGuid which uses a slightly modified version of debug.rb.

Remote Debugging

I've just released a new version of ruby-debug which has a new feature: remote debugging. In order to use it you have to pass -s parameter to rdebug script:

    $ rdebug -s <script.rb>

This option makes the debugger start listening for an incoming TCP connection on port 8989, by default. You can change the default port and host name with -p and -h option respectively.

Now when the debugger is started this way, you can connect to it using this command:

    $ rdebug -c

Also since this feature is implemented using plain sockets, you can connect to the remote debugger using a plain telnet client:

    $ telnet localhost 8989
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.
    script/../config/../app/controllers/shop_controller.rb:28: product_id = params['id']
    PROMPT (rdb:3) 
    list
    [23, 32] in script/../config/../app/controllers/shop_controller.rb
       23      end
       24    end
       25  
       26    def product
       27      debugger
    => 28      product_id = params['id']
       29      unless product_id
       30        redirect_to :action => 'index'
       31        return
       32      end
    PROMPT (rdb:3) 
    cont

API Changes

One more thing. There are two important changes have been made.

• First off, starting from this version when you require 'ruby-debug', it doesn't activate the debugger by default. You have to explicitly activate it by calling Debugger#start method. Also if you want to activate remote debugging, you should start the listener server by calling Debugger#start_server(host, port) method.

  require 'ruby-debug'
  Debugger.start_server
  Debugger.start
  

• Second off, it used to be possible at the debugger prompt to type anything and the debugger evaluates the expression, now you should use eval or p command.

Win32 gem

Thanks to Max Muermann, win32 version of ruby-debug 0.1.5 is available for download. The ruby-debug 0.2 version is coming too.

Update: win32 version of ruby-debug 0.2 is now available.

Preface

Overcomplicated specifications lead to overcomplicated implementations. Lately I've been fixing issues with ActionWebService framework - a soon to be removed part of Ruby on Rails. I have to use SOAP in the web application I'm developing. My client needs to keep his product inventory in a good shape and he requested to implement some of the inventory management functionality using handheld devices. There is a .Net environment available for this kind of devices and it works very well. So I desperately needed a functional web service implementation for Rails. That why I volunteered to fix AWS. And that's when I found out that ruby-breakpoint just doesn't cut it. Don't get me wrong, soap4r is a fine piece of software, but SOAP is difficult and obscure, which leads to complicated libraries that implement it. The situation is even worse when such libraries have almost no documentation whatsoever.

So instead of just stopping at some point in your program to examine the environment (the facility offered by ruby-breakpoint library), ruby-debug extension offers the full-fledged debugger for Ruby. The main difference between ruby-debug and the standard debug.rb library is the speed of the execution. Major problem with debug.rb is that it uses Kernel#set_trace_func method, which requires creation of Binding object for each hook invocation. It is fine for small scripts, but for the real world applications like Rails ones, debug.rb is almost impossible to use. You just sit and watch how Ruby interpreter creates enormous amount of Binding objects, just in order to destroy them with the immediate garbage collection cycle. It also explains that ruby-debug doesn't support watchpoints for the same reason.