Rails and Response Streaming

Posted on March 17, 2006

During my usual server maintenance procedure the other day I noticed that rails dispatch.fcgi processes take unusual amount of memory. By unusual I meant that instead of 25M of RAM for each process, they occupy about 150M each. I started the investigation process.

The first thing I did I created a small plugin with logged the current status of the RSS and VSZ memory and calculated a difference from these numbers and numbers from the previous request. Note that I've implemented it only for Linux platform. Here is the code itself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20


  module MemoryLogging
    def self.included(controller)
      controller.after_filter :log_mem_stat  if PLATFORM =~ /linux/
    end
  
    private

    def log_mem_stat
      vm_info = File.read("/proc/self/status").grep(/^Vm(RSS|Size)/).map{|l| 
        l.chomp.gsub(/\t/, ' ').gsub(/ +/, ' ')
      }
      rss, size = vm_info.map{|l| l.scan(/\d+/).first.to_i}
      info = "PID #{$$}, #{vm_info.join(', ')}"
      info << ", RSS Diff #{rss - $vm_rss} kB" if $vm_rss
      info << ", Size Diff #{size - $vm_size} kB" if $vm_size
      logger.info "Virtual Memory #{self.class.name}##{action_name} (#{info})"
      $vm_rss, $vm_size = rss, size
    end
  end

With this plugin installed, I let the application run for awhile and then I searched for actions that resulted in the significant memory leap. It turned out that problem lied in our reports controller. At the end of each day our client goes to the admin and prints all pending invoices for the past day and there might be hundreds of them. And here is the problem: Rails doesn't use response streaming; it builds response string all in memory and only when it's all done it sends it to the browser.

Here comes the solution. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


  def print_invoices
    orders = Orders.find :all, :conditions => ["created_on", Date.today]
    render :text => lambda { |response, out| 
      out << render_template(<<-EOS)
        <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
        <html>
          <head>
            <meta http-equiv="Content-Language" content="en-us" />
            <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
            <%= stylesheet_link_tag 'admin', :media => 'screen' %>
            <%= stylesheet_link_tag 'print', :media => 'print' %>
            <title>Order Invoices</title>
          </head>
          <body>
      EOS
      orders.each do |order|
        out << render_partial :object => 'order', order
      end
      out << render_template(<<-EOS)
          </body>
        </html>
      EOS
    }
  end

Unfortunately, you can use layouts when you do the response streaming this way. Simply because when Rails renders a template with a layout, it first renders template itself and only then it renders the layout and substitutes @content_for_layout variable with the result calculated on the first step.

Now with this print_invoice action in place, our dispatchers are back to the normal memory consumption.

Filed under: Rails |