Recently in Ruby2C Category

add_tests("lasgn_call",
          "Rewriter"    => :same,
          "TypeChecker" => t(:lasgn, :c,
                             t(:call,
                               t(:lit, 2, Type.long),
                               :+,
                               t(:arglist,
                                 t(:lit, 3, Type.long)),
                               Type.long),
                             Type.long),
          "CRewriter"   => :same,
          "RubyToAnsiC" => "c = 2 + 3", # FIX: probably not "c = ..."
          "RubyToRubyC" => 'c = rb_funcall(LONG2NUM(2), rb_intern("+"), 1, LONG2NUM(3))')

Can you think of a better / cleaner way? This generates 5 test_lasgn_call methods in 5 different test classes. :same says "use previous output as my input", and Rewriter gets its input from ParseTree's test suite.

I've got 297 of these add_tests method calls. Gah. Just seems a bit much, but I can't really think of a better way ATM.

(to try to make up for the previous rant)

I've been making big fundamental changes to ParseTree and consequently ruby2c's implementation and testing. As a result, I'm drowning in hundreds of failures (869 tests, 2217 assertions, 238 failures, 193 errors). Even though they run fairly fast, autotest isn't much help at this stage, so I resorted to (*gulp*) manual test runs, but with a small twist:

ruby test/test_type_checker.rb -n "/test_[a-b]/" | unit_diff

This lets me focus on one class, with only about 10 or so failures/errors at a time to keep the clutter down. As my favorite mgr used to say:

"Big alligators, Little alligators, Swamp."

With this technique, I work from the bottom up and slowly wade through and eventually drain the swamp.

After almost a year, I'm pleased to announce that another beta of ruby2c is available. The project stagnated for a bit, but is coming back in force! We've solidified this release by gemifying it, splitting up our tail end translator to convert to either ANSI C or ruby internals C, and making it much more happy with other tool-chains (like our obfuscator).

ruby2c translates a static ruby subset to C. Hopefully it works.

Enough Propaganda! Just install it!

sudo gem install RubyToC

CHANGES:

+ 6 minor enhancements:
+ Split RubyToC to RubyToRubyC and RubyToAnsiC.
+ Extended Environment to be more flexible for various situations.
+ Removed propaganda (bloat) from release.
+ Gemified and reorganized things. Support still needs splitting up.
+ Flipped a lot of internal naming to use Unique.
+ Added ruby_to_c_show (like parse_tree_show).

+ 4(ish) bug fixes:
+ Use ivars instead of cvars so inheritance won't bugger the translator.
+ Corrected unsupported node lists in pipeline.
+ Fixed bugs for splat args, iters, optional args, method name map.
+ Fixed many other bugs.

Phase 1: All the unit tests are passing.

Phase 2: We can even obfuscate the unit tests and run the binary version of them and they still pass.

Phase 3: We're currently working on obfuscating the obfuscator itself and passing the unit tests a third time.

We've got 6 methods (out of 56) not translating. 2 of them I think I can do w/o too much hassle, but the other 4 (all 1 related issue) are a PITA (closure semantics just don't map to C very well). I think I just brainstormed a solution that may actually work once I have rested my brain some.

One nice thing is that every lesson we learn from the obfuscator is something we can potentially use to improve ruby2c.

That said, we're damn close to having a full-fledged ruby obfuscator that we'll make commercially available in the near future.

Awesome hacking session tonight. Our ruby obfuscator is now actually on track to be generally usable by the end of the month. (Although we still need documentation--badly)

We have an awesome architecture to write these types of tools in, but there are several things that get complicated and having an overly clean architecture actually gets a bit hard to deal with. The biggies are that C and Ruby disagree on what is an expression vs statement... as such, we often have to rearrange the order of things, and the way our architecture works, sometimes that is easy, sometimes not.

Today we got things like map working. More specific iteration has worked for some time, but now we can do more general iterations with blocks. There are restrictions currently, like we don't even remotely honor block closure semantics (and, I'm not sure we ever will), but we're actually doing a fair amount now and it is going pretty smoothly.

Next up will be extending our splat arg support to convert to C var-args so they actually work and generalizing our block support (where possible).

P.S. I should mention, the work on the obfuscator is also a nice forcing function for solidifying ruby2c and ParseTree.

• Plethora of passing unit-tests: check
• Simple reproducible system: check
• Ability to override core classes: check
• Binary with our core classes: check
• Segfaults: check

% cd MR
% make libmetaruby.a
...
% cd RUBY
% patch -p0 < metaruby.patch
...
% ./configure --program-prefix=meta
...
% cp ~/Links/MR/libmetaruby.a .
% make metaruby
...
% ./metaruby bin/irb 
irb(main):001:0> 1+1
=> 2
irb(main):002:0> false
/usr/local/lib/ruby/1.8/irb.rb:298: [BUG] Segmentation fault
ruby 1.8.3 (2005-09-10) [powerpc-darwin8.2.0]

It is time to split the tail of ruby2c and start making a translator for ruby internals. So far we've only done generic C.

We are very close... very very close...

Toys

I just finished releasing a WHOLE bunch o' stuff, all gemified and cleaned up, mainly so that we could contribute the following solution to this week's ruby quiz (serializable procs):

require 'r2c_hacks'
class ProcStore # We have to have this because yaml calls allocate on Proc
  def initialize(&proc)
    @p = proc.to_ruby
  end

  def call(*args)
    eval(@p).call(*args)
  end
end

code = ProcStore.new { |x| return x+1 }
=> #<ProcStore:0x3db25c @p="proc do |x|\n  return (x + 1)\nend">

OK, well, that obviously isn't the whole solution, but now that we've added Proc.to_ruby, it really is that simple. To see how simple it really is, check out our implementation:

class Proc
  ProcStoreTmp = Class.new unless defined? ProcStoreTmp
  def to_ruby
    ProcStoreTmp.send(:define_method, :myproc, self)
    m = ProcStoreTmp.new.method(:myproc)
    result = m.to_ruby.sub!(/def myproc\(([^\)]+)\)/, 'proc do |\1|')
    return result
  end
end

We would have bypassed ProcStore entirely had we known how to get YAML to not call allocate on Proc when it was loading a proc back in. But, we were tired.

The Releases!

We released the following:

RubyInline 3.4.0 with:

+ 2 minor enhancements:
    + Changed inline to take the language and a hash of options.
        + Still backwards compatible, for now, and emits a warning.
    + Options are available via the builder passed to your block.
+ 2 bug fixes:
    + Modified caller discovery, yet again, due to changes in ruby 1.8.3.
    + More compatible and clean with non-gems systems.

ParseTree 1.3.7 with:

+ 3 bug fixes:
    + Fixed rubygem requires for non-gem systems.
    + Renamed on to on_error_in to make more clear.
    + Moved exceptions to their own tree to make catching cleaner.

ruby2c 1.0.0 beta 4 with:

+ 1 minor enhancements
    + Added gemspec (hastily).
+ 2 bug fixes
    + Translates bool type to VALUE since we were using Qtrue/Qfalse.
    + Fixed rubygems for non-gem systems.

and finally ZenHacks 1.0.1 with:

+ 4 minor enhancements:
    + Added Graph#save and edge accessors for customizations.
    + Added Proc#to_ruby in r2c_hacks.
    + Added RubyToRuby#rewrite_defn to support bmethods (read: procs).
    + Added Class#optimize(*methods) to zenoptimize.
    + Added bin/nopaste for cmdline access to nopaste on rafb.net.
    + Added bin/quickbench to generate ruby benchmark templates quickly.
+ 4 bug fixes:
    + Fixed gems for non-gem systems.
    + Fixed minor output problem with RubyToRuby#process_defn.
    + Added some minor (performance) enhancements to zenoptimize.
    + Added requirements to gemspec to quell whining.

My rubyconf 2005 presentation proposal was accepted!

I plan on talking about all the technologies and toys that have spawned out of the metaruby project. The whole fam.

I plan on illustrating the family's usefulness and power through examples including: an example of RubyInline, a language extension for RubyInline, aparse tree complexity metric grapher, a simple example of ruby2c translation, amaintainable and fast profiler, a Ruby obfuscator, and a dynamic Ruby optimizer that requires zero effort to use.

For fun, here is something I scratched up tonight with Eric to play with the idea of lisp-like ruby posted on redhanded recently. Check it, real s-expressions in ruby:

require 'ruby2ruby'

class Module
  def _(sexp)
    self.module_eval RubyToRuby.new.process(sexp)
  end
end

class Foo
  _ [:defn, :example, [:args], [:call, [:lit, 1], :+, [:array, [:lit, 1]]]]
end

p Foo.new.example
=> 2

I just released ruby2c 1.0.0 beta 3! Have at it. Here are the release notes:

Ruby2C provides a pipeline of SexpProcessor classes to work with ParseTree output. Processors included:

Rewriter - massages the sexp into a more consistent form.

TypeChecker - type inferencer for the above sexps.

RubyToC - converts a ruby (subset) sexp to C.

Remember folks: THIS IS BETA SOFTWARE!

Changes:

• 16 minor enhancements:
  • Added ivar and iasgn support. Needs more work.
  • Added limited support for self.
  • Added pipeline tests for bools, call_arglist, call_attrasgn, fbody.
  • Added process_not to RubyToC.
  • Added support for float and symbol literals.
  • Added support for gasgn, cvasgn, const (class consts, not classes).
  • Improved error handling/reporting, esp in RubyToC.
  • In TypeChecker.boostrap, pre-registered all base classes.
  • Modified process_class test to include a class const.
  • Processing :class now adds class constants to the local var scope.
  • Processing :const checks both genv and env now.
  • Rearchitected all tests into a pipeline test class.
  • Rewrite attrasgn into regular calls.
  • Rewrite fbody into a regular defn.
  • Rewrote :array inside call to :arglist.
  • Rewrote :or nodes in process_case to correctly be binary.
• 1 bug fix:
  • Fixed a bug where single line while bodies were missing a semicolon.