ParseTree: June 2008 Archives

And the winners are:

• ActionController's scaffolding.rb with a 100 node dstr!
• ActiveRecord's mysql.rb with a 274 node block!
• Date's format.rb with a 51 node hash!
• Net's telnet.rb with a 35 node dregx_once!
• Net::SFTP's attributes.rb with a 15 node args!
• ParseTree's pt_testcase.rb with a 567 node hash!
• TMail's racc generated parser with a 407 node array (4 times)!
• open-uri.rb with a 16 node dregx!

I have a file containing sexps for every file in standard lib and every gem I have installed. I should extend that out and do every latest gem out there, but that'd be unbelievably large. The current file is 18M.

But... with it, I can go spelunking for interesting tidbits... For example:

wtf = [[:args,       [15,  1]]
       [:array,      [407, 4]]
       [:block,      [274, 2]]
       [:case,       [51,  1]]
       [:dregx,      [16,  2]]
       [:dregx_once, [35,  1]]
       [:dstr,       [100, 1]]
       [:hash,       [567, 1]]]

wtf describes an interesting set of data. Each array contains a node-type, a size, and a frequency. How is this interesting? What does it mean? How about:

• someone, somewhere, decided that 15 args were needed in a method.
• four arrays in the code base that have 407 elements? Copy & paste anyone?
• Two blocks (method bodies, usually) have exactly 274 nodes. Curious size, no?
• A 51 node case statement (else + 25 whens). When was the last time you wrote 24 elsifs in a row? Were you fired for it?
• dregx and dstr nodes are regexps and strings with interpolation... Some string out there has ONE HUNDRED nodes (usually string/#{}/string/#{}/repeat). wow

Anyhow. As I go spelunking I'll try to come up with interesting tidbits and post them here.

ruby2ruby provides a means of generating pure ruby code easily from ParseTree's Sexps. This makes making dynamic language processors much easier in ruby than ever before.

Changes:

1.1.9 / 2008-06-09

• 5 minor enhancements:

  • Added more defensive programming in the tests to make it work with 1.9 and rubinius better.
  • Converted r2r_show to more plain parse style, no more discover_new_classes.
  • Made Proc#to_sexp and #to_ruby more resiliant.
  • Started to work on fallback to ruby_parser code. Should prolly do flog first.
  • Updated rakefile and readme format for hoe. Much cleaner!

• 6 bug fixes:

  • Added 1.9 fixes.
  • Added code to tests to isolate rubyinline builds.
  • Fixed miniunit-deprecated assertions
  • Fixes for const2/3, esp in class names
  • Renamed ProcStoreTmp#name to #new_name. dur.
  • Skip proc tests in 1.9 since they require ParseTree.

• http://seattlerb.rubyforge.org/

• http://rubyforge.org/projects/seattlerb

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

2.2.0 / 2008-06-09

• 18 minor enhancements:

  • Added 1.9 and rubinius to multiruby skip list.
  • Added 1.9 fixes for SexpProcessor.
  • Added compatibility changes for regexp option values.
  • Added custom compact since Array#compact is hostile to subclasses.
  • Added some tests for wonky index edge cases. some commented out.
  • Added test changes for 1.9 and r2r changes. Esp fixed regexp differences.
  • Added tests for dasgn, proc arities,
  • Added/updated some tests for ruby_parser.
  • Changed tests to default to nil for verbose.
  • Fixed all assertions deprecated in miniunit.
  • Raises LoadError if incompatible ruby, allows fallback to ruby_parser.
  • Removed Unified's rewrite_fbody and moved up into defn... not sure about that.
  • Removed argscat rewriter from unified_ruby.rb
  • Renamed shadowed variable name.
  • Reworked parsetreefor_(method|string) to take optional verbose arg.
  • Started removing ending newlines from pt_testcase.rb.
  • Uncommented all commented out tests. ruby_parser must not be a pansy.
  • Updated rakefile for new hoe abilities.

• 3 bug fixes:

  • Fixed 1.8.[45] wrt dasgn_curr declaration removal.
  • Fixed pt_testcase bug.
  • Fixes for colon2/3 in class/module/cdecls.

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com