Recently in ParseTree Category

spelunking, pt 2

| | Comments (1)

And the winners are:

  • ActionController's scaffolding.rb with a 100 node dstr!
  • ActiveRecord's mysql.rb with a 274 node block!
  • Date's format.rb with a 51 node hash!
  • Net's telnet.rb with a 35 node dregx_once!
  • Net::SFTP's attributes.rb with a 15 node args!
  • ParseTree's pt_testcase.rb with a 567 node hash!
  • TMail's racc generated parser with a 407 node array (4 times)!
  • open-uri.rb with a 16 node dregx!

spelunking, pt 1

| | Comments (0)

I have a file containing sexps for every file in standard lib and every gem I have installed. I should extend that out and do every latest gem out there, but that'd be unbelievably large. The current file is 18M.

But... with it, I can go spelunking for interesting tidbits... For example:

wtf = [[:args,       [15,  1]]
       [:array,      [407, 4]]
       [:block,      [274, 2]]
       [:case,       [51,  1]]
       [:dregx,      [16,  2]]
       [:dregx_once, [35,  1]]
       [:dstr,       [100, 1]]
       [:hash,       [567, 1]]]

wtf describes an interesting set of data. Each array contains a node-type, a size, and a frequency. How is this interesting? What does it mean? How about:

  • someone, somewhere, decided that 15 args were needed in a method.
  • four arrays in the code base that have 407 elements? Copy & paste anyone?
  • Two blocks (method bodies, usually) have exactly 274 nodes. Curious size, no?
  • A 51 node case statement (else + 25 whens). When was the last time you wrote 24 elsifs in a row? Were you fired for it?
  • dregx and dstr nodes are regexps and strings with interpolation... Some string out there has ONE HUNDRED nodes (usually string/#{}/string/#{}/repeat). wow

Anyhow. As I go spelunking I'll try to come up with interesting tidbits and post them here.

ruby2ruby provides a means of generating pure ruby code easily from ParseTree's Sexps. This makes making dynamic language processors much easier in ruby than ever before.

Changes:

1.1.9 / 2008-06-09

  • 5 minor enhancements:

    • Added more defensive programming in the tests to make it work with 1.9 and rubinius better.
    • Converted r2r_show to more plain parse style, no more discover_new_classes.
    • Made Proc#to_sexp and #to_ruby more resiliant.
    • Started to work on fallback to ruby_parser code. Should prolly do flog first.
    • Updated rakefile and readme format for hoe. Much cleaner!
  • 6 bug fixes:

    • Added 1.9 fixes.
    • Added code to tests to isolate rubyinline builds.
    • Fixed miniunit-deprecated assertions
    • Fixes for const2/3, esp in class names
    • Renamed ProcStoreTmp#name to #new_name. dur.
    • Skip proc tests in 1.9 since they require ParseTree.
  • http://seattlerb.rubyforge.org/

  • http://rubyforge.org/projects/seattlerb

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

2.2.0 / 2008-06-09

  • 18 minor enhancements:

    • Added 1.9 and rubinius to multiruby skip list.
    • Added 1.9 fixes for SexpProcessor.
    • Added compatibility changes for regexp option values.
    • Added custom compact since Array#compact is hostile to subclasses.
    • Added some tests for wonky index edge cases. some commented out.
    • Added test changes for 1.9 and r2r changes. Esp fixed regexp differences.
    • Added tests for dasgn, proc arities,
    • Added/updated some tests for ruby_parser.
    • Changed tests to default to nil for verbose.
    • Fixed all assertions deprecated in miniunit.
    • Raises LoadError if incompatible ruby, allows fallback to ruby_parser.
    • Removed Unified's rewrite_fbody and moved up into defn... not sure about that.
    • Removed argscat rewriter from unified_ruby.rb
    • Renamed shadowed variable name.
    • Reworked parsetreefor_(method|string) to take optional verbose arg.
    • Started removing ending newlines from pt_testcase.rb.
    • Uncommented all commented out tests. ruby_parser must not be a pansy.
    • Updated rakefile for new hoe abilities.
  • 3 bug fixes:

    • Fixed 1.8.[45] wrt dasgn_curr declaration removal.
    • Fixed pt_testcase bug.
    • Fixes for colon2/3 in class/module/cdecls.
  • http://rubyforge.org/projects/parsetree/

  • http://www.zenspider.com/ZSS/Products/ParseTree/
  • ryand-ruby@zenspider.com

Thanks to Luis Lavena, ParseTree now has gems precompiled for windows! This makes things like ruby2ruby, heckle, and much more that much more accessible to the OS-challenged!

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

  • Uses RubyInline, so it just drops in.
  • Includes SexpProcessor and CompositeSexpProcessor.
    • Allows you to write very clean filters.
  • Includes UnifiedRuby, allowing you to automatically rewrite ruby quirks.
  • ParseTree#parsetreefor_string lets you parse arbitrary strings of ruby.
  • Includes parsetreeshow, which lets you quickly snoop code.
    • echo "1+1" | parsetreeshow -f for quick snippet output.
  • Includes parsetreeabc, which lets you get abc metrics on code.
    • abc metrics = numbers of assignments, branches, and calls.
    • whitespace independent metric for method complexity.
  • Includes parsetreedeps, which shows you basic class level dependencies.
  • Does not work on the core classes, as they are not ruby (yet).

Changes:

2.1.1 / 2007-12-22

ruby_parser (RP) is a ruby parser written in pure ruby (utilizing racc--which does by default use a C extension). RP's output is the same as ParseTree's output: s-expressions using ruby's arrays and base types.

FEATURES/PROBLEMS:

  • Pure ruby, no compiles.
  • Incredibly simple interface.
  • Output is 100% equivalent to ParseTree.
    • Can utilize PT's SexpProcessor and UnifiedRuby for language processing.
  • Known Issue: Speed sucks currently. 5500 tests currently run in 21 min.
  • Known Issue: Code is waaay ugly. Port of a port. Not my fault. Will fix RSN.
  • Known Issue: I don't currently support newline nodes.
  • Known Issue: Totally awesome.
  • Known Issue: dasgn_curr decls can be out of order from ParseTree's.
  • TODO: Add comment nodes.

SYNOPSIS:

RubyParser.new.parse "1+1"
# => s(:call, s(:lit, 1), :+, s(:array, s(:lit, 1)))

Changes:

1.0.0 / 2007-12-20

ruby2ruby provides a means of generating pure ruby code easily from ParseTree's Sexps. This makes making dynamic language processors much easier in ruby than ever before.

Changes:

1.1.8 / 2007-08-21

  • 6 minor enhancements:

    • Added super awesome .autotest file. YAY!
    • Removed nil.method_missing... too many ppl bitching about it.
    • Renamed RubyToRuby (the class name) to Ruby2Ruby.
    • Restructured self-translation tests so they were friendlier when dying.
    • Strings are now always one line long only.
    • Fully in sync with ParseTree and ruby_parser.
  • 2 bug fixes:

    • Fixed a number of issues/bugs discovered via ruby_parser.
    • Cleaned out some dead code and hacks we don't need anymore.
  • http://seattlerb.rubyforge.org/

  • http://rubyforge.org/projects/seattlerb

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

  • Uses RubyInline, so it just drops in.
  • Includes SexpProcessor and CompositeSexpProcessor.
    • Allows you to write very clean filters.
  • Includes UnifiedRuby, allowing you to automatically rewrite ruby quirks.
  • ParseTree#parsetreefor_string lets you parse arbitrary strings of ruby.
  • Includes parsetreeshow, which lets you quickly snoop code.
    • echo "1+1" | parsetreeshow -f for quick snippet output.
  • Includes parsetreeabc, which lets you get abc metrics on code.
    • abc metrics = numbers of assignments, branches, and calls.
    • whitespace independent metric for method complexity.
  • Includes parsetreedeps, which shows you basic class level dependencies.
  • Does not work on the core classes, as they are not ruby (yet).

Changes:

  • 13 minor enhancements:

    • Added (partial) regexp flag support, currently numerical. ugh.
    • Added -a flag to parsetreeshow to turn on newline (all) nodes.
    • Added -r to parsetreeshow for raw arrays instead of sexps.
    • Added Unifier (SexpProcessor) class to unified_ruby.rb.
    • Added a ton of tests while working on ruby_parser.
    • Added ability to tell proc {} (nil arg slot) from proc {||} (0 arg slot)
    • Added context tracking to rewriting phase... slightly broken.
    • Added evstr support. (I hate evan)
    • Added usage for parsetreeshow.
    • Changed verbose to be true all the time in parsetreefor_string.
    • Removed process_level from SexpProcessor... just look at context size instead.
    • Revamped ParseTree. No more passing around newline. Pass around self instead.
    • I'm starting to dislike ruby's AST. It is REALLY inconsistent.
  • 6 bug fix:

    • SexpProcessor#assert_type now a bit safer with bad values.
    • Uncovered a bug in ruby (AST changes when -v used), added handler code.
    • Fixed NODE_BLOCK and massively simplified in the process.
    • Fixed rewrite_defs to deal with non-block asts.
    • Fixed test/unit hack so it does not die under miniunit.
    • Found a bug in PT where parsetreefor_string had some shadowed variables.
  • http://www.zenspider.com/ZSS/Products/ParseTree/

No Longer Secret

| | Comments (7)

I'm currently on a plane flying home from the bay area. I just spent the last two days participating in the latest rubinius sprint. (I had to go hem early to teach my ruby class).

So, we got the go ahead/mandate to drop the secrecy. Eric Hodel and I are joining Evan Phoenix at Engine Yard to work on rubinius. This is just about the most bad-ass job I could possibly get (my dreams of becoming an astronaut long-dashed against the rocks of horrible vision and only marginally better grades).

I'm only just digging in. Last night I made some changes to autotest that will make it possible to be used with mspec (rubinius' mini version of rspec). This should help accelerate development.

I'm also feverishly working on a new ruby parser written in pure ruby (+ racc) that outputs the same as ParseTree. I'm 99% done (measuring against my tests + dynamic tests generated from stdlib+all my rubygems). I hope to have a release soon. It does have a long way to go after correctness has been addressed. Namely, it is a port of a port and could use a lot of clean-up esp now that we have the true power of ruby behind it.

Anyhow. I'm terribly excited by this. I can't wait to see where we go next.

About this Archive

This page is a archive of recent entries in the ParseTree category.

MetaRuby is the previous category.

Planning is the next category.

Find recent content on the main index or look in the archives to find all content.

Pages

Powered by Movable Type 4.1