Recently in ParseTree Category

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.6 / 2010-09-01

• 1 bug fix:

  • fixed requires (e-tobi)

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.5 / 2010-03-27

• 1 bug fix:

  • Fixed NODE_METHOD but also commented that you shouldn't ever see it naturally (tmm1)

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.4 / 2009-06-23

• 4 minor enhancements:

  • Pulled unique.rb from ParseTree to sexp_processor.
  • Switched to minitest
  • UnifiedRuby strips flip2 back down to expected values
  • Updated Rakefile for new hoe

• 2 bug fixes:

  • 1.8.6p287 & 1.8.7 changed flip2 with fixnum start. fixed
  • 1.8.6p287 & 1.8.7 removed the begin node. fixed.

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

There seems to be some confusion and/or panic about ParseTree that I'd like to clear up:

Because of changes to internals in 1.9, ParseTree simply can not work. I asked for hooks/options to allow us to get to the information but they never arrived.

Specifically if you're using ParseTree to access the AST of a live method/block/proc, you're SOL. If you're just using ParseTree to do static analysis, then you can switch to ruby_parser in about a minute of work and you're good to go.

Here is the plan for my projects:

• RubyToC-1.0.0.5 = switching to ruby_parser
• flog-2.1.0 = switching to ruby_parser
• heckle-1.4.2 = dead, unless we can think of something soon.
• ruby2ruby-1.2.2 = dropping block/proc support.
• ruby_parser-2.0.2 = no clue why it has a dependency still, no worries here.
• ZenHacks-1.0.1 = dropping block/proc support -- not that I support this.

I don't know the status of most of the projects dependent on PT or how they're going to deal with this issue. If you use one of these projects directly or indirectly and that project relies on PT for live method/block/proc, then you're probably going to be stuck on 1.8 for a while:

• SuperCaller-1.0.0 = unsupported - can drop live method support
• ambition-0.5.4
• argible-0.1.1
• integrity-0.1.9.3
• merb-action-args-1.0.11
• nitpick-1.0.2
• protocol-0.8.1
• red-4.1.7
• reek-1.0.0 = probably doesn't need live analysis
• roodi-1.3.5
• ruby_diff-0.2
• rubyjs-0.8.0
• sake-1.0.15
• thorero-action-args-0.9.4

There are probably other projects out there dependent on PT that I don't know about. All I've listed here are rubyforge gems that depend on PT in their gemspec.

You can check your gems using this:

gem list | egrep "(ambition|argible|integrity|merb-action-args|nitpick|protocol|red|reek|roodi|ruby_diff|rubyjs|sake|thorero-action-args)"

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.3 / 2009-01-20

• 4 bug fixes:

  • Added gauntlet_parsetree.rb. Bug finding/fixing much faster & easier now.
  • Fixed alias tests for r2r. Added masgn and heredoc tests.
  • Fixed conflicting flags (-u vs -r).
  • Unwrap RHS from array IF it is only a splat node.

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

% gauntlet report parsetree
 47369 93.73% good
  2011  3.98% bad
  1160  2.30% skip
...
% gauntlet parse_tree
...
keyword_prospector-0.8.1
  setup.rb: good
  ALL GOOD!
kickstart-0.2.2.1
  lib/extentions/array.rb: good
  lib/extentions/class.rb: bad
KirbyBase-2.6
  lib/kirbybase.rb: bad
...

the fact that I can incrementally fix something and then run it against the known failures makes doing compatibility work via gauntlet pure joy. I've been using gauntlet to compare parsetree to rubyparser and to do round trips between rubyparser and ruby2ruby. It's finding _a lot of stuff for me to work on.

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.2 / 2008-11-04

• 1 minor enhancement:

  • parsetreeshow now defaults to unified, added -U to UNunify output.

• 4 bug fixes:

  • UnifiedRuby#rewrite_defn kinda sorta deals with cfunc now. Stupid rails.
  • Fixed ambiguities in splatted nodes in a bunch of contexts.
  • Fixed 1 arg form of parsetreefor_proc
  • KNOWN ISSUE: 0 arg form (||) of parsetreefor_proc segfaults.

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.1 / 2008-10-23

• 1 bug fix

  • Fixed the dependency on sexp_processor (accidentally camelcased). thanks ez!

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

Flog reports the most tortured code in an easy to read pain report. The higher the score, the more pain the code is in.

Changes:

1.2.0 / 2008-10-22

• 14 minor enhancements:

  • Added -c flag to continue dispite errors.
  • Added -m to only report code in methods (skips #none).
  • Added -n flag to give NO method details (summary only)
  • Added -n to skip method details... pussies should learn grep.
  • Added -q to quiet method details (total per method only)
  • Added avg & stddev to total.
  • Added avg score per method to report.
  • Added lots of doco from contributors (hugh sasse?).
  • Fixed class names when const2/3.
  • Fixed unified ruby changes
  • Refactored flog with help from flay.
  • Refactored getsourceindex
  • Refactored into gem_updater.rb and cleaned up.
  • Works with new incremental rubygems, albiet slower than before.

• http://ruby.sadi.st/

• http://rubyforge.org/projects/seattlerb

ruby_parser (RP) is a ruby parser written in pure ruby (utilizing racc--which does by default use a C extension). RP's output is the same as ParseTree's output: s-expressions using ruby's arrays and base types.

Changes:

2.0.0 / 2008-10-22

• 1 major enhancement

  • Brought on the AWESOME! 4x faster! no known lexing/parsing bugs!

• 71 minor enhancements

  • 1.9: Added Fixnum#ord.
  • 1.9: Added missing Regexp constants and did it so it'd work on 1.9.
  • Added #store_comment and #comments
  • Added StringScanner #beginofline?
  • Added a bunch of tests for regexp escape chars, #parsestring, #readescape, ? numbers, ? whitespace.
  • Added a hack for rubinius' r2l eval bug.
  • Added a new token type tSTRING that bypasses tSTRING_BEG/END entirely. Only does non-interpolated strings and then falls back to the old way. MUCH cleaner tho.
  • Added bin/ruby_parse
  • Added compare rule to Rakefile.
  • Added coverage files/dirs to clean rule.
  • Added file and line numbers to all sexp nodes. Column/ranges to come.
  • Added lex_state change for lvars at the end of yylex.
  • Added lexed comments to defn/defs/class/module nodes.
  • Added stats gathering for yylex. Reordered yylex for avg data
  • Added tSYMBOL token type and parser rule to speed up symbol lexing.
  • Added tally output for getch, unread, and unread_many.
  • Added tests for ambigous uminus/uplus, backtick in cmdarg, square and curly brackets, numeric gvars, eos edge cases, string quoting %<> and %%%.
  • All cases throughout yylex now return directly if they match, no passthroughs.
  • All lexer cases now slurp entire token in one swoop.
  • All zarrays are now just empty arrays.
  • Changed s(:block_arg, :blah) to :"&blah" in args sexp.
  • Cleaned up lexer error handling. Now just raises all over.
  • Cleaned up readescape and regxoptions
  • Cleaned up tokadd_string (for some definition of cleaned).
  • Converted single quoted strings to new tSTRING token type.
  • Coverage is currently 94.4% on lexer.
  • Done what I can to clean up heredoc lexing... still sucks.
  • Flattened resbodies in rescue node. Fixed .autotest file.
  • Folded lex_keywords back in now that it screams.
  • Found very last instanceof ILiteralNode in the code. haha!
  • Got the tests subclassing PTTC and cleaned up a lot. YAY
  • Handle yield(*ary) properly
  • MASSIVELY cleaned out =begin/=end comment processor.
  • Massive overhaul on Keyword class. All hail the mighty Hash!
  • Massively cleaned up ident= edge cases and fixed a stupid bug from jruby.
  • Merged @/@@ scanner together, going to try to do the same everywhere.
  • Refactored fixarglex_state, common across the lexer.
  • Refactored newfcall into newcall.
  • Refactored some code to get better profile numbers.
  • Refactored some more #fixarglex_state.
  • Refactored tail of yylex into its own method.
  • Removed Module#kill
  • Removed Token, replaced with Sexp.
  • Removed all parsenumber and parsequote tests.
  • Removed argspush, argscat. YAY!
  • Removed as many token_buffer.split(//)'s as possible. 1 to go.
  • Removed begins from compstmts
  • Removed buffer arg for tokadd_string.
  • Removed crufty (?) solo '@' token... wtf was that anyhow?
  • Removed most jruby/stringio cruft from StringScanner.
  • Removed one unread_many... 2 to go. They're harder.
  • Removed store_comment, now done directly.
  • Removed token_buffer. Now I just use token ivar.
  • Removed use of s() from lexer. Changed the way line numbers are gathered.
  • Renamed *qwords to *awords.
  • Renamed StringScanner to RPStringScanner (a subclass) to fix namespace trashing.
  • Renamed parse to process and aliased to parse.
  • Renamed tokenbuffer to stringbuffer since that arcane shit still needs it.
  • Resolved the rest of the lexing issues I brought up w/ ruby-core.
  • Revamped tokadd_escape.
  • Rewrote Keyword and KWtable.
  • Rewrote RubyLexer using StringScanner.
  • Rewrote tokadd_escape. 79 lines down to 21.
  • Split out lib/rubyparserextras.rb so lexer is standalone.
  • Started to clean up the parser and make it as skinny as possible
  • Stripped out as much code as possible.
  • Stripped yylex of some dead code.
  • Switched from StringIO to StringScanner.
  • Updated rakefile for new hoe.
  • Uses pure ruby racc if ENV['PURE_RUBY'], otherwise use c.
  • Wrote a ton of lexer tests. Coverage is as close to 100% as possible.
  • Wrote args to clean up the big nasty args processing grammar section.
  • lex_strterm is now a plain array, removed RubyLexer#s(...).
  • yield and super now flatten args.

• 21+ bug fixes:

  • I'm sure this list is missing a lot:
  • Fixed 2 bugs both involving attrasgn (and ilk) esp when lhs is an array.
  • Fixed a bug in the lexer for strings with single digit hex escapes.
  • Fixed a bug parsing: a (args) { expr }... the space caused a different route to be followed and all hell broke loose.
  • Fixed a bug with x\n=beginvar not putting begin back.
  • Fixed attrasgn to have arglists, not arrays.
  • Fixed bug in defn/defs with block fixing.
  • Fixed class/module's name slot if colon2/3.
  • Fixed dstr with empty interpolation body.
  • Fixed for 1.9 string/char changes.
  • Fixed lexer BS wrt determining token type of words.
  • Fixed lexer BS wrt pass through values and lexing words. SO STUPID.
  • Fixed lexing of floats.
  • Fixed lexing of identifiers followed by equals. I hope.
  • Fixed masgn with splat on lhs
  • Fixed newsuper to deal with blockpass correctly.
  • Fixed parser's treatment of :colon2 and :colon3.
  • Fixed regexp scanning of escaped numbers, ANY number is valid, not just octs.
  • Fixed string scanning of escaped octs, allowing 1-3 chars.
  • Fixed unescape for \n
  • Fixed: omg this is stupid. '()' was returning bare nil
  • Fixed: remove_begin now goes to the end, not sure why it didn't before.