ParseTree: October 2008 Archives

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.1 / 2008-10-23

• 1 bug fix

  • Fixed the dependency on sexp_processor (accidentally camelcased). thanks ez!

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com

Flog reports the most tortured code in an easy to read pain report. The higher the score, the more pain the code is in.

Changes:

1.2.0 / 2008-10-22

• 14 minor enhancements:

  • Added -c flag to continue dispite errors.
  • Added -m to only report code in methods (skips #none).
  • Added -n flag to give NO method details (summary only)
  • Added -n to skip method details... pussies should learn grep.
  • Added -q to quiet method details (total per method only)
  • Added avg & stddev to total.
  • Added avg score per method to report.
  • Added lots of doco from contributors (hugh sasse?).
  • Fixed class names when const2/3.
  • Fixed unified ruby changes
  • Refactored flog with help from flay.
  • Refactored getsourceindex
  • Refactored into gem_updater.rb and cleaned up.
  • Works with new incremental rubygems, albiet slower than before.

• http://ruby.sadi.st/

• http://rubyforge.org/projects/seattlerb

ruby_parser (RP) is a ruby parser written in pure ruby (utilizing racc--which does by default use a C extension). RP's output is the same as ParseTree's output: s-expressions using ruby's arrays and base types.

Changes:

2.0.0 / 2008-10-22

• 1 major enhancement

  • Brought on the AWESOME! 4x faster! no known lexing/parsing bugs!

• 71 minor enhancements

  • 1.9: Added Fixnum#ord.
  • 1.9: Added missing Regexp constants and did it so it'd work on 1.9.
  • Added #store_comment and #comments
  • Added StringScanner #beginofline?
  • Added a bunch of tests for regexp escape chars, #parsestring, #readescape, ? numbers, ? whitespace.
  • Added a hack for rubinius' r2l eval bug.
  • Added a new token type tSTRING that bypasses tSTRING_BEG/END entirely. Only does non-interpolated strings and then falls back to the old way. MUCH cleaner tho.
  • Added bin/ruby_parse
  • Added compare rule to Rakefile.
  • Added coverage files/dirs to clean rule.
  • Added file and line numbers to all sexp nodes. Column/ranges to come.
  • Added lex_state change for lvars at the end of yylex.
  • Added lexed comments to defn/defs/class/module nodes.
  • Added stats gathering for yylex. Reordered yylex for avg data
  • Added tSYMBOL token type and parser rule to speed up symbol lexing.
  • Added tally output for getch, unread, and unread_many.
  • Added tests for ambigous uminus/uplus, backtick in cmdarg, square and curly brackets, numeric gvars, eos edge cases, string quoting %<> and %%%.
  • All cases throughout yylex now return directly if they match, no passthroughs.
  • All lexer cases now slurp entire token in one swoop.
  • All zarrays are now just empty arrays.
  • Changed s(:block_arg, :blah) to :"&blah" in args sexp.
  • Cleaned up lexer error handling. Now just raises all over.
  • Cleaned up readescape and regxoptions
  • Cleaned up tokadd_string (for some definition of cleaned).
  • Converted single quoted strings to new tSTRING token type.
  • Coverage is currently 94.4% on lexer.
  • Done what I can to clean up heredoc lexing... still sucks.
  • Flattened resbodies in rescue node. Fixed .autotest file.
  • Folded lex_keywords back in now that it screams.
  • Found very last instanceof ILiteralNode in the code. haha!
  • Got the tests subclassing PTTC and cleaned up a lot. YAY
  • Handle yield(*ary) properly
  • MASSIVELY cleaned out =begin/=end comment processor.
  • Massive overhaul on Keyword class. All hail the mighty Hash!
  • Massively cleaned up ident= edge cases and fixed a stupid bug from jruby.
  • Merged @/@@ scanner together, going to try to do the same everywhere.
  • Refactored fixarglex_state, common across the lexer.
  • Refactored newfcall into newcall.
  • Refactored some code to get better profile numbers.
  • Refactored some more #fixarglex_state.
  • Refactored tail of yylex into its own method.
  • Removed Module#kill
  • Removed Token, replaced with Sexp.
  • Removed all parsenumber and parsequote tests.
  • Removed argspush, argscat. YAY!
  • Removed as many token_buffer.split(//)'s as possible. 1 to go.
  • Removed begins from compstmts
  • Removed buffer arg for tokadd_string.
  • Removed crufty (?) solo '@' token... wtf was that anyhow?
  • Removed most jruby/stringio cruft from StringScanner.
  • Removed one unread_many... 2 to go. They're harder.
  • Removed store_comment, now done directly.
  • Removed token_buffer. Now I just use token ivar.
  • Removed use of s() from lexer. Changed the way line numbers are gathered.
  • Renamed *qwords to *awords.
  • Renamed StringScanner to RPStringScanner (a subclass) to fix namespace trashing.
  • Renamed parse to process and aliased to parse.
  • Renamed tokenbuffer to stringbuffer since that arcane shit still needs it.
  • Resolved the rest of the lexing issues I brought up w/ ruby-core.
  • Revamped tokadd_escape.
  • Rewrote Keyword and KWtable.
  • Rewrote RubyLexer using StringScanner.
  • Rewrote tokadd_escape. 79 lines down to 21.
  • Split out lib/rubyparserextras.rb so lexer is standalone.
  • Started to clean up the parser and make it as skinny as possible
  • Stripped out as much code as possible.
  • Stripped yylex of some dead code.
  • Switched from StringIO to StringScanner.
  • Updated rakefile for new hoe.
  • Uses pure ruby racc if ENV['PURE_RUBY'], otherwise use c.
  • Wrote a ton of lexer tests. Coverage is as close to 100% as possible.
  • Wrote args to clean up the big nasty args processing grammar section.
  • lex_strterm is now a plain array, removed RubyLexer#s(...).
  • yield and super now flatten args.

• 21+ bug fixes:

  • I'm sure this list is missing a lot:
  • Fixed 2 bugs both involving attrasgn (and ilk) esp when lhs is an array.
  • Fixed a bug in the lexer for strings with single digit hex escapes.
  • Fixed a bug parsing: a (args) { expr }... the space caused a different route to be followed and all hell broke loose.
  • Fixed a bug with x\n=beginvar not putting begin back.
  • Fixed attrasgn to have arglists, not arrays.
  • Fixed bug in defn/defs with block fixing.
  • Fixed class/module's name slot if colon2/3.
  • Fixed dstr with empty interpolation body.
  • Fixed for 1.9 string/char changes.
  • Fixed lexer BS wrt determining token type of words.
  • Fixed lexer BS wrt pass through values and lexing words. SO STUPID.
  • Fixed lexing of floats.
  • Fixed lexing of identifiers followed by equals. I hope.
  • Fixed masgn with splat on lhs
  • Fixed newsuper to deal with blockpass correctly.
  • Fixed parser's treatment of :colon2 and :colon3.
  • Fixed regexp scanning of escaped numbers, ANY number is valid, not just octs.
  • Fixed string scanning of escaped octs, allowing 1-3 chars.
  • Fixed unescape for \n
  • Fixed: omg this is stupid. '()' was returning bare nil
  • Fixed: remove_begin now goes to the end, not sure why it didn't before.

FIX (describe your package)

Changes:

3.0.0 / 2008-10-22

• 2 major enhancements:

  • Released as its own project, splitting from ParseTree
  • Added Environment to SexpProcessor and built it in. YAY!

• 6 minor enhancements:

  • Allowed CompositeSexpProcessor to be more ducktypey.
  • Refactored Sexp#methodmissing into findnode and find_nodes.
  • Removed Sexp#for and other PT specific code.
  • SexpProcessor#process now runs rewriters before everything else.
  • SexpProcessor#rewrite context only for subs, EMPTY for top level rewrites.
  • SexpProcessor#rewrite will stop iterating if the result isn't another Sexp.

ruby2ruby provides a means of generating pure ruby code easily from ParseTree's Sexps. This makes making dynamic language processors much easier in ruby than ever before.

Changes:

1.2.0 / 2008-10-22

• 2 minor enhancements:

  • Removed all PT dependent code to PT project (see parsetreeextensions.rb).
  • Revamped. Got working with new unified ruby output. Much much cleaner.

• http://seattlerb.rubyforge.org/

• http://rubyforge.org/projects/seattlerb

ParseTree is a C extension (using RubyInline) that extracts the parse tree for an entire class or a specific method and returns it as a s-expression (aka sexp) using ruby's arrays, strings, symbols, and integers.

As an example:

def conditional1(arg1) if arg1 == 0 then return 1 end return 0 end

becomes:

[:defn, :conditional1, [:scope, [:block, [:args, :arg1], [:if, [:call, [:lvar, :arg1], :==, [:array, [:lit, 0]]], [:return, [:lit, 1]], nil], [:return, [:lit, 0]]]]]

Changes:

3.0.0 / 2008-10-22

• 3 major enhancements:

  • Split out sexp_processor project.
  • ParseTree#process outputs a UnifiedRuby parse tree.
  • Switched all ParseTree tests to RawParseTree tests.
  • Added ParseTree tests.
  • Added UnifiedRuby#process.

• 28 minor enhancements:

  • PTTC: Added a comprehensive suite of tests for every combo of defn args.
  • PTTC: Added a ton of new tests.
  • PTTC: Added extra tests for mri verbose flag testing (horrid horrid bug imhfo)
  • PTTC: Don't generate tests if class has "TestCase" in the class name.
  • PTTC: Made self.previous more overridable.
  • PTTC: Now all tests that have "mriverboseflag" activate $VERBOSE
  • PTTC: Officially added clonesame phase to generatetests
  • PTTC: Refactored test generation to make subclasses more flexible.
  • PTTC: Refactored to use new add_tests. Much much cleaner.
  • PTTC: Removed Unique from PTTC... subclasses need to deal with this
  • PTTC: Removed some stupid tests.
  • PTTC: Renamed a bunch of tests to be more uniform.
  • UR: Added rewrite_attrasgn to normalize arg array into arglist.
  • UR: Added rewriteopasgn1 to ensure the arglist is an arglist.
  • UR: Flattened resbodies in rescue node.
  • UR: Flattened super and yield args yet, deal with array arg properly.
  • UR: Handle yield(*ary) properly.
  • UR: Nuked argspush.
  • UR: Removed begin node from unified sexps.
  • UR: Removed block from rewritten resbody if only 1 element.
  • UR: Removed dasgn, dasgn_curr, and dvar nodes.
  • UR: Removed s(:block_arg, :block) in favor of :"&block"
  • UR: Rewrote argscat to be a splat inside an array.
  • UR: Rewrote block_pass into arglist of inner call/super.
  • UR: Rewrote call to ALWAYS have an argslist (no more splat as arglist).
  • UR: Rewrote rewrite_resbody from scratch... MUUCH cleaner.
  • UR: zarray -> array
  • Used Hoe#addincludedirs to clean up rakefile.
  • Moved PT dependent code from ruby2ruby to here (eg Proc#to_sexp & friends).

• 2 bug fixes:

  • UR: Fixed a bug with resbody that starts with an lasgn.
  • UR: Fixed args for attrset
  • Fixed export of symbols to work across platforms.

• http://rubyforge.org/projects/parsetree/

• http://www.zenspider.com/ZSS/Products/ParseTree/
• ryand-ruby@zenspider.com