kotovalexarian-likes-github/jashkenas--coffeescript
1
0
Fork 0
mirror of https://github.com/jashkenas/coffeescript.git synced 2022-11-09 12:23:24 -05:00
Code Releases Activity
jashkenas--coffeescript/documentation/docs/rewriter.html
Jeremy Ashkenas 56eeeebed1 updating the docco
2010-03-09 23:53:21 -05:00

202 lines
No EOL
53 KiB
HTML
Raw Blame History

<!DOCTYPE html> <html> <head> <title>rewriter.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To &hellip; <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> rewriter.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-1">#</a> </div> <p>The CoffeeScript language has a decent amount of optional syntax,
implicit syntax, and shorthand syntax. These things can greatly complicate a
grammar and bloat the resulting parse table. Instead of making the parser
handle it all, we take a series of passes over the token stream,
using this <strong>Rewriter</strong> to convert shorthand into the unambiguous long form,
add implicit indentation and parentheses, balance incorrect nestings, and
generally clean things up.</p> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-2">#</a> </div> <p>Set up exported variables for both Node.js and the browser.</p> </td> <td class="code"> <div class="highlight"><pre><span class="k">if</span> <span class="nx">process</span><span class="o">?</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">mixin</span> <span class="nx">require</span> <span class="s1">&#39;./helpers&#39;</span>
<span class="k">else</span>
<span class="k">this</span><span class="p">.</span><span class="nv">exports: </span><span class="k">this</span></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-3">#</a> </div> <p>The <strong>Rewriter</strong> class is used by the <a href="lexer.html">Lexer</a>, directly against
its internal array of tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">exports.Rewriter: </span><span class="nx">class</span> <span class="nx">Rewriter</span></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-4">#</a> </div> <p>Rewrite the token stream in multiple passes, one logical filter at
a time. This could certainly be changed into a single pass through the
stream, with a big ol' efficient switch, but it's much nicer to work with
like this. The order of these passes matters -- indentation must be
corrected before implicit parentheses can be wrapped around blocks of code.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">rewrite: </span><span class="p">(</span><span class="nx">tokens</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="err">@</span><span class="nv">tokens: </span><span class="nx">tokens</span>
<span class="err">@</span><span class="nx">adjust_comments</span><span class="p">()</span>
<span class="err">@</span><span class="nx">remove_leading_newlines</span><span class="p">()</span>
<span class="err">@</span><span class="nx">remove_mid_expression_newlines</span><span class="p">()</span>
<span class="err">@</span><span class="nx">close_open_calls_and_indexes</span><span class="p">()</span>
<span class="err">@</span><span class="nx">add_implicit_indentation</span><span class="p">()</span>
<span class="err">@</span><span class="nx">add_implicit_parentheses</span><span class="p">()</span>
<span class="err">@</span><span class="nx">ensure_balance</span><span class="p">(</span><span class="nx">BALANCED_PAIRS</span><span class="p">)</span>
<span class="err">@</span><span class="nx">rewrite_closing_parens</span><span class="p">()</span>
<span class="err">@</span><span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-5">#</a> </div> <p>Rewrite the token stream, looking one token ahead and behind.
Allow the return value of the block to tell us how many tokens to move
forwards (or backwards) in the stream, to make sure we don't miss anything
as tokens are inserted and removed, and the stream changes length under
our feet.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">scan_tokens: </span><span class="p">(</span><span class="nx">block</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nv">i: </span><span class="mi">0</span>
<span class="k">while</span> <span class="kc">true</span>
<span class="k">break</span> <span class="nx">unless</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span>
<span class="nv">move: </span><span class="nx">block</span><span class="p">(</span><span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span> <span class="o">-</span> <span class="mi">1</span><span class="p">],</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">],</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">],</span> <span class="nx">i</span><span class="p">)</span>
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">move</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-6">#</a> </div> <p>Massage newlines and indentations so that comments don't have to be
correctly indented, or appear on a line of their own.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">adjust_comments: </span><span class="o">-&gt;</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="k">return</span> <span class="mi">1</span> <span class="nx">unless</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;COMMENT&#39;</span>
<span class="nv">after: </span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">2</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">after</span> <span class="o">and</span> <span class="nx">after</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;INDENT&#39;</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">after</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">&#39;TERMINATOR&#39;</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">&#39;INDENT&#39;</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">&#39;OUTDENT&#39;</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;TERMINATOR&#39;</span><span class="p">,</span> <span class="s2">&quot;\n&quot;</span><span class="p">,</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">2</span><span class="p">]])</span>
<span class="k">return</span> <span class="mi">2</span>
<span class="k">else</span>
<span class="k">return</span> <span class="mi">1</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-7">#</a> </div> <p>Leading newlines would introduce an ambiguity in the grammar, so we
dispatch them here.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">remove_leading_newlines: </span><span class="o">-&gt;</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">shift</span><span class="p">()</span> <span class="k">while</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">and</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;TERMINATOR&#39;</span></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-8">#</a> </div> <p>Some blocks occur in the middle of expressions -- when we're expecting
this, remove their trailing newlines.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">remove_mid_expression_newlines: </span><span class="o">-&gt;</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="k">return</span> <span class="mi">1</span> <span class="nx">unless</span> <span class="nx">post</span> <span class="o">and</span> <span class="nx">include</span><span class="p">(</span><span class="nx">EXPRESSION_CLOSE</span><span class="p">,</span> <span class="nx">post</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">and</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;TERMINATOR&#39;</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">0</span></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-9">#</a> </div> <p>The lexer has tagged the opening parenthesis of a method call, and the
opening bracket of an indexing operation. Match them with their paired
close.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">close_open_calls_and_indexes: </span><span class="o">-&gt;</span>
<span class="nv">parens: </span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nv">brackets: </span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="k">switch</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">when</span> <span class="s1">&#39;CALL_START&#39;</span> <span class="k">then</span> <span class="nx">parens</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">&#39;INDEX_START&#39;</span> <span class="k">then</span> <span class="nx">brackets</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">&#39;(&#39;</span> <span class="k">then</span> <span class="nx">parens</span><span class="p">[</span><span class="nx">parens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">when</span> <span class="s1">&#39;[&#39;</span> <span class="k">then</span> <span class="nx">brackets</span><span class="p">[</span><span class="nx">brackets</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">when</span> <span class="s1">&#39;)&#39;</span>
<span class="k">if</span> <span class="nx">parens</span><span class="p">[</span><span class="nx">parens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">is</span> <span class="mi">0</span>
<span class="nx">parens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">&#39;CALL_END&#39;</span>
<span class="k">else</span>
<span class="nx">parens</span><span class="p">[</span><span class="nx">parens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">when</span> <span class="s1">&#39;]&#39;</span>
<span class="k">if</span> <span class="nx">brackets</span><span class="p">[</span><span class="nx">brackets</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span>
<span class="nx">brackets</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">&#39;INDEX_END&#39;</span>
<span class="k">else</span>
<span class="nx">brackets</span><span class="p">[</span><span class="nx">brackets</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="mi">1</span></pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-10">#</a> </div> <p>Methods may be optionally called without parentheses, for simple cases.
Insert the implicit parentheses here, so that the parser doesn't have to
deal with them.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">add_implicit_parentheses: </span><span class="o">-&gt;</span>
<span class="nv">stack: </span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nv">calls: </span><span class="mi">0</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="nv">tag: </span><span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">switch</span> <span class="nx">tag</span>
<span class="k">when</span> <span class="s1">&#39;CALL_START&#39;</span> <span class="k">then</span> <span class="nx">calls</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">when</span> <span class="s1">&#39;CALL_END&#39;</span> <span class="k">then</span> <span class="nx">calls</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">when</span> <span class="s1">&#39;INDENT&#39;</span> <span class="k">then</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">&#39;OUTDENT&#39;</span>
<span class="nv">last: </span><span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="nx">last</span>
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;CALL_END&#39;</span> <span class="o">and</span> <span class="nx">calls</span> <span class="o">&lt;</span> <span class="mi">0</span>
<span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;CALL_END&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="nx">token</span><span class="p">[</span><span class="mi">2</span><span class="p">]])</span>
<span class="k">return</span> <span class="mi">2</span>
<span class="k">if</span> <span class="o">!</span><span class="nx">post</span><span class="o">?</span> <span class="o">or</span> <span class="nx">include</span> <span class="nx">IMPLICIT_END</span><span class="p">,</span> <span class="nx">tag</span>
<span class="k">return</span> <span class="mi">1</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;INDENT&#39;</span> <span class="o">and</span> <span class="nx">prev</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">IMPLICIT_BLOCK</span><span class="p">,</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="o">or</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;INDENT&#39;</span>
<span class="nv">idx: </span><span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;OUTDENT&#39;</span> <span class="k">then</span> <span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span> <span class="k">else</span> <span class="nx">i</span>
<span class="nv">stack_pointer: </span><span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;INDENT&#39;</span> <span class="k">then</span> <span class="mi">2</span> <span class="k">else</span> <span class="mi">1</span>
<span class="k">for</span> <span class="nx">tmp</span> <span class="k">in</span> <span class="p">[</span><span class="mi">0</span><span class="p">...</span><span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="nx">stack_pointer</span><span class="p">]]</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">idx</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;CALL_END&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="nx">token</span><span class="p">[</span><span class="mi">2</span><span class="p">]])</span>
<span class="nv">size: </span><span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="nx">stack_pointer</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span>
<span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="nx">stack_pointer</span><span class="p">]</span><span class="o">:</span> <span class="mi">0</span>
<span class="k">return</span> <span class="nx">size</span>
<span class="k">return</span> <span class="mi">1</span> <span class="nx">unless</span> <span class="nx">prev</span> <span class="o">and</span> <span class="nx">include</span><span class="p">(</span><span class="nx">IMPLICIT_FUNC</span><span class="p">,</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">IMPLICIT_CALL</span><span class="p">,</span> <span class="nx">tag</span>
<span class="nv">calls: </span><span class="mi">0</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;CALL_START&#39;</span><span class="p">,</span> <span class="s1">&#39;(&#39;</span><span class="p">,</span> <span class="nx">token</span><span class="p">[</span><span class="mi">2</span><span class="p">]])</span>
<span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="mi">2</span></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-11">#</a> </div> <p>Because our grammar is LALR(1), it can't handle some single-line
expressions that lack ending delimiters. The <strong>Rewriter</strong> adds the implicit
blocks, so it doesn't need to. ')' can close a single-line block,
but we need to make sure it's balanced.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">add_implicit_indentation: </span><span class="o">-&gt;</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="k">return</span> <span class="mi">1</span> <span class="nx">unless</span> <span class="nx">include</span><span class="p">(</span><span class="nx">SINGLE_LINERS</span><span class="p">,</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">and</span>
<span class="nx">post</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">&#39;INDENT&#39;</span> <span class="o">and</span>
<span class="o">not</span> <span class="p">(</span><span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;ELSE&#39;</span> <span class="o">and</span> <span class="nx">post</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;IF&#39;</span><span class="p">)</span>
<span class="nv">starter: </span><span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;INDENT&#39;</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="nx">token</span><span class="p">[</span><span class="mi">2</span><span class="p">]])</span>
<span class="nv">idx: </span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
<span class="nv">parens: </span><span class="mi">0</span>
<span class="k">while</span> <span class="kc">true</span>
<span class="nx">idx</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="nv">tok: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">idx</span><span class="p">]</span>
<span class="nv">pre: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">idx</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="p">(</span><span class="o">not</span> <span class="nx">tok</span> <span class="o">or</span>
<span class="p">(</span><span class="nx">include</span><span class="p">(</span><span class="nx">SINGLE_CLOSERS</span><span class="p">,</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">and</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">&#39;;&#39;</span><span class="p">)</span> <span class="o">or</span>
<span class="p">(</span><span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;)&#39;</span> <span class="o">&amp;&amp;</span> <span class="nx">parens</span> <span class="o">is</span> <span class="mi">0</span><span class="p">))</span> <span class="o">and</span>
<span class="o">not</span> <span class="p">(</span><span class="nx">starter</span> <span class="o">is</span> <span class="s1">&#39;ELSE&#39;</span> <span class="o">and</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;ELSE&#39;</span><span class="p">)</span>
<span class="nv">insertion: </span><span class="k">if</span> <span class="nx">pre</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s2">&quot;,&quot;</span> <span class="k">then</span> <span class="nx">idx</span> <span class="o">-</span> <span class="mi">1</span> <span class="k">else</span> <span class="nx">idx</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">insertion</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;OUTDENT&#39;</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="nx">token</span><span class="p">[</span><span class="mi">2</span><span class="p">]])</span>
<span class="k">break</span>
<span class="nx">parens</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">if</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;(&#39;</span>
<span class="nx">parens</span> <span class="o">-=</span> <span class="mi">1</span> <span class="k">if</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;)&#39;</span>
<span class="k">return</span> <span class="mi">1</span> <span class="nx">unless</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;THEN&#39;</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">0</span></pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-12">#</a> </div> <p>Ensure that all listed pairs of tokens are correctly balanced throughout
the course of the token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">ensure_balance: </span><span class="p">(</span><span class="nx">pairs</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nv">levels: </span><span class="p">{}</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="k">for</span> <span class="nx">pair</span> <span class="k">in</span> <span class="nx">pairs</span>
<span class="p">[</span><span class="nx">open</span><span class="p">,</span> <span class="nx">close</span><span class="p">]</span><span class="o">:</span> <span class="nx">pair</span>
<span class="nx">levels</span><span class="p">[</span><span class="nx">open</span><span class="p">]</span> <span class="o">||=</span> <span class="mi">0</span>
<span class="nx">levels</span><span class="p">[</span><span class="nx">open</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">if</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="nx">open</span>
<span class="nx">levels</span><span class="p">[</span><span class="nx">open</span><span class="p">]</span> <span class="o">-=</span> <span class="mi">1</span> <span class="k">if</span> <span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="nx">close</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">&quot;too many ${token[1]} on line ${token[2] + 1}&quot;</span><span class="p">)</span> <span class="k">if</span> <span class="nx">levels</span><span class="p">[</span><span class="nx">open</span><span class="p">]</span> <span class="o">&lt;</span> <span class="mi">0</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="nv">unclosed: </span><span class="nx">key</span> <span class="k">for</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">value</span> <span class="k">of</span> <span class="nx">levels</span> <span class="k">when</span> <span class="nx">value</span> <span class="o">&gt;</span> <span class="mi">0</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">&quot;unclosed ${unclosed[0]}&quot;</span><span class="p">)</span> <span class="k">if</span> <span class="nx">unclosed</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-13">#</a> </div> <p>We'd like to support syntax like this:</p>
<pre><code>el.click((event) -&gt;
el.hide())
</code></pre>
<p>In order to accomplish this, move outdents that follow closing parens
inwards, safely. The steps to accomplish this are:</p>
<ol>
<li>Check that all paired tokens are balanced and in order.</li>
<li>Rewrite the stream with a stack: if you see an '(' or INDENT, add it
to the stack. If you see an ')' or OUTDENT, pop the stack and replace
it with the inverse of what we've just popped.</li>
<li>Keep track of "debt" for tokens that we fake, to make sure we end
up balanced in the end.</li>
</ol> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">rewrite_closing_parens: </span><span class="o">-&gt;</span>
<span class="nv">stack: </span><span class="p">[]</span>
<span class="nv">debt: </span> <span class="p">{}</span>
<span class="p">(</span><span class="nx">debt</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span><span class="o">:</span> <span class="mi">0</span><span class="p">)</span> <span class="k">for</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">val</span> <span class="k">of</span> <span class="nx">INVERSES</span>
<span class="err">@</span><span class="nx">scan_tokens</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">post</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="o">=&gt;</span>
<span class="nv">tag: </span><span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="nv">inv: </span><span class="nx">INVERSES</span><span class="p">[</span><span class="nx">token</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span>
<span class="k">if</span> <span class="nx">include</span> <span class="nx">EXPRESSION_START</span><span class="p">,</span> <span class="nx">tag</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nx">token</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">EXPRESSION_END</span><span class="p">,</span> <span class="nx">tag</span>
<span class="k">if</span> <span class="nx">debt</span><span class="p">[</span><span class="nx">inv</span><span class="p">]</span> <span class="o">&gt;</span> <span class="mi">0</span>
<span class="nx">debt</span><span class="p">[</span><span class="nx">inv</span><span class="p">]</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span> <span class="nx">i</span><span class="p">,</span> <span class="mi">1</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="k">else</span>
<span class="nv">match: </span><span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nv">mtag: </span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">return</span> <span class="mi">1</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="nx">INVERSES</span><span class="p">[</span><span class="nx">mtag</span><span class="p">]</span>
<span class="nx">debt</span><span class="p">[</span><span class="nx">mtag</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="nv">val: </span><span class="k">if</span> <span class="nx">mtag</span> <span class="o">is</span> <span class="s1">&#39;INDENT&#39;</span> <span class="k">then</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">else</span> <span class="nx">INVERSES</span><span class="p">[</span><span class="nx">mtag</span><span class="p">]</span>
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span> <span class="nx">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="nx">INVERSES</span><span class="p">[</span><span class="nx">mtag</span><span class="p">],</span> <span class="nx">val</span><span class="p">]</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="k">else</span>
<span class="k">return</span> <span class="mi">1</span></pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-14">#</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-15">#</a> </div> <p>List of the token pairs that must be balanced.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">BALANCED_PAIRS: </span><span class="p">[[</span><span class="s1">&#39;(&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;[&#39;</span><span class="p">,</span> <span class="s1">&#39;]&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;{&#39;</span><span class="p">,</span> <span class="s1">&#39;}&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;INDENT&#39;</span><span class="p">,</span> <span class="s1">&#39;OUTDENT&#39;</span><span class="p">],</span>
<span class="p">[</span><span class="s1">&#39;PARAM_START&#39;</span><span class="p">,</span> <span class="s1">&#39;PARAM_END&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;CALL_START&#39;</span><span class="p">,</span> <span class="s1">&#39;CALL_END&#39;</span><span class="p">],</span>
<span class="p">[</span><span class="s1">&#39;INDEX_START&#39;</span><span class="p">,</span> <span class="s1">&#39;INDEX_END&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;SOAKED_INDEX_START&#39;</span><span class="p">,</span> <span class="s1">&#39;SOAKED_INDEX_END&#39;</span><span class="p">]]</span></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-16">#</a> </div> <p>The inverse mappings of <code>BALANCED_PAIRS</code> we're trying to fix up, so we can
look things up from either end.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">INVERSES: </span><span class="p">{}</span>
<span class="k">for</span> <span class="nx">pair</span> <span class="k">in</span> <span class="nx">BALANCED_PAIRS</span>
<span class="nx">INVERSES</span><span class="p">[</span><span class="nx">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span><span class="o">:</span> <span class="nx">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="nx">INVERSES</span><span class="p">[</span><span class="nx">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span><span class="o">:</span> <span class="nx">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-17">#</a> </div> <p>The tokens that signal the start of a balanced pair.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">EXPRESSION_START: </span><span class="nx">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="nx">pair</span> <span class="k">in</span> <span class="nx">BALANCED_PAIRS</span></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-18">#</a> </div> <p>The tokens that signal the end of a balanced pair.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">EXPRESSION_END: </span> <span class="nx">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="nx">pair</span> <span class="k">in</span> <span class="nx">BALANCED_PAIRS</span></pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-19">#</a> </div> <p>Tokens that indicate the close of a clause of an expression.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">EXPRESSION_CLOSE: </span><span class="p">[</span><span class="s1">&#39;CATCH&#39;</span><span class="p">,</span> <span class="s1">&#39;WHEN&#39;</span><span class="p">,</span> <span class="s1">&#39;ELSE&#39;</span><span class="p">,</span> <span class="s1">&#39;FINALLY&#39;</span><span class="p">].</span><span class="nx">concat</span> <span class="nx">EXPRESSION_END</span></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-20">#</a> </div> <p>Tokens that, if followed by an <code>IMPLICIT_CALL</code>, indicate a function invocation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IMPLICIT_FUNC: </span> <span class="p">[</span><span class="s1">&#39;IDENTIFIER&#39;</span><span class="p">,</span> <span class="s1">&#39;SUPER&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="s1">&#39;CALL_END&#39;</span><span class="p">,</span> <span class="s1">&#39;]&#39;</span><span class="p">,</span> <span class="s1">&#39;INDEX_END&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-21">#</a> </div> <p>If preceded by an <code>IMPLICIT_FUNC</code>, indicates a function invocation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IMPLICIT_CALL: </span> <span class="p">[</span><span class="s1">&#39;IDENTIFIER&#39;</span><span class="p">,</span> <span class="s1">&#39;NUMBER&#39;</span><span class="p">,</span> <span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s1">&#39;JS&#39;</span><span class="p">,</span> <span class="s1">&#39;REGEX&#39;</span><span class="p">,</span> <span class="s1">&#39;NEW&#39;</span><span class="p">,</span> <span class="s1">&#39;PARAM_START&#39;</span><span class="p">,</span>
<span class="s1">&#39;TRY&#39;</span><span class="p">,</span> <span class="s1">&#39;DELETE&#39;</span><span class="p">,</span> <span class="s1">&#39;TYPEOF&#39;</span><span class="p">,</span> <span class="s1">&#39;SWITCH&#39;</span><span class="p">,</span> <span class="s1">&#39;EXTENSION&#39;</span><span class="p">,</span>
<span class="s1">&#39;TRUE&#39;</span><span class="p">,</span> <span class="s1">&#39;FALSE&#39;</span><span class="p">,</span> <span class="s1">&#39;YES&#39;</span><span class="p">,</span> <span class="s1">&#39;NO&#39;</span><span class="p">,</span> <span class="s1">&#39;ON&#39;</span><span class="p">,</span> <span class="s1">&#39;OFF&#39;</span><span class="p">,</span> <span class="s1">&#39;!&#39;</span><span class="p">,</span> <span class="s1">&#39;!!&#39;</span><span class="p">,</span> <span class="s1">&#39;NOT&#39;</span><span class="p">,</span>
<span class="s1">&#39;@&#39;</span><span class="p">,</span> <span class="s1">&#39;-&gt;&#39;</span><span class="p">,</span> <span class="s1">&#39;=&gt;&#39;</span><span class="p">,</span> <span class="s1">&#39;[&#39;</span><span class="p">,</span> <span class="s1">&#39;(&#39;</span><span class="p">,</span> <span class="s1">&#39;{&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-22">#</a> </div> <p>Tokens indicating that the implicit call must enclose a block of expressions.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IMPLICIT_BLOCK: </span><span class="p">[</span><span class="s1">&#39;-&gt;&#39;</span><span class="p">,</span> <span class="s1">&#39;=&gt;&#39;</span><span class="p">,</span> <span class="s1">&#39;{&#39;</span><span class="p">,</span> <span class="s1">&#39;[&#39;</span><span class="p">,</span> <span class="s1">&#39;,&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-23">#</a> </div> <p>Tokens that always mark the end of an implicit call for single-liners.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IMPLICIT_END: </span> <span class="p">[</span><span class="s1">&#39;IF&#39;</span><span class="p">,</span> <span class="s1">&#39;UNLESS&#39;</span><span class="p">,</span> <span class="s1">&#39;FOR&#39;</span><span class="p">,</span> <span class="s1">&#39;WHILE&#39;</span><span class="p">,</span> <span class="s1">&#39;TERMINATOR&#39;</span><span class="p">,</span> <span class="s1">&#39;INDENT&#39;</span><span class="p">,</span> <span class="s1">&#39;OUTDENT&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-24">#</a> </div> <p>Single-line flavors of block expressions that have unclosed endings.
The grammar can't disambiguate them, so we insert the implicit indentation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">SINGLE_LINERS: </span><span class="p">[</span><span class="s1">&#39;ELSE&#39;</span><span class="p">,</span> <span class="s2">&quot;-&gt;&quot;</span><span class="p">,</span> <span class="s2">&quot;=&gt;&quot;</span><span class="p">,</span> <span class="s1">&#39;TRY&#39;</span><span class="p">,</span> <span class="s1">&#39;FINALLY&#39;</span><span class="p">,</span> <span class="s1">&#39;THEN&#39;</span><span class="p">]</span>
<span class="nv">SINGLE_CLOSERS: </span><span class="p">[</span><span class="s1">&#39;TERMINATOR&#39;</span><span class="p">,</span> <span class="s1">&#39;CATCH&#39;</span><span class="p">,</span> <span class="s1">&#39;FINALLY&#39;</span><span class="p">,</span> <span class="s1">&#39;ELSE&#39;</span><span class="p">,</span> <span class="s1">&#39;OUTDENT&#39;</span><span class="p">,</span> <span class="s1">&#39;LEADING_WHEN&#39;</span><span class="p">]</span>
</pre></div> </td> </tr> </tbody> </table> </div> </body> </html>
View git blame Copy permalink