508 lines
112 KiB
HTML
508 lines
112 KiB
HTML
<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To … <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> lexer.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt
|
|
matches against the beginning of the source code. When a match is found,
|
|
a token is produced, we consume the match, and start again. Tokens are in the
|
|
form:</p>
|
|
|
|
<pre><code>[tag, value, lineNumber]
|
|
</code></pre>
|
|
|
|
<p>Which is a format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">Rewriter</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span> <span class="s1">'./rewriter'</span></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> <p>Import the helpers we need.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">count</span><span class="p">,</span> <span class="nx">starts</span><span class="p">,</span> <span class="nx">compact</span><span class="p">,</span> <span class="nx">last</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span> <span class="s1">'./helpers'</span></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> <h2>The Lexer Class</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged
|
|
tokens. Some potential ambiguity in the grammar has been avoided by
|
|
pushing some extra smarts into the Lexer.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">exports.Lexer = </span><span class="nx">class</span> <span class="nx">Lexer</span></pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> <p><strong>tokenize</strong> is the Lexer's main method. Scan by attempting to match tokens
|
|
one at a time, using a regular expression anchored at the start of the
|
|
remaining code, or a custom recursive token-matching method
|
|
(for interpolations). When the next token has been recorded, we move forward
|
|
within the code past the token, and begin again.</p>
|
|
|
|
<p>Each tokenizing method is responsible for returning the number of characters
|
|
it has consumed.</p>
|
|
|
|
<p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>
|
|
unless explicitly asked not to.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tokenize</span><span class="o">:</span> <span class="p">(</span><span class="nx">code</span><span class="p">,</span> <span class="nv">opts = </span><span class="p">{})</span> <span class="o">-></span>
|
|
<span class="nv">code = </span><span class="s2">"\n#{code}"</span> <span class="k">if</span> <span class="nx">WHITESPACE</span><span class="p">.</span><span class="nx">test</span> <span class="nx">code</span>
|
|
<span class="nv">code = </span><span class="nx">code</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/\r/g</span><span class="p">,</span> <span class="s1">''</span><span class="p">).</span><span class="nx">replace</span> <span class="nx">TRAILING_SPACES</span><span class="p">,</span> <span class="s1">''</span>
|
|
|
|
<span class="vi">@code = </span><span class="nx">code</span> <span class="c1"># The remainder of the source code.</span>
|
|
<span class="vi">@line = </span><span class="nx">opts</span><span class="p">.</span><span class="nx">line</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The current line.</span>
|
|
<span class="vi">@indent = </span><span class="mi">0</span> <span class="c1"># The current indentation level.</span>
|
|
<span class="vi">@indebt = </span><span class="mi">0</span> <span class="c1"># The over-indentation at the current level.</span>
|
|
<span class="vi">@outdebt = </span><span class="mi">0</span> <span class="c1"># The under-outdentation at the current level.</span>
|
|
<span class="vi">@indents = </span><span class="p">[]</span> <span class="c1"># The stack of all current indentation levels.</span>
|
|
<span class="vi">@tokens = </span><span class="p">[]</span> <span class="c1"># Stream of parsed tokens in the form `['TYPE', value, line]`.</span></pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> <p>At every position, run through this list of attempted matches,
|
|
short-circuiting if any of them succeed. Their order determines precedence:
|
|
<code>@literalToken</code> is the fallback catch-all.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">i = </span><span class="mi">0</span>
|
|
<span class="k">while</span> <span class="vi">@chunk = </span><span class="nx">code</span><span class="p">.</span><span class="nx">slice</span> <span class="nx">i</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">@identifierToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@commentToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@whitespaceToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@lineToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@heredocToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@stringToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@numberToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@regexToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@jsToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@literalToken</span><span class="p">()</span>
|
|
|
|
<span class="nx">@closeIndentation</span><span class="p">()</span>
|
|
<span class="k">return</span> <span class="nx">@tokens</span> <span class="k">if</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">rewrite</span> <span class="o">is</span> <span class="kc">off</span>
|
|
<span class="p">(</span><span class="k">new</span> <span class="nx">Rewriter</span><span class="p">).</span><span class="nx">rewrite</span> <span class="nx">@tokens</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> <p>Matches identifying literals: variables, keywords, method names, etc.
|
|
Check to ensure that JavaScript reserved words aren't being used as
|
|
identifiers. Because CoffeeScript reserves a handful of keywords that are
|
|
allowed in JavaScript, we're careful not to tag them as keywords when
|
|
referenced as property names here, so you can still do <code>jQuery.is()</code> even
|
|
though <code>is</code> means <code>===</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">identifierToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">IDENTIFIER</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="p">[</span><span class="nx">input</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">colon</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
|
|
<span class="k">if</span> <span class="nx">id</span> <span class="o">is</span> <span class="s1">'own'</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'FOR'</span>
|
|
<span class="nx">@token</span> <span class="s1">'OWN'</span><span class="p">,</span> <span class="nx">id</span>
|
|
<span class="k">return</span> <span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">forcedIdentifier = </span><span class="nx">colon</span> <span class="o">or</span>
|
|
<span class="p">(</span><span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">)</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="p">[</span><span class="s1">'.'</span><span class="p">,</span> <span class="s1">'?.'</span><span class="p">,</span> <span class="s1">'::'</span><span class="p">]</span> <span class="o">or</span>
|
|
<span class="o">not</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'@'</span><span class="p">)</span>
|
|
<span class="nv">tag = </span><span class="s1">'IDENTIFIER'</span>
|
|
|
|
<span class="k">if</span> <span class="o">not</span> <span class="nx">forcedIdentifier</span> <span class="o">and</span> <span class="p">(</span><span class="nx">id</span> <span class="k">in</span> <span class="nx">JS_KEYWORDS</span> <span class="o">or</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">COFFEE_KEYWORDS</span><span class="p">)</span>
|
|
<span class="nv">tag = </span><span class="nx">id</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">()</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'WHEN'</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="k">in</span> <span class="nx">LINE_BREAK</span>
|
|
<span class="nv">tag = </span><span class="s1">'LEADING_WHEN'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'FOR'</span>
|
|
<span class="vi">@seenFor = </span><span class="kc">yes</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'UNLESS'</span>
|
|
<span class="nv">tag = </span><span class="s1">'IF'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">in</span> <span class="nx">UNARY</span>
|
|
<span class="nv">tag = </span><span class="s1">'UNARY'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">in</span> <span class="nx">RELATION</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">isnt</span> <span class="s1">'INSTANCEOF'</span> <span class="o">and</span> <span class="nx">@seenFor</span>
|
|
<span class="nv">tag = </span><span class="s1">'FOR'</span> <span class="o">+</span> <span class="nx">tag</span>
|
|
<span class="vi">@seenFor = </span><span class="kc">no</span>
|
|
<span class="k">else</span>
|
|
<span class="nv">tag = </span><span class="s1">'RELATION'</span>
|
|
<span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'!'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nv">id = </span><span class="s1">'!'</span> <span class="o">+</span> <span class="nx">id</span>
|
|
|
|
<span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">JS_FORBIDDEN</span>
|
|
<span class="k">if</span> <span class="nx">forcedIdentifier</span>
|
|
<span class="nv">tag = </span><span class="s1">'IDENTIFIER'</span>
|
|
<span class="nv">id = </span><span class="k">new</span> <span class="nb">String</span> <span class="nx">id</span>
|
|
<span class="nv">id.reserved = </span><span class="kc">yes</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">RESERVED</span>
|
|
<span class="nx">@identifierError</span> <span class="nx">id</span>
|
|
|
|
<span class="nx">unless</span> <span class="nx">forcedIdentifier</span>
|
|
<span class="nv">id = </span><span class="nx">COFFEE_ALIAS_MAP</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">COFFEE_ALIASES</span>
|
|
<span class="nv">tag = </span><span class="k">switch</span> <span class="nx">id</span>
|
|
<span class="k">when</span> <span class="s1">'!'</span> <span class="k">then</span> <span class="s1">'UNARY'</span>
|
|
<span class="k">when</span> <span class="s1">'=='</span><span class="p">,</span> <span class="s1">'!='</span> <span class="k">then</span> <span class="s1">'COMPARE'</span>
|
|
<span class="k">when</span> <span class="s1">'&&'</span><span class="p">,</span> <span class="s1">'||'</span> <span class="k">then</span> <span class="s1">'LOGIC'</span>
|
|
<span class="k">when</span> <span class="s1">'true'</span><span class="p">,</span> <span class="s1">'false'</span><span class="p">,</span> <span class="s1">'null'</span><span class="p">,</span> <span class="s1">'undefined'</span> <span class="k">then</span> <span class="s1">'BOOL'</span>
|
|
<span class="k">when</span> <span class="s1">'break'</span><span class="p">,</span> <span class="s1">'continue'</span><span class="p">,</span> <span class="s1">'debugger'</span> <span class="k">then</span> <span class="s1">'STATEMENT'</span>
|
|
<span class="k">else</span> <span class="nx">tag</span>
|
|
|
|
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">id</span>
|
|
<span class="nx">@token</span> <span class="s1">':'</span><span class="p">,</span> <span class="s1">':'</span> <span class="k">if</span> <span class="nx">colon</span>
|
|
<span class="nx">input</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> <p>Matches numbers, including decimals, hex, and exponential notation.
|
|
Be careful not to interfere with ranges-in-progress.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">numberToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">NUMBER</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">number = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="nx">@token</span> <span class="s1">'NUMBER'</span><span class="p">,</span> <span class="nx">number</span>
|
|
<span class="nx">number</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> <p>Matches strings, including multi-line strings. Ensures that quotation marks
|
|
are balanced within the string's contents, and within nested interpolations.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">stringToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">switch</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
|
|
<span class="k">when</span> <span class="s2">"'"</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">SIMPLESTR</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nx">@token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="p">(</span><span class="nv">string = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]).</span><span class="nx">replace</span> <span class="nx">MULTILINER</span><span class="p">,</span> <span class="s1">'\\\n'</span>
|
|
<span class="k">when</span> <span class="s1">'"'</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">string = </span><span class="nx">@balancedString</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="s1">'"'</span>
|
|
<span class="k">if</span> <span class="mi">0</span> <span class="o"><</span> <span class="nx">string</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s1">'#{'</span><span class="p">,</span> <span class="mi">1</span>
|
|
<span class="nx">@interpolateString</span> <span class="nx">string</span><span class="p">.</span><span class="nx">slice</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span>
|
|
<span class="k">else</span>
|
|
<span class="nx">@token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="nx">@escapeLines</span> <span class="nx">string</span>
|
|
<span class="k">else</span>
|
|
<span class="k">return</span> <span class="mi">0</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">string</span><span class="p">,</span> <span class="s1">'\n'</span>
|
|
<span class="nx">string</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> <p>Matches heredocs, adjusting indentation to the correct level, as heredocs
|
|
preserve whitespace, but ignore indentation to the left.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">heredocToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">HEREDOC</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">heredoc = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="nv">quote = </span><span class="nx">heredoc</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
|
|
<span class="nv">doc = </span><span class="nx">@sanitizeHeredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="nx">quote</span><span class="o">:</span> <span class="nx">quote</span><span class="p">,</span> <span class="nx">indent</span><span class="o">:</span> <span class="kc">null</span>
|
|
<span class="k">if</span> <span class="nx">quote</span> <span class="o">is</span> <span class="s1">'"'</span> <span class="o">and</span> <span class="mi">0</span> <span class="o"><=</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s1">'#{'</span>
|
|
<span class="nx">@interpolateString</span> <span class="nx">doc</span><span class="p">,</span> <span class="nx">heredoc</span><span class="o">:</span> <span class="kc">yes</span>
|
|
<span class="k">else</span>
|
|
<span class="nx">@token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span> <span class="nx">doc</span><span class="p">,</span> <span class="nx">quote</span><span class="p">,</span> <span class="kc">yes</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">heredoc</span><span class="p">,</span> <span class="s1">'\n'</span>
|
|
<span class="nx">heredoc</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> <p>Matches and consumes comments.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">commentToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">COMMENT</span>
|
|
<span class="p">[</span><span class="nx">comment</span><span class="p">,</span> <span class="nx">here</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="k">if</span> <span class="nx">here</span>
|
|
<span class="nx">@token</span> <span class="s1">'HERECOMMENT'</span><span class="p">,</span> <span class="nx">@sanitizeHeredoc</span> <span class="nx">here</span><span class="p">,</span>
|
|
<span class="nx">herecomment</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">indent</span><span class="o">:</span> <span class="nb">Array</span><span class="p">(</span><span class="nx">@indent</span> <span class="o">+</span> <span class="mi">1</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
|
|
<span class="nx">@token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s1">'\n'</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">comment</span><span class="p">,</span> <span class="s1">'\n'</span>
|
|
<span class="nx">comment</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> <p>Matches JavaScript interpolated directly into the source via backticks.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">jsToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'`'</span> <span class="o">and</span> <span class="nv">match = </span><span class="nx">JSTOKEN</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nx">@token</span> <span class="s1">'JS'</span><span class="p">,</span> <span class="p">(</span><span class="nv">script = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]).</span><span class="nx">slice</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span>
|
|
<span class="nx">script</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> <p>Matches regular expression literals. Lexing regular expressions is difficult
|
|
to distinguish from division, so we borrow some basic heuristics from
|
|
JavaScript and Ruby.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">regexToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">isnt</span> <span class="s1">'/'</span>
|
|
<span class="k">if</span> <span class="nv">match = </span><span class="nx">HEREGEX</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">length = </span><span class="nx">@heregexToken</span> <span class="nx">match</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s1">'\n'</span>
|
|
<span class="k">return</span> <span class="nx">length</span>
|
|
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="p">(</span><span class="k">if</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span> <span class="k">then</span> <span class="nx">NOT_REGEX</span> <span class="k">else</span> <span class="nx">NOT_SPACED_REGEX</span><span class="p">))</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">REGEX</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="p">[</span><span class="nx">regex</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="nx">@token</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="k">if</span> <span class="nx">regex</span> <span class="o">is</span> <span class="s1">'//'</span> <span class="k">then</span> <span class="s1">'/(?:)/'</span> <span class="k">else</span> <span class="nx">regex</span>
|
|
<span class="nx">regex</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-15">¶</a> </div> <p>Matches multiline extended regular expressions.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">heregexToken</span><span class="o">:</span> <span class="p">(</span><span class="nx">match</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="p">[</span><span class="nx">heregex</span><span class="p">,</span> <span class="nx">body</span><span class="p">,</span> <span class="nx">flags</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="k">if</span> <span class="mi">0</span> <span class="o">></span> <span class="nx">body</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s1">'#{'</span>
|
|
<span class="nv">re = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">HEREGEX_OMIT</span><span class="p">,</span> <span class="s1">''</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/\//g</span><span class="p">,</span> <span class="s1">'\\/'</span><span class="p">)</span>
|
|
<span class="nx">@token</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="s2">"/#{ re or '(?:)' }/#{flags}"</span>
|
|
<span class="k">return</span> <span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nx">@token</span> <span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'RegExp'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'CALL_START'</span><span class="p">,</span> <span class="s1">'('</span><span class="p">]</span>
|
|
<span class="nv">tokens = </span><span class="p">[]</span>
|
|
<span class="k">for</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">]</span> <span class="k">in</span> <span class="nx">@interpolateString</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nx">regex</span><span class="o">:</span> <span class="kc">yes</span><span class="p">)</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'TOKENS'</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">value</span><span class="p">...</span>
|
|
<span class="k">else</span>
|
|
<span class="k">continue</span> <span class="nx">unless</span> <span class="nv">value = </span><span class="nx">value</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">HEREGEX_OMIT</span><span class="p">,</span> <span class="s1">''</span>
|
|
<span class="nv">value = </span><span class="nx">value</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/\\/g</span><span class="p">,</span> <span class="s1">'\\\\'</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span><span class="p">(</span><span class="nx">value</span><span class="p">,</span> <span class="s1">'"'</span><span class="p">,</span> <span class="kc">yes</span><span class="p">)]</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'+'</span><span class="p">,</span> <span class="s1">'+'</span><span class="p">]</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'""'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'+'</span><span class="p">,</span> <span class="s1">'+'</span><span class="p">]</span> <span class="nx">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">?</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'STRING'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">tokens</span><span class="p">...</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">','</span><span class="p">,</span> <span class="s1">','</span><span class="p">],</span> <span class="p">[</span><span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'"'</span> <span class="o">+</span> <span class="nx">flags</span> <span class="o">+</span> <span class="s1">'"'</span><span class="p">]</span> <span class="k">if</span> <span class="nx">flags</span>
|
|
<span class="nx">@token</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span>
|
|
<span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-16">¶</a> </div> <p>Matches newlines, indents, and outdents, and determines which is which.
|
|
If we can detect that the current line is continued onto the the next line,
|
|
then the newline is suppressed:</p>
|
|
|
|
<pre><code>elements
|
|
.each( ... )
|
|
.map( ... )
|
|
</code></pre>
|
|
|
|
<p>Keeps track of the level of indentation, because a single outdent token
|
|
can close multiple indents, so we need to know how far in we happen to be.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">lineToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="nv">match = </span><span class="nx">MULTI_DENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">indent = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">indent</span><span class="p">,</span> <span class="s1">'\n'</span>
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="mi">1</span>
|
|
<span class="nv">size = </span><span class="nx">indent</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span> <span class="o">-</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">lastIndexOf</span> <span class="s1">'\n'</span>
|
|
<span class="nv">noNewlines = </span><span class="nx">@unfinished</span><span class="p">()</span>
|
|
<span class="k">if</span> <span class="nx">size</span> <span class="o">-</span> <span class="nx">@indebt</span> <span class="o">is</span> <span class="nx">@indent</span>
|
|
<span class="k">if</span> <span class="nx">noNewlines</span> <span class="k">then</span> <span class="nx">@suppressNewlines</span><span class="p">()</span> <span class="k">else</span> <span class="nx">@newlineToken</span><span class="p">()</span>
|
|
<span class="k">return</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nx">size</span> <span class="o">></span> <span class="nx">@indent</span>
|
|
<span class="k">if</span> <span class="nx">noNewlines</span>
|
|
<span class="vi">@indebt = </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span>
|
|
<span class="nx">@suppressNewlines</span><span class="p">()</span>
|
|
<span class="k">return</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">diff = </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span> <span class="o">+</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">@token</span> <span class="s1">'INDENT'</span><span class="p">,</span> <span class="nx">diff</span>
|
|
<span class="nx">@indents</span><span class="p">.</span><span class="nx">push</span> <span class="nx">diff</span>
|
|
<span class="vi">@outdebt = @indebt = </span><span class="mi">0</span>
|
|
<span class="k">else</span>
|
|
<span class="vi">@indebt = </span><span class="mi">0</span>
|
|
<span class="nx">@outdentToken</span> <span class="nx">@indent</span> <span class="o">-</span> <span class="nx">size</span><span class="p">,</span> <span class="nx">noNewlines</span>
|
|
<span class="vi">@indent = </span><span class="nx">size</span>
|
|
<span class="nx">indent</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-17">¶</a> </div> <p>Record an outdent token or multiple tokens, if we happen to be moving back
|
|
inwards past several recorded indents.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">outdentToken</span><span class="o">:</span> <span class="p">(</span><span class="nx">moveOut</span><span class="p">,</span> <span class="nx">noNewlines</span><span class="p">,</span> <span class="nx">close</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="k">while</span> <span class="nx">moveOut</span> <span class="o">></span> <span class="mi">0</span>
|
|
<span class="nv">len = </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
|
<span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">is</span> <span class="kc">undefined</span>
|
|
<span class="nv">moveOut = </span><span class="mi">0</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">is</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">@outdebt</span>
|
|
<span class="vi">@outdebt = </span><span class="mi">0</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o"><</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">@outdebt</span> <span class="o">-=</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span>
|
|
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span>
|
|
<span class="k">else</span>
|
|
<span class="nv">dent = </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="o">-</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">dent</span>
|
|
<span class="vi">@outdebt = </span><span class="mi">0</span>
|
|
<span class="nx">@token</span> <span class="s1">'OUTDENT'</span><span class="p">,</span> <span class="nx">dent</span>
|
|
<span class="nx">@outdebt</span> <span class="o">-=</span> <span class="nx">moveOut</span> <span class="k">if</span> <span class="nx">dent</span>
|
|
<span class="nx">@token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s1">'\n'</span> <span class="nx">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span> <span class="o">or</span> <span class="nx">noNewlines</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-18">¶</a> </div> <p>Matches and consumes non-meaningful whitespace. Tag the previous token
|
|
as being "spaced", because there are some cases where it makes a difference.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">whitespaceToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="nx">unless</span> <span class="p">(</span><span class="nv">match = </span><span class="nx">WHITESPACE</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span><span class="p">)</span> <span class="o">or</span>
|
|
<span class="p">(</span><span class="nv">nline = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'\n'</span><span class="p">)</span>
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="k">if</span> <span class="nx">match</span> <span class="k">then</span> <span class="s1">'spaced'</span> <span class="k">else</span> <span class="s1">'newLine'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">true</span> <span class="k">if</span> <span class="nx">prev</span>
|
|
<span class="k">if</span> <span class="nx">match</span> <span class="k">then</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span> <span class="k">else</span> <span class="mi">0</span></pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-19">¶</a> </div> <p>Generate a newline token. Consecutive newlines get merged together.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">newlineToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="nx">@token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s1">'\n'</span> <span class="nx">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-20">¶</a> </div> <p>Use a <code>\</code> at a line-ending to suppress the newline.
|
|
The slash is removed here once its job is done.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">suppressNewlines</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'\\'</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-21">¶</a> </div> <p>We treat all other single characters as a token. E.g.: <code>( ) , . !</code>
|
|
Multi-character operators are also literal tokens, so that Jison can assign
|
|
the proper order of operations. There are some symbols that we tag specially
|
|
here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish
|
|
parentheses that indicate a method call from regular parentheses, and so on.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">literalToken</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">if</span> <span class="nv">match = </span><span class="nx">OPERATOR</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="p">[</span><span class="nx">value</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="nx">@tagParameters</span><span class="p">()</span> <span class="k">if</span> <span class="nx">CODE</span><span class="p">.</span><span class="nx">test</span> <span class="nx">value</span>
|
|
<span class="k">else</span>
|
|
<span class="nv">value = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
|
|
<span class="nv">tag = </span><span class="nx">value</span>
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
|
|
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'='</span> <span class="o">and</span> <span class="nx">prev</span>
|
|
<span class="nx">@assignmentError</span><span class="p">()</span> <span class="k">if</span> <span class="o">not</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">reserved</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">in</span> <span class="nx">JS_FORBIDDEN</span>
|
|
<span class="k">if</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">in</span> <span class="p">[</span><span class="s1">'||'</span><span class="p">,</span> <span class="s1">'&&'</span><span class="p">]</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'COMPOUND_ASSIGN'</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="s1">'='</span>
|
|
<span class="k">return</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">';'</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'TERMINATOR'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">MATH</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'MATH'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">COMPARE</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'COMPARE'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">COMPOUND_ASSIGN</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'COMPOUND_ASSIGN'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">UNARY</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'UNARY'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">SHIFT</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'SHIFT'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">LOGIC</span> <span class="o">or</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'?'</span> <span class="o">and</span> <span class="nx">prev</span><span class="o">?</span><span class="p">.</span><span class="nx">spaced</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s1">'LOGIC'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span>
|
|
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'('</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="nx">CALLABLE</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'FUNC_EXIST'</span> <span class="k">if</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'?'</span>
|
|
<span class="nv">tag = </span><span class="s1">'CALL_START'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'['</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="nx">INDEXABLE</span>
|
|
<span class="nv">tag = </span><span class="s1">'INDEX_START'</span>
|
|
<span class="k">switch</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="k">when</span> <span class="s1">'?'</span> <span class="k">then</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'INDEX_SOAK'</span>
|
|
<span class="k">when</span> <span class="s1">'::'</span> <span class="k">then</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'INDEX_PROTO'</span>
|
|
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
|
|
<span class="nx">value</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-22">¶</a> </div> <h2>Token Manipulators</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-23">¶</a> </div> <p>Sanitize a heredoc or herecomment by
|
|
erasing all external indentation on the left-hand side.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">sanitizeHeredoc</span><span class="o">:</span> <span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">options</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="p">{</span><span class="nx">indent</span><span class="p">,</span> <span class="nx">herecomment</span><span class="p">}</span> <span class="o">=</span> <span class="nx">options</span>
|
|
<span class="k">if</span> <span class="nx">herecomment</span>
|
|
<span class="k">if</span> <span class="nx">HEREDOC_ILLEGAL</span><span class="p">.</span><span class="nx">test</span> <span class="nx">doc</span>
|
|
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s2">"block comment cannot contain \"*/\", starting on line #{@line + 1}"</span>
|
|
<span class="k">return</span> <span class="nx">doc</span> <span class="k">if</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="s1">'\n'</span><span class="p">)</span> <span class="o"><=</span> <span class="mi">0</span>
|
|
<span class="k">else</span>
|
|
<span class="k">while</span> <span class="nv">match = </span><span class="nx">HEREDOC_INDENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">doc</span>
|
|
<span class="nv">attempt = </span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
|
|
<span class="nv">indent = </span><span class="nx">attempt</span> <span class="k">if</span> <span class="nx">indent</span> <span class="o">is</span> <span class="kc">null</span> <span class="o">or</span> <span class="mi">0</span> <span class="o"><</span> <span class="nx">attempt</span><span class="p">.</span><span class="nx">length</span> <span class="o"><</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">doc = </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span> <span class="err">/// \n #{indent} ///g, '\n' if indent</span>
|
|
<span class="nv">doc = </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/^\n/</span><span class="p">,</span> <span class="s1">''</span> <span class="nx">unless</span> <span class="nx">herecomment</span>
|
|
<span class="nx">doc</span></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-24">¶</a> </div> <p>A source of ambiguity in our grammar used to be parameter lists in function
|
|
definitions versus argument lists in function calls. Walk backwards, tagging
|
|
parameters specially in order to make things easier for the parser.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tagParameters</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="k">this</span> <span class="k">if</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">isnt</span> <span class="s1">')'</span>
|
|
<span class="nv">stack = </span><span class="p">[]</span>
|
|
<span class="p">{</span><span class="nx">tokens</span><span class="p">}</span> <span class="o">=</span> <span class="k">this</span>
|
|
<span class="nv">i = </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nx">tokens</span><span class="p">[</span><span class="o">--</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'PARAM_END'</span>
|
|
<span class="k">while</span> <span class="nv">tok = </span><span class="nx">tokens</span><span class="p">[</span><span class="o">--</span><span class="nx">i</span><span class="p">]</span>
|
|
<span class="k">switch</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="k">when</span> <span class="s1">')'</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nx">tok</span>
|
|
<span class="k">when</span> <span class="s1">'('</span><span class="p">,</span> <span class="s1">'CALL_START'</span>
|
|
<span class="k">if</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="k">then</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'('</span>
|
|
<span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'PARAM_START'</span>
|
|
<span class="k">return</span> <span class="k">this</span>
|
|
<span class="k">else</span> <span class="k">return</span> <span class="k">this</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-25">¶</a> </div> <p>Close up all remaining open blocks at the end of the file.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">closeIndentation</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="nx">@outdentToken</span> <span class="nx">@indent</span></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-26">¶</a> </div> <p>The error for when you try to use a forbidden word in JavaScript as
|
|
an identifier.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">identifierError</span><span class="o">:</span> <span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="k">throw</span> <span class="nx">SyntaxError</span> <span class="s2">"Reserved word \"#{word}\" on line #{@line + 1}"</span></pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-27">¶</a> </div> <p>The error for when you try to assign to a reserved word in JavaScript,
|
|
like "function" or "default".</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">assignmentError</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="k">throw</span> <span class="nx">SyntaxError</span> <span class="s2">"Reserved word \"#{@value()}\" on line #{@line + 1} can't be assigned"</span></pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-28">¶</a> </div> <p>Matches a balanced group such as a single or double-quoted string. Pass in
|
|
a series of delimiters, all of which must be nested correctly within the
|
|
contents of the string. This method allows us to have strings within
|
|
interpolations within strings, ad infinitum.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">balancedString</span><span class="o">:</span> <span class="p">(</span><span class="nx">str</span><span class="p">,</span> <span class="nx">end</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="nv">stack = </span><span class="p">[</span><span class="nx">end</span><span class="p">]</span>
|
|
<span class="k">for</span> <span class="nx">i</span> <span class="k">in</span> <span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="nx">str</span><span class="p">.</span><span class="nx">length</span><span class="p">]</span>
|
|
<span class="k">switch</span> <span class="nv">letter = </span><span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span> <span class="nx">i</span>
|
|
<span class="k">when</span> <span class="s1">'\\'</span>
|
|
<span class="nx">i</span><span class="o">++</span>
|
|
<span class="k">continue</span>
|
|
<span class="k">when</span> <span class="nx">end</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nx">unless</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">return</span> <span class="nx">str</span><span class="p">.</span><span class="nx">slice</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
|
|
<span class="nv">end = </span><span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span>
|
|
<span class="k">continue</span>
|
|
<span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s1">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="k">in</span> <span class="p">[</span><span class="s1">'"'</span><span class="p">,</span> <span class="s2">"'"</span><span class="p">]</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="nx">letter</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s1">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s1">'/'</span> <span class="o">and</span> <span class="nv">match = </span><span class="p">(</span><span class="nx">HEREGEX</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">str</span><span class="p">.</span><span class="nx">slice</span> <span class="nx">i</span><span class="p">)</span> <span class="o">or</span> <span class="nx">REGEX</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">str</span><span class="p">.</span><span class="nx">slice</span> <span class="nx">i</span><span class="p">))</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s1">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s1">'{'</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="s1">'}'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s1">'"'</span> <span class="o">and</span> <span class="nx">prev</span> <span class="o">is</span> <span class="s1">'#'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s1">'{'</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="s1">'}'</span>
|
|
<span class="nv">prev = </span><span class="nx">letter</span>
|
|
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s2">"missing #{ stack.pop() }, starting on line #{ @line + 1 }"</span></pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-29">¶</a> </div> <p>Expand variables and expressions inside double-quoted strings using
|
|
Ruby-like notation for substitution of arbitrary expressions.</p>
|
|
|
|
<pre><code>"Hello #{name.capitalize()}."
|
|
</code></pre>
|
|
|
|
<p>If it encounters an interpolation, this method will recursively create a
|
|
new Lexer, tokenize the interpolated contents, and merge them into the
|
|
token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">interpolateString</span><span class="o">:</span> <span class="p">(</span><span class="nx">str</span><span class="p">,</span> <span class="nv">options = </span><span class="p">{})</span> <span class="o">-></span>
|
|
<span class="p">{</span><span class="nx">heredoc</span><span class="p">,</span> <span class="nx">regex</span><span class="p">}</span> <span class="o">=</span> <span class="nx">options</span>
|
|
<span class="nv">tokens = </span><span class="p">[]</span>
|
|
<span class="nv">pi = </span><span class="mi">0</span>
|
|
<span class="nv">i = </span><span class="o">-</span><span class="mi">1</span>
|
|
<span class="k">while</span> <span class="nv">letter = </span><span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span> <span class="nx">i</span> <span class="o">+=</span> <span class="mi">1</span>
|
|
<span class="k">if</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s1">'\\'</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="mi">1</span>
|
|
<span class="k">continue</span>
|
|
<span class="nx">unless</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s1">'#'</span> <span class="o">and</span> <span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="nx">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'{'</span> <span class="o">and</span>
|
|
<span class="p">(</span><span class="nv">expr = </span><span class="nx">@balancedString</span> <span class="nx">str</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">),</span> <span class="s1">'}'</span><span class="p">)</span>
|
|
<span class="k">continue</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'NEOSTRING'</span><span class="p">,</span> <span class="nx">str</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="nx">pi</span><span class="p">,</span> <span class="nx">i</span><span class="p">)]</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">i</span>
|
|
<span class="nv">inner = </span><span class="nx">expr</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
|
|
<span class="k">if</span> <span class="nx">inner</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">nested = </span><span class="k">new</span> <span class="nx">Lexer</span><span class="p">().</span><span class="nx">tokenize</span> <span class="nx">inner</span><span class="p">,</span> <span class="nx">line</span><span class="o">:</span> <span class="nx">@line</span><span class="p">,</span> <span class="nx">rewrite</span><span class="o">:</span> <span class="kc">off</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">shift</span><span class="p">()</span> <span class="k">if</span> <span class="nx">nested</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">?</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span>
|
|
<span class="k">if</span> <span class="nv">len = </span><span class="nx">nested</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nx">len</span> <span class="o">></span> <span class="mi">1</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">unshift</span> <span class="p">[</span><span class="s1">'('</span><span class="p">,</span> <span class="s1">'('</span><span class="p">]</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">]</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'TOKENS'</span><span class="p">,</span> <span class="nx">nested</span><span class="p">]</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">pi = </span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'NEOSTRING'</span><span class="p">,</span> <span class="nx">str</span><span class="p">.</span><span class="nx">slice</span> <span class="nx">pi</span><span class="p">]</span> <span class="k">if</span> <span class="nx">i</span> <span class="o">></span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">return</span> <span class="nx">tokens</span> <span class="k">if</span> <span class="nx">regex</span>
|
|
<span class="k">return</span> <span class="nx">@token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'""'</span> <span class="nx">unless</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">unshift</span> <span class="p">[</span><span class="s1">''</span><span class="p">,</span> <span class="s1">''</span><span class="p">]</span> <span class="nx">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'NEOSTRING'</span>
|
|
<span class="nx">@token</span> <span class="s1">'('</span><span class="p">,</span> <span class="s1">'('</span> <span class="k">if</span> <span class="nv">interpolated = </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">></span> <span class="mi">1</span>
|
|
<span class="k">for</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">],</span> <span class="nx">i</span> <span class="k">in</span> <span class="nx">tokens</span>
|
|
<span class="nx">@token</span> <span class="s1">'+'</span><span class="p">,</span> <span class="s1">'+'</span> <span class="k">if</span> <span class="nx">i</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'TOKENS'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">value</span><span class="p">...</span>
|
|
<span class="k">else</span>
|
|
<span class="nx">@token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span> <span class="nx">value</span><span class="p">,</span> <span class="s1">'"'</span><span class="p">,</span> <span class="nx">heredoc</span>
|
|
<span class="nx">@token</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span> <span class="k">if</span> <span class="nx">interpolated</span>
|
|
<span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-30">¶</a> </div> <h2>Helpers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-31">¶</a> </div> <p>Add a token to the results, taking note of the line number.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">token</span><span class="o">:</span> <span class="p">(</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">@line</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-32"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-32">¶</a> </div> <p>Peek at a tag in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">tag</span><span class="o">:</span> <span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">tag</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="p">(</span><span class="nv">tok = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">and</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="nx">tag</span> <span class="k">else</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-33">¶</a> </div> <p>Peek at a value in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">value</span><span class="o">:</span> <span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="p">(</span><span class="nv">tok = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">and</span> <span class="k">if</span> <span class="nx">val</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="nx">val</span> <span class="k">else</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-34">¶</a> </div> <p>Are we in the midst of an unfinished expression?</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">unfinished</span><span class="o">:</span> <span class="o">-></span>
|
|
<span class="nx">LINE_CONTINUER</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">)</span> <span class="o">or</span>
|
|
<span class="p">(</span><span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">'.'</span> <span class="o">and</span>
|
|
<span class="p">(</span><span class="nv">value = </span><span class="nx">@value</span><span class="p">())</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">value</span><span class="p">.</span><span class="nx">reserved</span> <span class="o">and</span>
|
|
<span class="nx">NO_NEWLINE</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">value</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">CODE</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">value</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">ASSIGNED</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-35">¶</a> </div> <p>Converts newlines for string literals.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">escapeLines</span><span class="o">:</span> <span class="p">(</span><span class="nx">str</span><span class="p">,</span> <span class="nx">heredoc</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="nx">str</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">MULTILINER</span><span class="p">,</span> <span class="k">if</span> <span class="nx">heredoc</span> <span class="k">then</span> <span class="s1">'\\n'</span> <span class="k">else</span> <span class="s1">''</span></pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-36">¶</a> </div> <p>Constructs a string token by escaping quotes and newlines.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">makeString</span><span class="o">:</span> <span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nx">quote</span><span class="p">,</span> <span class="nx">heredoc</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="k">return</span> <span class="nx">quote</span> <span class="o">+</span> <span class="nx">quote</span> <span class="nx">unless</span> <span class="nx">body</span>
|
|
<span class="nv">body = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/\\([\s\S])/g</span><span class="p">,</span> <span class="p">(</span><span class="nx">match</span><span class="p">,</span> <span class="nx">contents</span><span class="p">)</span> <span class="o">-></span>
|
|
<span class="k">if</span> <span class="nx">contents</span> <span class="k">in</span> <span class="p">[</span><span class="s1">'\n'</span><span class="p">,</span> <span class="nx">quote</span><span class="p">]</span> <span class="k">then</span> <span class="nx">contents</span> <span class="k">else</span> <span class="nx">match</span>
|
|
<span class="nv">body = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span> <span class="err">/// #{quote} ///g, '\\$&'</span>
|
|
<span class="nx">quote</span> <span class="o">+</span> <span class="nx">@escapeLines</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nx">heredoc</span><span class="p">)</span> <span class="o">+</span> <span class="nx">quote</span></pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-37">¶</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-38">¶</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_KEYWORDS = </span><span class="p">[</span>
|
|
<span class="s1">'true'</span><span class="p">,</span> <span class="s1">'false'</span><span class="p">,</span> <span class="s1">'null'</span><span class="p">,</span> <span class="s1">'this'</span>
|
|
<span class="s1">'new'</span><span class="p">,</span> <span class="s1">'delete'</span><span class="p">,</span> <span class="s1">'typeof'</span><span class="p">,</span> <span class="s1">'in'</span><span class="p">,</span> <span class="s1">'instanceof'</span>
|
|
<span class="s1">'return'</span><span class="p">,</span> <span class="s1">'throw'</span><span class="p">,</span> <span class="s1">'break'</span><span class="p">,</span> <span class="s1">'continue'</span><span class="p">,</span> <span class="s1">'debugger'</span>
|
|
<span class="s1">'if'</span><span class="p">,</span> <span class="s1">'else'</span><span class="p">,</span> <span class="s1">'switch'</span><span class="p">,</span> <span class="s1">'for'</span><span class="p">,</span> <span class="s1">'while'</span><span class="p">,</span> <span class="s1">'do'</span><span class="p">,</span> <span class="s1">'try'</span><span class="p">,</span> <span class="s1">'catch'</span><span class="p">,</span> <span class="s1">'finally'</span>
|
|
<span class="s1">'class'</span><span class="p">,</span> <span class="s1">'extends'</span><span class="p">,</span> <span class="s1">'super'</span>
|
|
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-39">¶</a> </div> <p>CoffeeScript-only keywords.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COFFEE_KEYWORDS = </span><span class="p">[</span><span class="s1">'undefined'</span><span class="p">,</span> <span class="s1">'then'</span><span class="p">,</span> <span class="s1">'unless'</span><span class="p">,</span> <span class="s1">'until'</span><span class="p">,</span> <span class="s1">'loop'</span><span class="p">,</span> <span class="s1">'of'</span><span class="p">,</span> <span class="s1">'by'</span><span class="p">,</span> <span class="s1">'when'</span><span class="p">]</span>
|
|
|
|
<span class="nv">COFFEE_ALIAS_MAP =</span>
|
|
<span class="o">and</span> <span class="o">:</span> <span class="s1">'&&'</span>
|
|
<span class="o">or</span> <span class="o">:</span> <span class="s1">'||'</span>
|
|
<span class="o">is</span> <span class="o">:</span> <span class="s1">'=='</span>
|
|
<span class="o">isnt</span> <span class="o">:</span> <span class="s1">'!='</span>
|
|
<span class="o">not</span> <span class="o">:</span> <span class="s1">'!'</span>
|
|
<span class="kc">yes</span> <span class="o">:</span> <span class="s1">'true'</span>
|
|
<span class="kc">no</span> <span class="o">:</span> <span class="s1">'false'</span>
|
|
<span class="kc">on</span> <span class="o">:</span> <span class="s1">'true'</span>
|
|
<span class="kc">off</span> <span class="o">:</span> <span class="s1">'false'</span>
|
|
|
|
<span class="nv">COFFEE_ALIASES = </span><span class="p">(</span><span class="nx">key</span> <span class="k">for</span> <span class="nx">key</span> <span class="k">of</span> <span class="nx">COFFEE_ALIAS_MAP</span><span class="p">)</span>
|
|
<span class="nv">COFFEE_KEYWORDS = </span><span class="nx">COFFEE_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">COFFEE_ALIASES</span></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-40">¶</a> </div> <p>The list of keywords that are reserved by JavaScript, but not used, or are
|
|
used by CoffeeScript internally. We throw an error when these are encountered,
|
|
to avoid having a JavaScript error at runtime.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RESERVED = </span><span class="p">[</span>
|
|
<span class="s1">'case'</span><span class="p">,</span> <span class="s1">'default'</span><span class="p">,</span> <span class="s1">'function'</span><span class="p">,</span> <span class="s1">'var'</span><span class="p">,</span> <span class="s1">'void'</span><span class="p">,</span> <span class="s1">'with'</span>
|
|
<span class="s1">'const'</span><span class="p">,</span> <span class="s1">'let'</span><span class="p">,</span> <span class="s1">'enum'</span><span class="p">,</span> <span class="s1">'export'</span><span class="p">,</span> <span class="s1">'import'</span><span class="p">,</span> <span class="s1">'native'</span>
|
|
<span class="s1">'__hasProp'</span><span class="p">,</span> <span class="s1">'__extends'</span><span class="p">,</span> <span class="s1">'__slice'</span><span class="p">,</span> <span class="s1">'__bind'</span><span class="p">,</span> <span class="s1">'__indexOf'</span>
|
|
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-41">¶</a> </div> <p>The superset of both JavaScript keywords and reserved words, none of which may
|
|
be used as identifiers or properties.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_FORBIDDEN = </span><span class="nx">JS_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">RESERVED</span>
|
|
|
|
<span class="nv">exports.RESERVED = </span><span class="nx">RESERVED</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">JS_KEYWORDS</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">COFFEE_KEYWORDS</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-42">¶</a> </div> <p>Token matching regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IDENTIFIER = </span><span class="err">/// ^</span>
|
|
<span class="p">(</span> <span class="p">[</span><span class="nx">$A</span><span class="o">-</span><span class="nx">Za</span><span class="o">-</span><span class="nx">z_</span><span class="err">\</span><span class="nx">x7f</span><span class="o">-</span><span class="err">\</span><span class="nx">uffff</span><span class="p">][</span><span class="nx">$</span><span class="err">\</span><span class="nx">w</span><span class="err">\</span><span class="nx">x7f</span><span class="o">-</span><span class="err">\</span><span class="nx">uffff</span><span class="p">]</span><span class="o">*</span> <span class="p">)</span>
|
|
<span class="p">(</span> <span class="p">[</span><span class="o">^</span><span class="err">\</span><span class="nx">n</span><span class="err">\</span><span class="nx">S</span><span class="p">]</span><span class="o">*</span> <span class="o">:</span> <span class="p">(</span><span class="o">?!:</span><span class="p">)</span> <span class="p">)</span><span class="o">?</span> <span class="c1"># Is this a property name?</span>
|
|
<span class="err">///</span>
|
|
|
|
<span class="nv">NUMBER = </span><span class="err">///</span>
|
|
<span class="o">^</span> <span class="mi">0</span><span class="nx">x</span><span class="p">[</span><span class="err">\</span><span class="nx">da</span><span class="o">-</span><span class="nx">f</span><span class="p">]</span><span class="o">+</span> <span class="o">|</span> <span class="c1"># hex</span>
|
|
<span class="o">^</span> <span class="err">\</span><span class="nx">d</span><span class="o">*</span><span class="err">\</span><span class="p">.</span><span class="o">?</span><span class="err">\</span><span class="nx">d</span><span class="o">+</span> <span class="p">(</span><span class="o">?:</span><span class="nx">e</span><span class="p">[</span><span class="o">+-</span><span class="p">]</span><span class="o">?</span><span class="err">\</span><span class="nx">d</span><span class="o">+</span><span class="p">)</span><span class="o">?</span> <span class="c1"># decimal</span>
|
|
<span class="err">///i</span>
|
|
|
|
<span class="nv">HEREDOC = </span><span class="err">/// ^ ("""|''') ([\s\S]*?) (?:\n[^\n\S]*)? \1 ///</span>
|
|
|
|
<span class="nv">OPERATOR = </span><span class="err">/// ^ (</span>
|
|
<span class="o">?:</span> <span class="p">[</span><span class="o">-=</span><span class="p">]</span><span class="o">></span> <span class="c1"># function</span>
|
|
<span class="o">|</span> <span class="p">[</span><span class="o">-+*</span><span class="sr">/%<>&|^!?=]= # compound assign /</span> <span class="nx">compare</span>
|
|
<span class="o">|</span> <span class="o">>>>=?</span> <span class="c1"># zero-fill right shift</span>
|
|
<span class="o">|</span> <span class="p">([</span><span class="o">-+:</span><span class="p">])</span><span class="err">\</span><span class="mi">1</span> <span class="c1"># doubles</span>
|
|
<span class="o">|</span> <span class="p">([</span><span class="o">&|<></span><span class="p">])</span><span class="err">\</span><span class="mi">2</span><span class="o">=?</span> <span class="c1"># logic / shift</span>
|
|
<span class="o">|</span> <span class="err">\</span><span class="o">?</span><span class="err">\</span><span class="p">.</span> <span class="c1"># soak access</span>
|
|
<span class="o">|</span> <span class="err">\</span><span class="p">.{</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">}</span> <span class="c1"># range or splat</span>
|
|
<span class="p">)</span> <span class="o">/</span><span class="err">//</span>
|
|
|
|
<span class="nv">WHITESPACE = </span><span class="sr">/^[^\n\S]+/</span>
|
|
|
|
<span class="nv">COMMENT = </span><span class="sr">/^###([^#][\s\S]*?)(?:###[^\n\S]*|(?:###)?$)|^(?:\s*#(?!##[^#]).*)+/</span>
|
|
|
|
<span class="nv">CODE = </span><span class="sr">/^[-=]>/</span>
|
|
|
|
<span class="nv">MULTI_DENT = </span><span class="sr">/^(?:\n[^\n\S]*)+/</span>
|
|
|
|
<span class="nv">SIMPLESTR = </span><span class="sr">/^'[^\\']*(?:\\.[^\\']*)*'/</span>
|
|
|
|
<span class="nv">JSTOKEN = </span><span class="sr">/^`[^\\`]*(?:\\.[^\\`]*)*`/</span></pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-43">¶</a> </div> <p>Regex-matching-regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">REGEX = </span><span class="err">/// ^</span>
|
|
<span class="o">/</span> <span class="p">(</span><span class="o">?!</span> <span class="p">[</span><span class="err">\</span><span class="nx">s</span><span class="o">=</span><span class="p">]</span> <span class="p">)</span> <span class="c1"># disallow leading whitespace or equals signs</span>
|
|
<span class="p">[</span><span class="o">^</span> <span class="p">[</span> <span class="err">/ \n \\ ]* # every other thing</span>
|
|
<span class="p">(</span><span class="o">?:</span>
|
|
<span class="p">(</span><span class="o">?:</span> <span class="err">\\</span><span class="p">[</span><span class="err">\</span><span class="nx">s</span><span class="err">\</span><span class="nx">S</span><span class="p">]</span> <span class="c1"># anything escaped</span>
|
|
<span class="o">|</span> <span class="err">\</span><span class="p">[</span> <span class="c1"># character class</span>
|
|
<span class="p">[</span><span class="o">^</span> <span class="err">\</span><span class="p">]</span> <span class="err">\</span><span class="nx">n</span> <span class="err">\\</span> <span class="p">]</span><span class="o">*</span>
|
|
<span class="p">(</span><span class="o">?:</span> <span class="err">\\</span><span class="p">[</span><span class="err">\</span><span class="nx">s</span><span class="err">\</span><span class="nx">S</span><span class="p">]</span> <span class="p">[</span><span class="o">^</span> <span class="err">\</span><span class="p">]</span> <span class="err">\</span><span class="nx">n</span> <span class="err">\\</span> <span class="p">]</span><span class="o">*</span> <span class="p">)</span><span class="o">*</span>
|
|
<span class="p">]</span>
|
|
<span class="p">)</span> <span class="p">[</span><span class="o">^</span> <span class="p">[</span> <span class="err">/ \n \\ ]*</span>
|
|
<span class="p">)</span><span class="o">*</span>
|
|
<span class="err">/ [imgy]{0,4} (?!\w)</span>
|
|
<span class="o">/</span><span class="err">//</span>
|
|
|
|
<span class="nv">HEREGEX = </span><span class="err">/// ^ /{3} ([\s\S]+?) /{3} ([imgy]{0,4}) (?!\w) ///</span>
|
|
|
|
<span class="nv">HEREGEX_OMIT = </span><span class="sr">/\s+(?:#.*)?/g</span></pre></div> </td> </tr> <tr id="section-44"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-44">¶</a> </div> <p>Token cleaning regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">MULTILINER = </span><span class="sr">/\n/g</span>
|
|
|
|
<span class="nv">HEREDOC_INDENT = </span><span class="sr">/\n+([^\n\S]*)/g</span>
|
|
|
|
<span class="nv">HEREDOC_ILLEGAL = </span><span class="sr">/\*\//</span>
|
|
|
|
<span class="nv">ASSIGNED = </span><span class="sr">/^\s*@?([$A-Za-z_][$\w\x7f-\uffff]*|['"].*['"])[^\n\S]*?[:=][^:=>]/</span>
|
|
|
|
<span class="nv">LINE_CONTINUER = </span><span class="err">/// ^ \s* (?: , | \??\.(?![.\d]) | :: ) ///</span>
|
|
|
|
<span class="nv">TRAILING_SPACES = </span><span class="sr">/\s+$/</span>
|
|
|
|
<span class="nv">NO_NEWLINE = </span><span class="err">/// ^ (?: # non-capturing group</span>
|
|
<span class="p">[</span><span class="o">-+*&|</span><span class="err">/%=<>!.\\][<>=&|]* | # symbol operators</span>
|
|
<span class="o">and</span> <span class="o">|</span> <span class="o">or</span> <span class="o">|</span> <span class="o">is</span><span class="p">(</span><span class="o">?:</span><span class="nx">nt</span><span class="p">)</span><span class="o">?</span> <span class="o">|</span> <span class="nx">n</span><span class="p">(</span><span class="o">?:</span><span class="nx">ot</span><span class="o">|</span><span class="nx">ew</span><span class="p">)</span> <span class="o">|</span> <span class="c1"># word operators</span>
|
|
<span class="k">delete</span> <span class="o">|</span> <span class="k">typeof</span> <span class="o">|</span> <span class="k">instanceof</span>
|
|
<span class="p">)</span> <span class="nx">$</span> <span class="err">///</span></pre></div> </td> </tr> <tr id="section-45"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-45">¶</a> </div> <p>Compound assignment tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COMPOUND_ASSIGN = </span><span class="p">[</span>
|
|
<span class="s1">'-='</span><span class="p">,</span> <span class="s1">'+='</span><span class="p">,</span> <span class="s1">'/='</span><span class="p">,</span> <span class="s1">'*='</span><span class="p">,</span> <span class="s1">'%='</span><span class="p">,</span> <span class="s1">'||='</span><span class="p">,</span> <span class="s1">'&&='</span><span class="p">,</span> <span class="s1">'?='</span><span class="p">,</span> <span class="s1">'<<='</span><span class="p">,</span> <span class="s1">'>>='</span><span class="p">,</span> <span class="s1">'>>>='</span><span class="p">,</span> <span class="s1">'&='</span><span class="p">,</span> <span class="s1">'^='</span><span class="p">,</span> <span class="s1">'|='</span>
|
|
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-46"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-46">¶</a> </div> <p>Unary tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">UNARY = </span><span class="p">[</span><span class="s1">'!'</span><span class="p">,</span> <span class="s1">'~'</span><span class="p">,</span> <span class="s1">'NEW'</span><span class="p">,</span> <span class="s1">'TYPEOF'</span><span class="p">,</span> <span class="s1">'DELETE'</span><span class="p">,</span> <span class="s1">'DO'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-47"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-47">¶</a> </div> <p>Logical tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LOGIC = </span><span class="p">[</span><span class="s1">'&&'</span><span class="p">,</span> <span class="s1">'||'</span><span class="p">,</span> <span class="s1">'&'</span><span class="p">,</span> <span class="s1">'|'</span><span class="p">,</span> <span class="s1">'^'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-48"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-48">¶</a> </div> <p>Bit-shifting tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">SHIFT = </span><span class="p">[</span><span class="s1">'<<'</span><span class="p">,</span> <span class="s1">'>>'</span><span class="p">,</span> <span class="s1">'>>>'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-49">¶</a> </div> <p>Comparison tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COMPARE = </span><span class="p">[</span><span class="s1">'=='</span><span class="p">,</span> <span class="s1">'!='</span><span class="p">,</span> <span class="s1">'<'</span><span class="p">,</span> <span class="s1">'>'</span><span class="p">,</span> <span class="s1">'<='</span><span class="p">,</span> <span class="s1">'>='</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-50">¶</a> </div> <p>Mathematical tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">MATH = </span><span class="p">[</span><span class="s1">'*'</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">,</span> <span class="s1">'%'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-51"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-51">¶</a> </div> <p>Relational tokens that are negatable with <code>not</code> prefix.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RELATION = </span><span class="p">[</span><span class="s1">'IN'</span><span class="p">,</span> <span class="s1">'OF'</span><span class="p">,</span> <span class="s1">'INSTANCEOF'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-52">¶</a> </div> <p>Boolean tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">BOOL = </span><span class="p">[</span><span class="s1">'TRUE'</span><span class="p">,</span> <span class="s1">'FALSE'</span><span class="p">,</span> <span class="s1">'NULL'</span><span class="p">,</span> <span class="s1">'UNDEFINED'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-53">¶</a> </div> <p>Tokens which a regular expression will never immediately follow, but which
|
|
a division operator might.</p>
|
|
|
|
<p>See: http://www.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions</p>
|
|
|
|
<p>Our list is shorter, due to sans-parentheses method calls.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_REGEX = </span><span class="p">[</span><span class="s1">'NUMBER'</span><span class="p">,</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="s1">'BOOL'</span><span class="p">,</span> <span class="s1">'++'</span><span class="p">,</span> <span class="s1">'--'</span><span class="p">,</span> <span class="s1">']'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-54"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-54">¶</a> </div> <p>If the previous token is not spaced, there are more preceding tokens that
|
|
force a division parse:</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_SPACED_REGEX = </span><span class="nx">NOT_REGEX</span><span class="p">.</span><span class="nx">concat</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">'}'</span><span class="p">,</span> <span class="s1">'THIS'</span><span class="p">,</span> <span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'STRING'</span></pre></div> </td> </tr> <tr id="section-55"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-55">¶</a> </div> <p>Tokens which could legitimately be invoked or indexed. An opening
|
|
parentheses or bracket following these tokens will be recorded as the start
|
|
of a function invocation or indexing operation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CALLABLE = </span><span class="p">[</span><span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">']'</span><span class="p">,</span> <span class="s1">'}'</span><span class="p">,</span> <span class="s1">'?'</span><span class="p">,</span> <span class="s1">'::'</span><span class="p">,</span> <span class="s1">'@'</span><span class="p">,</span> <span class="s1">'THIS'</span><span class="p">,</span> <span class="s1">'SUPER'</span><span class="p">]</span>
|
|
<span class="nv">INDEXABLE = </span><span class="nx">CALLABLE</span><span class="p">.</span><span class="nx">concat</span> <span class="s1">'NUMBER'</span><span class="p">,</span> <span class="s1">'BOOL'</span></pre></div> </td> </tr> <tr id="section-56"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-56">¶</a> </div> <p>Tokens that, when immediately preceding a <code>WHEN</code>, indicate that the <code>WHEN</code>
|
|
occurs at the start of a line. We disambiguate these from trailing whens to
|
|
avoid an ambiguity in the grammar.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LINE_BREAK = </span><span class="p">[</span><span class="s1">'INDENT'</span><span class="p">,</span> <span class="s1">'OUTDENT'</span><span class="p">,</span> <span class="s1">'TERMINATOR'</span><span class="p">]</span>
|
|
|
|
</pre></div> </td> </tr> </tbody> </table> </div> </body> </html> |