mirror of
https://github.com/jashkenas/coffeescript.git
synced 2022-11-09 12:23:24 -05:00
545 lines
No EOL
114 KiB
HTML
545 lines
No EOL
114 KiB
HTML
<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To … <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> lexer.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt
|
|
matches against the beginning of the source code. When a match is found,
|
|
a token is produced, we consume the match, and start again. Tokens are in the
|
|
form:</p>
|
|
|
|
<pre><code>[tag, value, lineNumber]
|
|
</code></pre>
|
|
|
|
<p>Which is a format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">Rewriter</span><span class="p">,</span> <span class="nx">INVERSES</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'./rewriter'</span></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> <p>Import the helpers we need.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">count</span><span class="p">,</span> <span class="nx">starts</span><span class="p">,</span> <span class="nx">compact</span><span class="p">,</span> <span class="nx">last</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span> <span class="s">'./helpers'</span></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> <h2>The Lexer Class</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged
|
|
tokens. Some potential ambiguity in the grammar has been avoided by
|
|
pushing some extra smarts into the Lexer.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">exports.Lexer = </span><span class="k">class</span> <span class="nx">Lexer</span></pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> <p><strong>tokenize</strong> is the Lexer's main method. Scan by attempting to match tokens
|
|
one at a time, using a regular expression anchored at the start of the
|
|
remaining code, or a custom recursive token-matching method
|
|
(for interpolations). When the next token has been recorded, we move forward
|
|
within the code past the token, and begin again.</p>
|
|
|
|
<p>Each tokenizing method is responsible for returning the number of characters
|
|
it has consumed.</p>
|
|
|
|
<p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>
|
|
unless explicitly asked not to.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tokenize: </span><span class="nf">(code, opts = {}) -></span>
|
|
<span class="nv">code = </span><span class="s">"\n</span><span class="si">#{</span><span class="nx">code</span><span class="si">}</span><span class="s">"</span> <span class="k">if</span> <span class="nx">WHITESPACE</span><span class="p">.</span><span class="nx">test</span> <span class="nx">code</span>
|
|
<span class="nv">code = </span><span class="nx">code</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/\r/g</span><span class="p">,</span> <span class="s">''</span><span class="p">).</span><span class="nx">replace</span> <span class="nx">TRAILING_SPACES</span><span class="p">,</span> <span class="s">''</span>
|
|
|
|
<span class="vi">@code = </span><span class="nx">code</span> <span class="c1"># The remainder of the source code.</span>
|
|
<span class="vi">@line = </span><span class="nx">opts</span><span class="p">.</span><span class="nx">line</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The current line.</span>
|
|
<span class="vi">@indent = </span><span class="mi">0</span> <span class="c1"># The current indentation level.</span>
|
|
<span class="vi">@indebt = </span><span class="mi">0</span> <span class="c1"># The over-indentation at the current level.</span>
|
|
<span class="vi">@outdebt = </span><span class="mi">0</span> <span class="c1"># The under-outdentation at the current level.</span>
|
|
<span class="vi">@indents = </span><span class="p">[]</span> <span class="c1"># The stack of all current indentation levels.</span>
|
|
<span class="vi">@ends = </span><span class="p">[]</span> <span class="c1"># The stack for pairing up tokens.</span>
|
|
<span class="vi">@tokens = </span><span class="p">[]</span> <span class="c1"># Stream of parsed tokens in the form `['TYPE', value, line]`.</span></pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> <p>At every position, run through this list of attempted matches,
|
|
short-circuiting if any of them succeed. Their order determines precedence:
|
|
<code>@literalToken</code> is the fallback catch-all.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">i = </span><span class="mi">0</span>
|
|
<span class="k">while</span> <span class="vi">@chunk = </span><span class="nx">code</span><span class="p">[</span><span class="nx">i</span><span class="p">..]</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">@identifierToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@commentToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@whitespaceToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@lineToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@heredocToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@stringToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@numberToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@regexToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@jsToken</span><span class="p">()</span> <span class="o">or</span>
|
|
<span class="nx">@literalToken</span><span class="p">()</span>
|
|
|
|
<span class="nx">@closeIndentation</span><span class="p">()</span>
|
|
<span class="nx">@error</span> <span class="s">"missing </span><span class="si">#{</span><span class="nx">tag</span><span class="si">}</span><span class="s">"</span> <span class="k">if</span> <span class="nv">tag = </span><span class="nx">@ends</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="k">return</span> <span class="nx">@tokens</span> <span class="k">if</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">rewrite</span> <span class="o">is</span> <span class="kc">off</span>
|
|
<span class="p">(</span><span class="k">new</span> <span class="nx">Rewriter</span><span class="p">).</span><span class="nx">rewrite</span> <span class="nx">@tokens</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> <p>Matches identifying literals: variables, keywords, method names, etc.
|
|
Check to ensure that JavaScript reserved words aren't being used as
|
|
identifiers. Because CoffeeScript reserves a handful of keywords that are
|
|
allowed in JavaScript, we're careful not to tag them as keywords when
|
|
referenced as property names here, so you can still do <code>jQuery.is()</code> even
|
|
though <code>is</code> means <code>===</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">identifierToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">IDENTIFIER</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="p">[</span><span class="nx">input</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">colon</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
|
|
<span class="k">if</span> <span class="nx">id</span> <span class="o">is</span> <span class="s">'own'</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s">'FOR'</span>
|
|
<span class="nx">@token</span> <span class="s">'OWN'</span><span class="p">,</span> <span class="nx">id</span>
|
|
<span class="k">return</span> <span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">forcedIdentifier = </span><span class="nx">colon</span> <span class="o">or</span>
|
|
<span class="p">(</span><span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">)</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="p">[</span><span class="s">'.'</span><span class="p">,</span> <span class="s">'?.'</span><span class="p">,</span> <span class="s">'::'</span><span class="p">]</span> <span class="o">or</span>
|
|
<span class="o">not</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'@'</span><span class="p">)</span>
|
|
<span class="nv">tag = </span><span class="s">'IDENTIFIER'</span>
|
|
|
|
<span class="k">if</span> <span class="o">not</span> <span class="nx">forcedIdentifier</span> <span class="o">and</span> <span class="p">(</span><span class="nx">id</span> <span class="k">in</span> <span class="nx">JS_KEYWORDS</span> <span class="o">or</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">COFFEE_KEYWORDS</span><span class="p">)</span>
|
|
<span class="nv">tag = </span><span class="nx">id</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">()</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'WHEN'</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="k">in</span> <span class="nx">LINE_BREAK</span>
|
|
<span class="nv">tag = </span><span class="s">'LEADING_WHEN'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'FOR'</span>
|
|
<span class="vi">@seenFor = </span><span class="kc">yes</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'UNLESS'</span>
|
|
<span class="nv">tag = </span><span class="s">'IF'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">in</span> <span class="nx">UNARY</span>
|
|
<span class="nv">tag = </span><span class="s">'UNARY'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">in</span> <span class="nx">RELATION</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">isnt</span> <span class="s">'INSTANCEOF'</span> <span class="o">and</span> <span class="nx">@seenFor</span>
|
|
<span class="nv">tag = </span><span class="s">'FOR'</span> <span class="o">+</span> <span class="nx">tag</span>
|
|
<span class="vi">@seenFor = </span><span class="kc">no</span>
|
|
<span class="k">else</span>
|
|
<span class="nv">tag = </span><span class="s">'RELATION'</span>
|
|
<span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">'!'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nv">id = </span><span class="s">'!'</span> <span class="o">+</span> <span class="nx">id</span>
|
|
|
|
<span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">JS_FORBIDDEN</span>
|
|
<span class="k">if</span> <span class="nx">forcedIdentifier</span>
|
|
<span class="nv">tag = </span><span class="s">'IDENTIFIER'</span>
|
|
<span class="nv">id = </span><span class="k">new</span> <span class="nb">String</span> <span class="nx">id</span>
|
|
<span class="nv">id.reserved = </span><span class="kc">yes</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">RESERVED</span>
|
|
<span class="nx">@error</span> <span class="s">"reserved word \"</span><span class="si">#{</span><span class="nx">id</span><span class="si">}</span><span class="s">\""</span>
|
|
|
|
<span class="k">unless</span> <span class="nx">forcedIdentifier</span>
|
|
<span class="nv">id = </span><span class="nx">COFFEE_ALIAS_MAP</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="k">if</span> <span class="nx">id</span> <span class="k">in</span> <span class="nx">COFFEE_ALIASES</span>
|
|
<span class="nv">tag = </span><span class="k">switch</span> <span class="nx">id</span>
|
|
<span class="k">when</span> <span class="s">'!'</span> <span class="k">then</span> <span class="s">'UNARY'</span>
|
|
<span class="k">when</span> <span class="s">'=='</span><span class="p">,</span> <span class="s">'!='</span> <span class="k">then</span> <span class="s">'COMPARE'</span>
|
|
<span class="k">when</span> <span class="s">'&&'</span><span class="p">,</span> <span class="s">'||'</span> <span class="k">then</span> <span class="s">'LOGIC'</span>
|
|
<span class="k">when</span> <span class="s">'true'</span><span class="p">,</span> <span class="s">'false'</span> <span class="k">then</span> <span class="s">'BOOL'</span>
|
|
<span class="k">when</span> <span class="s">'break'</span><span class="p">,</span> <span class="s">'continue'</span> <span class="k">then</span> <span class="s">'STATEMENT'</span>
|
|
<span class="k">else</span> <span class="nx">tag</span>
|
|
|
|
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">id</span>
|
|
<span class="nx">@token</span> <span class="s">':'</span><span class="p">,</span> <span class="s">':'</span> <span class="k">if</span> <span class="nx">colon</span>
|
|
<span class="nx">input</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> <p>Matches numbers, including decimals, hex, and exponential notation.
|
|
Be careful not to interfere with ranges-in-progress.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">numberToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">NUMBER</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">number = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="k">if</span> <span class="sr">/^0[BOX]/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
|
|
<span class="nx">@error</span> <span class="s">"radix prefix '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must be lowercase"</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="sr">/E/</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">number</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="sr">/^0x/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
|
|
<span class="nx">@error</span> <span class="s">"exponential notation '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must be indicated with a lowercase 'e'"</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="sr">/^0\d*[89]/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
|
|
<span class="nx">@error</span> <span class="s">"decimal literal '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must not be prefixed with '0'"</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="sr">/^0\d+/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">number</span>
|
|
<span class="nx">@error</span> <span class="s">"octal literal '</span><span class="si">#{</span><span class="nx">number</span><span class="si">}</span><span class="s">' must be prefixed with '0o'"</span>
|
|
<span class="nv">lexedLength = </span><span class="nx">number</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nv">octalLiteral = </span><span class="sr">/^0o([0-7]+)/</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">number</span>
|
|
<span class="nv">number = </span><span class="s">'0x'</span> <span class="o">+</span> <span class="p">(</span><span class="nb">parseInt</span> <span class="nx">octalLiteral</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="mi">8</span><span class="p">).</span><span class="nx">toString</span> <span class="mi">16</span>
|
|
<span class="k">if</span> <span class="nv">binaryLiteral = </span><span class="sr">/^0b([01]+)/</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">number</span>
|
|
<span class="nv">number = </span><span class="s">'0x'</span> <span class="o">+</span> <span class="p">(</span><span class="nb">parseInt</span> <span class="nx">binaryLiteral</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="mi">2</span><span class="p">).</span><span class="nx">toString</span> <span class="mi">16</span>
|
|
<span class="nx">@token</span> <span class="s">'NUMBER'</span><span class="p">,</span> <span class="nx">number</span>
|
|
<span class="nx">lexedLength</span></pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> <p>Matches strings, including multi-line strings. Ensures that quotation marks
|
|
are balanced within the string's contents, and within nested interpolations.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">stringToken: </span><span class="nf">-></span>
|
|
<span class="k">switch</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
|
|
<span class="k">when</span> <span class="s">"'"</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">SIMPLESTR</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="p">(</span><span class="nv">string = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]).</span><span class="nx">replace</span> <span class="nx">MULTILINER</span><span class="p">,</span> <span class="s">'\\\n'</span>
|
|
<span class="k">when</span> <span class="s">'"'</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">string = </span><span class="nx">@balancedString</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="s">'"'</span>
|
|
<span class="k">if</span> <span class="mi">0</span> <span class="o"><</span> <span class="nx">string</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s">'#{'</span><span class="p">,</span> <span class="mi">1</span>
|
|
<span class="nx">@interpolateString</span> <span class="nx">string</span><span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
|
|
<span class="k">else</span>
|
|
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="nx">@escapeLines</span> <span class="nx">string</span>
|
|
<span class="k">else</span>
|
|
<span class="k">return</span> <span class="mi">0</span>
|
|
<span class="k">if</span> <span class="nv">octalEsc = </span><span class="sr">/^(?:\\.|[^\\])*\\(?:0[0-7]|[1-7])/</span><span class="p">.</span><span class="nx">test</span> <span class="nx">string</span>
|
|
<span class="nx">@error</span> <span class="s">"octal escape sequences </span><span class="si">#{</span><span class="nx">string</span><span class="si">}</span><span class="s"> are not allowed"</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">string</span><span class="p">,</span> <span class="s">'\n'</span>
|
|
<span class="nx">string</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> <p>Matches heredocs, adjusting indentation to the correct level, as heredocs
|
|
preserve whitespace, but ignore indentation to the left.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">heredocToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">HEREDOC</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">heredoc = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="nv">quote = </span><span class="nx">heredoc</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
|
|
<span class="nv">doc = </span><span class="nx">@sanitizeHeredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="nv">quote: </span><span class="nx">quote</span><span class="p">,</span> <span class="nv">indent: </span><span class="kc">null</span>
|
|
<span class="k">if</span> <span class="nx">quote</span> <span class="o">is</span> <span class="s">'"'</span> <span class="o">and</span> <span class="mi">0</span> <span class="o"><=</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s">'#{'</span>
|
|
<span class="nx">@interpolateString</span> <span class="nx">doc</span><span class="p">,</span> <span class="nv">heredoc: </span><span class="kc">yes</span>
|
|
<span class="k">else</span>
|
|
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span> <span class="nx">doc</span><span class="p">,</span> <span class="nx">quote</span><span class="p">,</span> <span class="kc">yes</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">heredoc</span><span class="p">,</span> <span class="s">'\n'</span>
|
|
<span class="nx">heredoc</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> <p>Matches and consumes comments.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">commentToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">COMMENT</span>
|
|
<span class="p">[</span><span class="nx">comment</span><span class="p">,</span> <span class="nx">here</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="k">if</span> <span class="nx">here</span>
|
|
<span class="nx">@token</span> <span class="s">'HERECOMMENT'</span><span class="p">,</span> <span class="nx">@sanitizeHeredoc</span> <span class="nx">here</span><span class="p">,</span>
|
|
<span class="nv">herecomment: </span><span class="kc">true</span><span class="p">,</span> <span class="nv">indent: </span><span class="nb">Array</span><span class="p">(</span><span class="nx">@indent</span> <span class="o">+</span> <span class="mi">1</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="s">' '</span><span class="p">)</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">comment</span><span class="p">,</span> <span class="s">'\n'</span>
|
|
<span class="nx">comment</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> <p>Matches JavaScript interpolated directly into the source via backticks.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">jsToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="s">'`'</span> <span class="o">and</span> <span class="nv">match = </span><span class="nx">JSTOKEN</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nx">@token</span> <span class="s">'JS'</span><span class="p">,</span> <span class="p">(</span><span class="nv">script = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">])[</span><span class="mi">1</span><span class="p">...</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">script</span><span class="p">,</span> <span class="s">'\n'</span>
|
|
<span class="nx">script</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> <p>Matches regular expression literals. Lexing regular expressions is difficult
|
|
to distinguish from division, so we borrow some basic heuristics from
|
|
JavaScript and Ruby.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">regexToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">isnt</span> <span class="s">'/'</span>
|
|
<span class="k">if</span> <span class="nv">match = </span><span class="nx">HEREGEX</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">length = </span><span class="nx">@heregexToken</span> <span class="nx">match</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s">'\n'</span>
|
|
<span class="k">return</span> <span class="nx">length</span>
|
|
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="p">(</span><span class="k">if</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span> <span class="k">then</span> <span class="nx">NOT_REGEX</span> <span class="k">else</span> <span class="nx">NOT_SPACED_REGEX</span><span class="p">))</span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">REGEX</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="p">[</span><span class="nx">match</span><span class="p">,</span> <span class="nx">regex</span><span class="p">,</span> <span class="nx">flags</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="k">if</span> <span class="nx">regex</span><span class="p">[..</span><span class="mi">1</span><span class="p">]</span> <span class="o">is</span> <span class="s">'/*'</span> <span class="k">then</span> <span class="nx">@error</span> <span class="s">'regular expressions cannot begin with `*`'</span>
|
|
<span class="k">if</span> <span class="nx">regex</span> <span class="o">is</span> <span class="s">'//'</span> <span class="k">then</span> <span class="nv">regex = </span><span class="s">'/(?:)/'</span>
|
|
<span class="nx">@token</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">"</span><span class="si">#{</span><span class="nx">regex</span><span class="si">}#{</span><span class="nx">flags</span><span class="si">}</span><span class="s">"</span>
|
|
<span class="nx">match</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-15">¶</a> </div> <p>Matches multiline extended regular expressions.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">heregexToken: </span><span class="nf">(match) -></span>
|
|
<span class="p">[</span><span class="nx">heregex</span><span class="p">,</span> <span class="nx">body</span><span class="p">,</span> <span class="nx">flags</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="k">if</span> <span class="mi">0</span> <span class="o">></span> <span class="nx">body</span><span class="p">.</span><span class="nx">indexOf</span> <span class="s">'#{'</span>
|
|
<span class="nv">re = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">HEREGEX_OMIT</span><span class="p">,</span> <span class="s">''</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/\//g</span><span class="p">,</span> <span class="s">'\\/'</span><span class="p">)</span>
|
|
<span class="k">if</span> <span class="nx">re</span><span class="p">.</span><span class="nx">match</span> <span class="sr">/^\*/</span> <span class="k">then</span> <span class="nx">@error</span> <span class="s">'regular expressions cannot begin with `*`'</span>
|
|
<span class="nx">@token</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">"/</span><span class="si">#{</span> <span class="nx">re</span> <span class="o">or</span> <span class="s">'(?:)'</span> <span class="si">}</span><span class="s">/</span><span class="si">#{</span><span class="nx">flags</span><span class="si">}</span><span class="s">"</span>
|
|
<span class="k">return</span> <span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nx">@token</span> <span class="s">'IDENTIFIER'</span><span class="p">,</span> <span class="s">'RegExp'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'CALL_START'</span><span class="p">,</span> <span class="s">'('</span><span class="p">]</span>
|
|
<span class="nv">tokens = </span><span class="p">[]</span>
|
|
<span class="k">for</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">]</span> <span class="k">in</span> <span class="nx">@interpolateString</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nv">regex: </span><span class="kc">yes</span><span class="p">)</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'TOKENS'</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">value</span><span class="p">...</span>
|
|
<span class="k">else</span>
|
|
<span class="k">continue</span> <span class="k">unless</span> <span class="nv">value = </span><span class="nx">value</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">HEREGEX_OMIT</span><span class="p">,</span> <span class="s">''</span>
|
|
<span class="nv">value = </span><span class="nx">value</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/\\/g</span><span class="p">,</span> <span class="s">'\\\\'</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span><span class="p">(</span><span class="nx">value</span><span class="p">,</span> <span class="s">'"'</span><span class="p">,</span> <span class="kc">yes</span><span class="p">)]</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'+'</span><span class="p">,</span> <span class="s">'+'</span><span class="p">]</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'STRING'</span><span class="p">,</span> <span class="s">'""'</span><span class="p">],</span> <span class="p">[</span><span class="s">'+'</span><span class="p">,</span> <span class="s">'+'</span><span class="p">]</span> <span class="k">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">?</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'STRING'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">tokens</span><span class="p">...</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">','</span><span class="p">,</span> <span class="s">','</span><span class="p">],</span> <span class="p">[</span><span class="s">'STRING'</span><span class="p">,</span> <span class="s">'"'</span> <span class="o">+</span> <span class="nx">flags</span> <span class="o">+</span> <span class="s">'"'</span><span class="p">]</span> <span class="k">if</span> <span class="nx">flags</span>
|
|
<span class="nx">@token</span> <span class="s">')'</span><span class="p">,</span> <span class="s">')'</span>
|
|
<span class="nx">heregex</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-16">¶</a> </div> <p>Matches newlines, indents, and outdents, and determines which is which.
|
|
If we can detect that the current line is continued onto the the next line,
|
|
then the newline is suppressed:</p>
|
|
|
|
<pre><code>elements
|
|
.each( ... )
|
|
.map( ... )
|
|
</code></pre>
|
|
|
|
<p>Keeps track of the level of indentation, because a single outdent token
|
|
can close multiple indents, so we need to know how far in we happen to be.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">lineToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="nv">match = </span><span class="nx">MULTI_DENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="nv">indent = </span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="nx">@line</span> <span class="o">+=</span> <span class="nx">count</span> <span class="nx">indent</span><span class="p">,</span> <span class="s">'\n'</span>
|
|
<span class="vi">@seenFor = </span><span class="kc">no</span>
|
|
<span class="nv">size = </span><span class="nx">indent</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span> <span class="o">-</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">lastIndexOf</span> <span class="s">'\n'</span>
|
|
<span class="nv">noNewlines = </span><span class="nx">@unfinished</span><span class="p">()</span>
|
|
<span class="k">if</span> <span class="nx">size</span> <span class="o">-</span> <span class="nx">@indebt</span> <span class="o">is</span> <span class="nx">@indent</span>
|
|
<span class="k">if</span> <span class="nx">noNewlines</span> <span class="k">then</span> <span class="nx">@suppressNewlines</span><span class="p">()</span> <span class="k">else</span> <span class="nx">@newlineToken</span><span class="p">()</span>
|
|
<span class="k">return</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nx">size</span> <span class="o">></span> <span class="nx">@indent</span>
|
|
<span class="k">if</span> <span class="nx">noNewlines</span>
|
|
<span class="vi">@indebt = </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span>
|
|
<span class="nx">@suppressNewlines</span><span class="p">()</span>
|
|
<span class="k">return</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">diff = </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span> <span class="o">+</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">@token</span> <span class="s">'INDENT'</span><span class="p">,</span> <span class="nx">diff</span>
|
|
<span class="nx">@indents</span><span class="p">.</span><span class="nx">push</span> <span class="nx">diff</span>
|
|
<span class="nx">@ends</span><span class="p">.</span><span class="nx">push</span> <span class="s">'OUTDENT'</span>
|
|
<span class="vi">@outdebt = @indebt = </span><span class="mi">0</span>
|
|
<span class="k">else</span>
|
|
<span class="vi">@indebt = </span><span class="mi">0</span>
|
|
<span class="nx">@outdentToken</span> <span class="nx">@indent</span> <span class="o">-</span> <span class="nx">size</span><span class="p">,</span> <span class="nx">noNewlines</span>
|
|
<span class="vi">@indent = </span><span class="nx">size</span>
|
|
<span class="nx">indent</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-17">¶</a> </div> <p>Record an outdent token or multiple tokens, if we happen to be moving back
|
|
inwards past several recorded indents.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">outdentToken: </span><span class="nf">(moveOut, noNewlines) -></span>
|
|
<span class="k">while</span> <span class="nx">moveOut</span> <span class="o">></span> <span class="mi">0</span>
|
|
<span class="nv">len = </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
|
<span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">is</span> <span class="kc">undefined</span>
|
|
<span class="nv">moveOut = </span><span class="mi">0</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">is</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">@outdebt</span>
|
|
<span class="vi">@outdebt = </span><span class="mi">0</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o"><</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">@outdebt</span> <span class="o">-=</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span>
|
|
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">@indents</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span>
|
|
<span class="k">else</span>
|
|
<span class="nv">dent = </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="o">-</span> <span class="nx">@outdebt</span>
|
|
<span class="nx">moveOut</span> <span class="o">-=</span> <span class="nx">dent</span>
|
|
<span class="vi">@outdebt = </span><span class="mi">0</span>
|
|
<span class="nx">@pair</span> <span class="s">'OUTDENT'</span>
|
|
<span class="nx">@token</span> <span class="s">'OUTDENT'</span><span class="p">,</span> <span class="nx">dent</span>
|
|
<span class="nx">@outdebt</span> <span class="o">-=</span> <span class="nx">moveOut</span> <span class="k">if</span> <span class="nx">dent</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">while</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">';'</span>
|
|
<span class="nx">@token</span> <span class="s">'TERMINATOR'</span><span class="p">,</span> <span class="s">'\n'</span> <span class="k">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s">'TERMINATOR'</span> <span class="o">or</span> <span class="nx">noNewlines</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-18">¶</a> </div> <p>Matches and consumes non-meaningful whitespace. Tag the previous token
|
|
as being "spaced", because there are some cases where it makes a difference.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">whitespaceToken: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="mi">0</span> <span class="k">unless</span> <span class="p">(</span><span class="nv">match = </span><span class="nx">WHITESPACE</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span><span class="p">)</span> <span class="o">or</span>
|
|
<span class="p">(</span><span class="nv">nline = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">is</span> <span class="s">'\n'</span><span class="p">)</span>
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="k">if</span> <span class="nx">match</span> <span class="k">then</span> <span class="s">'spaced'</span> <span class="k">else</span> <span class="s">'newLine'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">true</span> <span class="k">if</span> <span class="nx">prev</span>
|
|
<span class="k">if</span> <span class="nx">match</span> <span class="k">then</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span> <span class="k">else</span> <span class="mi">0</span></pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-19">¶</a> </div> <p>Generate a newline token. Consecutive newlines get merged together.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">newlineToken: </span><span class="nf">-></span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">while</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">';'</span>
|
|
<span class="nx">@token</span> <span class="s">'TERMINATOR'</span><span class="p">,</span> <span class="s">'\n'</span> <span class="k">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s">'TERMINATOR'</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-20">¶</a> </div> <p>Use a <code>\</code> at a line-ending to suppress the newline.
|
|
The slash is removed here once its job is done.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">suppressNewlines: </span><span class="nf">-></span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s">'\\'</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-21">¶</a> </div> <p>We treat all other single characters as a token. E.g.: <code>( ) , . !</code>
|
|
Multi-character operators are also literal tokens, so that Jison can assign
|
|
the proper order of operations. There are some symbols that we tag specially
|
|
here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish
|
|
parentheses that indicate a method call from regular parentheses, and so on.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">literalToken: </span><span class="nf">-></span>
|
|
<span class="k">if</span> <span class="nv">match = </span><span class="nx">OPERATOR</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">@chunk</span>
|
|
<span class="p">[</span><span class="nx">value</span><span class="p">]</span> <span class="o">=</span> <span class="nx">match</span>
|
|
<span class="nx">@tagParameters</span><span class="p">()</span> <span class="k">if</span> <span class="nx">CODE</span><span class="p">.</span><span class="nx">test</span> <span class="nx">value</span>
|
|
<span class="k">else</span>
|
|
<span class="nv">value = </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">charAt</span> <span class="mi">0</span>
|
|
<span class="nv">tag = </span><span class="nx">value</span>
|
|
<span class="nv">prev = </span><span class="nx">last</span> <span class="nx">@tokens</span>
|
|
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'='</span> <span class="o">and</span> <span class="nx">prev</span>
|
|
<span class="k">if</span> <span class="o">not</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">reserved</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">in</span> <span class="nx">JS_FORBIDDEN</span>
|
|
<span class="nx">@error</span> <span class="s">"reserved word \"</span><span class="si">#{</span><span class="nx">@value</span><span class="p">()</span><span class="si">}</span><span class="s">\" can't be assigned"</span>
|
|
<span class="k">if</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">in</span> <span class="p">[</span><span class="s">'||'</span><span class="p">,</span> <span class="s">'&&'</span><span class="p">]</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'COMPOUND_ASSIGN'</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="s">'='</span>
|
|
<span class="k">return</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">';'</span>
|
|
<span class="vi">@seenFor = </span><span class="kc">no</span>
|
|
<span class="nv">tag = </span><span class="s">'TERMINATOR'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">MATH</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'MATH'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">COMPARE</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'COMPARE'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">COMPOUND_ASSIGN</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'COMPOUND_ASSIGN'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">UNARY</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'UNARY'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">SHIFT</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'SHIFT'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="k">in</span> <span class="nx">LOGIC</span> <span class="o">or</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'?'</span> <span class="o">and</span> <span class="nx">prev</span><span class="o">?</span><span class="p">.</span><span class="nx">spaced</span> <span class="k">then</span> <span class="nv">tag = </span><span class="s">'LOGIC'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">prev</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev</span><span class="p">.</span><span class="nx">spaced</span>
|
|
<span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'('</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="nx">CALLABLE</span>
|
|
<span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'FUNC_EXIST'</span> <span class="k">if</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'?'</span>
|
|
<span class="nv">tag = </span><span class="s">'CALL_START'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s">'['</span> <span class="o">and</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">in</span> <span class="nx">INDEXABLE</span>
|
|
<span class="nv">tag = </span><span class="s">'INDEX_START'</span>
|
|
<span class="k">switch</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="k">when</span> <span class="s">'?'</span> <span class="k">then</span> <span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'INDEX_SOAK'</span>
|
|
<span class="k">switch</span> <span class="nx">value</span>
|
|
<span class="k">when</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'{'</span><span class="p">,</span> <span class="s">'['</span> <span class="k">then</span> <span class="nx">@ends</span><span class="p">.</span><span class="nx">push</span> <span class="nx">INVERSES</span><span class="p">[</span><span class="nx">value</span><span class="p">]</span>
|
|
<span class="k">when</span> <span class="s">')'</span><span class="p">,</span> <span class="s">'}'</span><span class="p">,</span> <span class="s">']'</span> <span class="k">then</span> <span class="nx">@pair</span> <span class="nx">value</span>
|
|
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
|
|
<span class="nx">value</span><span class="p">.</span><span class="nx">length</span></pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-22">¶</a> </div> <h2>Token Manipulators</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-23">¶</a> </div> <p>Sanitize a heredoc or herecomment by
|
|
erasing all external indentation on the left-hand side.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">sanitizeHeredoc: </span><span class="nf">(doc, options) -></span>
|
|
<span class="p">{</span><span class="nx">indent</span><span class="p">,</span> <span class="nx">herecomment</span><span class="p">}</span> <span class="o">=</span> <span class="nx">options</span>
|
|
<span class="k">if</span> <span class="nx">herecomment</span>
|
|
<span class="k">if</span> <span class="nx">HEREDOC_ILLEGAL</span><span class="p">.</span><span class="nx">test</span> <span class="nx">doc</span>
|
|
<span class="nx">@error</span> <span class="s">"block comment cannot contain \"*/\", starting"</span>
|
|
<span class="k">return</span> <span class="nx">doc</span> <span class="k">if</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="s">'\n'</span><span class="p">)</span> <span class="o"><=</span> <span class="mi">0</span>
|
|
<span class="k">else</span>
|
|
<span class="k">while</span> <span class="nv">match = </span><span class="nx">HEREDOC_INDENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">doc</span>
|
|
<span class="nv">attempt = </span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
|
|
<span class="nv">indent = </span><span class="nx">attempt</span> <span class="k">if</span> <span class="nx">indent</span> <span class="o">is</span> <span class="kc">null</span> <span class="o">or</span> <span class="mi">0</span> <span class="o"><</span> <span class="nx">attempt</span><span class="p">.</span><span class="nx">length</span> <span class="o"><</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">doc = </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/// \n </span><span class="si">#{</span><span class="nx">indent</span><span class="si">}</span><span class="sr"> ///g</span><span class="p">,</span> <span class="s">'\n'</span> <span class="k">if</span> <span class="nx">indent</span>
|
|
<span class="nv">doc = </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/^\n/</span><span class="p">,</span> <span class="s">''</span> <span class="k">unless</span> <span class="nx">herecomment</span>
|
|
<span class="nx">doc</span></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-24">¶</a> </div> <p>A source of ambiguity in our grammar used to be parameter lists in function
|
|
definitions versus argument lists in function calls. Walk backwards, tagging
|
|
parameters specially in order to make things easier for the parser.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tagParameters: </span><span class="nf">-></span>
|
|
<span class="k">return</span> <span class="k">this</span> <span class="k">if</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">isnt</span> <span class="s">')'</span>
|
|
<span class="nv">stack = </span><span class="p">[]</span>
|
|
<span class="p">{</span><span class="nx">tokens</span><span class="p">}</span> <span class="o">=</span> <span class="k">this</span>
|
|
<span class="nv">i = </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nx">tokens</span><span class="p">[</span><span class="o">--</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'PARAM_END'</span>
|
|
<span class="k">while</span> <span class="nv">tok = </span><span class="nx">tokens</span><span class="p">[</span><span class="o">--</span><span class="nx">i</span><span class="p">]</span>
|
|
<span class="k">switch</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
|
<span class="k">when</span> <span class="s">')'</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nx">tok</span>
|
|
<span class="k">when</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'CALL_START'</span>
|
|
<span class="k">if</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="k">then</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'('</span>
|
|
<span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">'PARAM_START'</span>
|
|
<span class="k">return</span> <span class="k">this</span>
|
|
<span class="k">else</span> <span class="k">return</span> <span class="k">this</span>
|
|
<span class="k">this</span></pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-25">¶</a> </div> <p>Close up all remaining open blocks at the end of the file.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">closeIndentation: </span><span class="nf">-></span>
|
|
<span class="nx">@outdentToken</span> <span class="nx">@indent</span></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-26">¶</a> </div> <p>Matches a balanced group such as a single or double-quoted string. Pass in
|
|
a series of delimiters, all of which must be nested correctly within the
|
|
contents of the string. This method allows us to have strings within
|
|
interpolations within strings, ad infinitum.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">balancedString: </span><span class="nf">(str, end) -></span>
|
|
<span class="nv">continueCount = </span><span class="mi">0</span>
|
|
<span class="nv">stack = </span><span class="p">[</span><span class="nx">end</span><span class="p">]</span>
|
|
<span class="k">for</span> <span class="nx">i</span> <span class="k">in</span> <span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="nx">str</span><span class="p">.</span><span class="nx">length</span><span class="p">]</span>
|
|
<span class="k">if</span> <span class="nx">continueCount</span>
|
|
<span class="o">--</span><span class="nx">continueCount</span>
|
|
<span class="k">continue</span>
|
|
<span class="k">switch</span> <span class="nv">letter = </span><span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span> <span class="nx">i</span>
|
|
<span class="k">when</span> <span class="s">'\\'</span>
|
|
<span class="o">++</span><span class="nx">continueCount</span>
|
|
<span class="k">continue</span>
|
|
<span class="k">when</span> <span class="nx">end</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="k">unless</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">return</span> <span class="nx">str</span><span class="p">[</span><span class="mi">0</span><span class="p">..</span><span class="nx">i</span><span class="p">]</span>
|
|
<span class="nv">end = </span><span class="nx">stack</span><span class="p">[</span><span class="nx">stack</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span>
|
|
<span class="k">continue</span>
|
|
<span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="k">in</span> <span class="p">[</span><span class="s">'"'</span><span class="p">,</span> <span class="s">"'"</span><span class="p">]</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="nx">letter</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'/'</span> <span class="o">and</span> <span class="nv">match = </span><span class="p">(</span><span class="nx">HEREGEX</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">str</span><span class="p">[</span><span class="nx">i</span><span class="p">..])</span> <span class="o">or</span> <span class="nx">REGEX</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">str</span><span class="p">[</span><span class="nx">i</span><span class="p">..]))</span>
|
|
<span class="nx">continueCount</span> <span class="o">+=</span> <span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'}'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'{'</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="s">'}'</span>
|
|
<span class="k">else</span> <span class="k">if</span> <span class="nx">end</span> <span class="o">is</span> <span class="s">'"'</span> <span class="o">and</span> <span class="nx">prev</span> <span class="o">is</span> <span class="s">'#'</span> <span class="o">and</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'{'</span>
|
|
<span class="nx">stack</span><span class="p">.</span><span class="nx">push</span> <span class="nv">end = </span><span class="s">'}'</span>
|
|
<span class="nv">prev = </span><span class="nx">letter</span>
|
|
<span class="nx">@error</span> <span class="s">"missing </span><span class="si">#{</span> <span class="nx">stack</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="si">}</span><span class="s">, starting"</span></pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-27">¶</a> </div> <p>Expand variables and expressions inside double-quoted strings using
|
|
Ruby-like notation for substitution of arbitrary expressions.</p>
|
|
|
|
<pre><code>"Hello #{name.capitalize()}."
|
|
</code></pre>
|
|
|
|
<p>If it encounters an interpolation, this method will recursively create a
|
|
new Lexer, tokenize the interpolated contents, and merge them into the
|
|
token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">interpolateString: </span><span class="nf">(str, options = {}) -></span>
|
|
<span class="p">{</span><span class="nx">heredoc</span><span class="p">,</span> <span class="nx">regex</span><span class="p">}</span> <span class="o">=</span> <span class="nx">options</span>
|
|
<span class="nv">tokens = </span><span class="p">[]</span>
|
|
<span class="nv">pi = </span><span class="mi">0</span>
|
|
<span class="nv">i = </span><span class="o">-</span><span class="mi">1</span>
|
|
<span class="k">while</span> <span class="nv">letter = </span><span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span> <span class="nx">i</span> <span class="o">+=</span> <span class="mi">1</span>
|
|
<span class="k">if</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'\\'</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="mi">1</span>
|
|
<span class="k">continue</span>
|
|
<span class="k">unless</span> <span class="nx">letter</span> <span class="o">is</span> <span class="s">'#'</span> <span class="o">and</span> <span class="nx">str</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="nx">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">is</span> <span class="s">'{'</span> <span class="o">and</span>
|
|
<span class="p">(</span><span class="nv">expr = </span><span class="nx">@balancedString</span> <span class="nx">str</span><span class="p">[</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">..],</span> <span class="s">'}'</span><span class="p">)</span>
|
|
<span class="k">continue</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'NEOSTRING'</span><span class="p">,</span> <span class="nx">str</span><span class="p">[</span><span class="nx">pi</span><span class="p">...</span><span class="nx">i</span><span class="p">]]</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">i</span>
|
|
<span class="nv">inner = </span><span class="nx">expr</span><span class="p">[</span><span class="mi">1</span><span class="p">...</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
|
|
<span class="k">if</span> <span class="nx">inner</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">nested = </span><span class="k">new</span> <span class="nx">Lexer</span><span class="p">().</span><span class="nx">tokenize</span> <span class="nx">inner</span><span class="p">,</span> <span class="nv">line: </span><span class="nx">@line</span><span class="p">,</span> <span class="nv">rewrite: </span><span class="kc">off</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">shift</span><span class="p">()</span> <span class="k">if</span> <span class="nx">nested</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">?</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'TERMINATOR'</span>
|
|
<span class="k">if</span> <span class="nv">len = </span><span class="nx">nested</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">if</span> <span class="nx">len</span> <span class="o">></span> <span class="mi">1</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">unshift</span> <span class="p">[</span><span class="s">'('</span><span class="p">,</span> <span class="s">'('</span><span class="p">,</span> <span class="nx">@line</span><span class="p">]</span>
|
|
<span class="nx">nested</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">')'</span><span class="p">,</span> <span class="s">')'</span><span class="p">,</span> <span class="nx">@line</span><span class="p">]</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'TOKENS'</span><span class="p">,</span> <span class="nx">nested</span><span class="p">]</span>
|
|
<span class="nx">i</span> <span class="o">+=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nv">pi = </span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s">'NEOSTRING'</span><span class="p">,</span> <span class="nx">str</span><span class="p">[</span><span class="nx">pi</span><span class="p">..]]</span> <span class="k">if</span> <span class="nx">i</span> <span class="o">></span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="k">return</span> <span class="nx">tokens</span> <span class="k">if</span> <span class="nx">regex</span>
|
|
<span class="k">return</span> <span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="s">'""'</span> <span class="k">unless</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span>
|
|
<span class="nx">tokens</span><span class="p">.</span><span class="nx">unshift</span> <span class="p">[</span><span class="s">''</span><span class="p">,</span> <span class="s">''</span><span class="p">]</span> <span class="k">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s">'NEOSTRING'</span>
|
|
<span class="nx">@token</span> <span class="s">'('</span><span class="p">,</span> <span class="s">'('</span> <span class="k">if</span> <span class="nv">interpolated = </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">></span> <span class="mi">1</span>
|
|
<span class="k">for</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">],</span> <span class="nx">i</span> <span class="k">in</span> <span class="nx">tokens</span>
|
|
<span class="nx">@token</span> <span class="s">'+'</span><span class="p">,</span> <span class="s">'+'</span> <span class="k">if</span> <span class="nx">i</span>
|
|
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s">'TOKENS'</span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="nx">value</span><span class="p">...</span>
|
|
<span class="k">else</span>
|
|
<span class="nx">@token</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="nx">@makeString</span> <span class="nx">value</span><span class="p">,</span> <span class="s">'"'</span><span class="p">,</span> <span class="nx">heredoc</span>
|
|
<span class="nx">@token</span> <span class="s">')'</span><span class="p">,</span> <span class="s">')'</span> <span class="k">if</span> <span class="nx">interpolated</span>
|
|
<span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-28">¶</a> </div> <p>Pairs up a closing token, ensuring that all listed pairs of tokens are
|
|
correctly balanced throughout the course of the token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">pair: </span><span class="nf">(tag) -></span>
|
|
<span class="k">unless</span> <span class="nx">tag</span> <span class="o">is</span> <span class="nv">wanted = </span><span class="nx">last</span> <span class="nx">@ends</span>
|
|
<span class="nx">@error</span> <span class="s">"unmatched </span><span class="si">#{</span><span class="nx">tag</span><span class="si">}</span><span class="s">"</span> <span class="k">unless</span> <span class="s">'OUTDENT'</span> <span class="o">is</span> <span class="nx">wanted</span></pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-29">¶</a> </div> <p>Auto-close INDENT to support syntax like this:</p>
|
|
|
|
<pre><code>el.click((event) ->
|
|
el.hide())
|
|
</code></pre> </td> <td class="code"> <div class="highlight"><pre> <span class="nx">@indent</span> <span class="o">-=</span> <span class="nv">size = </span><span class="nx">last</span> <span class="nx">@indents</span>
|
|
<span class="nx">@outdentToken</span> <span class="nx">size</span><span class="p">,</span> <span class="kc">true</span>
|
|
<span class="k">return</span> <span class="nx">@pair</span> <span class="nx">tag</span>
|
|
<span class="nx">@ends</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-30">¶</a> </div> <h2>Helpers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-31">¶</a> </div> <p>Add a token to the results, taking note of the line number.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">token: </span><span class="nf">(tag, value) -></span>
|
|
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">@line</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-32"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-32">¶</a> </div> <p>Peek at a tag in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag: </span><span class="nf">(index, tag) -></span>
|
|
<span class="p">(</span><span class="nv">tok = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">and</span> <span class="k">if</span> <span class="nx">tag</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="nx">tag</span> <span class="k">else</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-33">¶</a> </div> <p>Peek at a value in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">value: </span><span class="nf">(index, val) -></span>
|
|
<span class="p">(</span><span class="nv">tok = </span><span class="nx">last</span> <span class="nx">@tokens</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">and</span> <span class="k">if</span> <span class="nx">val</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="nx">val</span> <span class="k">else</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-34">¶</a> </div> <p>Are we in the midst of an unfinished expression?</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">unfinished: </span><span class="nf">-></span>
|
|
<span class="nx">LINE_CONTINUER</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">)</span> <span class="o">or</span>
|
|
<span class="nx">@tag</span><span class="p">()</span> <span class="k">in</span> <span class="p">[</span><span class="s">'\\'</span><span class="p">,</span> <span class="s">'.'</span><span class="p">,</span> <span class="s">'?.'</span><span class="p">,</span> <span class="s">'UNARY'</span><span class="p">,</span> <span class="s">'MATH'</span><span class="p">,</span> <span class="s">'+'</span><span class="p">,</span> <span class="s">'-'</span><span class="p">,</span> <span class="s">'SHIFT'</span><span class="p">,</span> <span class="s">'RELATION'</span>
|
|
<span class="s">'COMPARE'</span><span class="p">,</span> <span class="s">'LOGIC'</span><span class="p">,</span> <span class="s">'THROW'</span><span class="p">,</span> <span class="s">'EXTENDS'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-35">¶</a> </div> <p>Converts newlines for string literals.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">escapeLines: </span><span class="nf">(str, heredoc) -></span>
|
|
<span class="nx">str</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">MULTILINER</span><span class="p">,</span> <span class="k">if</span> <span class="nx">heredoc</span> <span class="k">then</span> <span class="s">'\\n'</span> <span class="k">else</span> <span class="s">''</span></pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-36">¶</a> </div> <p>Constructs a string token by escaping quotes and newlines.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">makeString: </span><span class="nf">(body, quote, heredoc) -></span>
|
|
<span class="k">return</span> <span class="nx">quote</span> <span class="o">+</span> <span class="nx">quote</span> <span class="k">unless</span> <span class="nx">body</span>
|
|
<span class="nv">body = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/\\([\s\S])/g</span><span class="p">,</span> <span class="nf">(match, contents) -></span>
|
|
<span class="k">if</span> <span class="nx">contents</span> <span class="k">in</span> <span class="p">[</span><span class="s">'\n'</span><span class="p">,</span> <span class="nx">quote</span><span class="p">]</span> <span class="k">then</span> <span class="nx">contents</span> <span class="k">else</span> <span class="nx">match</span>
|
|
<span class="nv">body = </span><span class="nx">body</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/// </span><span class="si">#{</span><span class="nx">quote</span><span class="si">}</span><span class="sr"> ///g</span><span class="p">,</span> <span class="s">'\\$&'</span>
|
|
<span class="nx">quote</span> <span class="o">+</span> <span class="nx">@escapeLines</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="nx">heredoc</span><span class="p">)</span> <span class="o">+</span> <span class="nx">quote</span></pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-37">¶</a> </div> <p>Throws a syntax error on the current <code>@line</code>.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">error: </span><span class="nf">(message) -></span>
|
|
<span class="k">throw</span> <span class="nx">SyntaxError</span> <span class="s">"</span><span class="si">#{</span><span class="nx">message</span><span class="si">}</span><span class="s"> on line </span><span class="si">#{</span> <span class="nx">@line</span> <span class="o">+</span> <span class="mi">1</span><span class="si">}</span><span class="s">"</span></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-38">¶</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-39">¶</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_KEYWORDS = </span><span class="p">[</span>
|
|
<span class="s">'true'</span><span class="p">,</span> <span class="s">'false'</span><span class="p">,</span> <span class="s">'null'</span><span class="p">,</span> <span class="s">'this'</span>
|
|
<span class="s">'new'</span><span class="p">,</span> <span class="s">'delete'</span><span class="p">,</span> <span class="s">'typeof'</span><span class="p">,</span> <span class="s">'in'</span><span class="p">,</span> <span class="s">'instanceof'</span>
|
|
<span class="s">'return'</span><span class="p">,</span> <span class="s">'throw'</span><span class="p">,</span> <span class="s">'break'</span><span class="p">,</span> <span class="s">'continue'</span><span class="p">,</span> <span class="s">'debugger'</span>
|
|
<span class="s">'if'</span><span class="p">,</span> <span class="s">'else'</span><span class="p">,</span> <span class="s">'switch'</span><span class="p">,</span> <span class="s">'for'</span><span class="p">,</span> <span class="s">'while'</span><span class="p">,</span> <span class="s">'do'</span><span class="p">,</span> <span class="s">'try'</span><span class="p">,</span> <span class="s">'catch'</span><span class="p">,</span> <span class="s">'finally'</span>
|
|
<span class="s">'class'</span><span class="p">,</span> <span class="s">'extends'</span><span class="p">,</span> <span class="s">'super'</span>
|
|
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-40">¶</a> </div> <p>CoffeeScript-only keywords.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COFFEE_KEYWORDS = </span><span class="p">[</span><span class="s">'undefined'</span><span class="p">,</span> <span class="s">'then'</span><span class="p">,</span> <span class="s">'unless'</span><span class="p">,</span> <span class="s">'until'</span><span class="p">,</span> <span class="s">'loop'</span><span class="p">,</span> <span class="s">'of'</span><span class="p">,</span> <span class="s">'by'</span><span class="p">,</span> <span class="s">'when'</span><span class="p">]</span>
|
|
|
|
<span class="nv">COFFEE_ALIAS_MAP =</span>
|
|
<span class="o">and</span> <span class="o">:</span> <span class="s">'&&'</span>
|
|
<span class="o">or</span> <span class="o">:</span> <span class="s">'||'</span>
|
|
<span class="o">is</span> <span class="o">:</span> <span class="s">'=='</span>
|
|
<span class="o">isnt</span> <span class="o">:</span> <span class="s">'!='</span>
|
|
<span class="o">not</span> <span class="o">:</span> <span class="s">'!'</span>
|
|
<span class="kc">yes</span> <span class="o">:</span> <span class="s">'true'</span>
|
|
<span class="kc">no</span> <span class="o">:</span> <span class="s">'false'</span>
|
|
<span class="kc">on</span> <span class="o">:</span> <span class="s">'true'</span>
|
|
<span class="kc">off</span> <span class="o">:</span> <span class="s">'false'</span>
|
|
|
|
<span class="nv">COFFEE_ALIASES = </span><span class="p">(</span><span class="nx">key</span> <span class="k">for</span> <span class="nx">key</span> <span class="k">of</span> <span class="nx">COFFEE_ALIAS_MAP</span><span class="p">)</span>
|
|
<span class="nv">COFFEE_KEYWORDS = </span><span class="nx">COFFEE_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">COFFEE_ALIASES</span></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-41">¶</a> </div> <p>The list of keywords that are reserved by JavaScript, but not used, or are
|
|
used by CoffeeScript internally. We throw an error when these are encountered,
|
|
to avoid having a JavaScript error at runtime.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RESERVED = </span><span class="p">[</span>
|
|
<span class="s">'case'</span><span class="p">,</span> <span class="s">'default'</span><span class="p">,</span> <span class="s">'function'</span><span class="p">,</span> <span class="s">'var'</span><span class="p">,</span> <span class="s">'void'</span><span class="p">,</span> <span class="s">'with'</span><span class="p">,</span> <span class="s">'const'</span><span class="p">,</span> <span class="s">'let'</span><span class="p">,</span> <span class="s">'enum'</span>
|
|
<span class="s">'export'</span><span class="p">,</span> <span class="s">'import'</span><span class="p">,</span> <span class="s">'native'</span><span class="p">,</span> <span class="s">'__hasProp'</span><span class="p">,</span> <span class="s">'__extends'</span><span class="p">,</span> <span class="s">'__slice'</span><span class="p">,</span> <span class="s">'__bind'</span>
|
|
<span class="s">'__indexOf'</span><span class="p">,</span> <span class="s">'implements'</span><span class="p">,</span> <span class="s">'interface'</span><span class="p">,</span> <span class="s">'package'</span><span class="p">,</span> <span class="s">'private'</span><span class="p">,</span> <span class="s">'protected'</span>
|
|
<span class="s">'public'</span><span class="p">,</span> <span class="s">'static'</span><span class="p">,</span> <span class="s">'yield'</span>
|
|
<span class="p">]</span>
|
|
|
|
<span class="nv">STRICT_PROSCRIBED = </span><span class="p">[</span><span class="s">'arguments'</span><span class="p">,</span> <span class="s">'eval'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-42">¶</a> </div> <p>The superset of both JavaScript keywords and reserved words, none of which may
|
|
be used as identifiers or properties.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_FORBIDDEN = </span><span class="nx">JS_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">RESERVED</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">STRICT_PROSCRIBED</span><span class="p">)</span>
|
|
|
|
<span class="nv">exports.RESERVED = </span><span class="nx">RESERVED</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">JS_KEYWORDS</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">COFFEE_KEYWORDS</span><span class="p">).</span><span class="nx">concat</span><span class="p">(</span><span class="nx">STRICT_PROSCRIBED</span><span class="p">)</span>
|
|
<span class="nv">exports.STRICT_PROSCRIBED = </span><span class="nx">STRICT_PROSCRIBED</span></pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-43">¶</a> </div> <p>Token matching regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IDENTIFIER = </span><span class="sr">/// ^</span>
|
|
<span class="sr"> ( [$A-Za-z_\x7f-\uffff][$\w\x7f-\uffff]* )</span>
|
|
<span class="sr"> ( [^\n\S]* : (?!:) )? # Is this a property name?</span>
|
|
<span class="sr">///</span>
|
|
|
|
<span class="nv">NUMBER = </span><span class="sr">///</span>
|
|
<span class="sr"> ^ 0b[01]+ | # binary</span>
|
|
<span class="sr"> ^ 0o[0-7]+ | # octal</span>
|
|
<span class="sr"> ^ 0x[\da-f]+ | # hex</span>
|
|
<span class="sr"> ^ \d*\.?\d+ (?:e[+-]?\d+)? # decimal</span>
|
|
<span class="sr">///i</span>
|
|
|
|
<span class="nv">HEREDOC = </span><span class="sr">/// ^ ("""|''') ([\s\S]*?) (?:\n[^\n\S]*)? \1 ///</span>
|
|
|
|
<span class="nv">OPERATOR = </span><span class="sr">/// ^ (</span>
|
|
<span class="sr"> ?: [-=]> # function</span>
|
|
<span class="sr"> | [-+*/%<>&|^!?=]= # compound assign / compare</span>
|
|
<span class="sr"> | >>>=? # zero-fill right shift</span>
|
|
<span class="sr"> | ([-+:])\1 # doubles</span>
|
|
<span class="sr"> | ([&|<>])\2=? # logic / shift</span>
|
|
<span class="sr"> | \?\. # soak access</span>
|
|
<span class="sr"> | \.{2,3} # range or splat</span>
|
|
<span class="sr">) ///</span>
|
|
|
|
<span class="nv">WHITESPACE = </span><span class="sr">/^[^\n\S]+/</span>
|
|
|
|
<span class="nv">COMMENT = </span><span class="sr">/^###([^#][\s\S]*?)(?:###[^\n\S]*|(?:###)?$)|^(?:\s*#(?!##[^#]).*)+/</span>
|
|
|
|
<span class="nv">CODE = </span><span class="sr">/^[-=]>/</span>
|
|
|
|
<span class="nv">MULTI_DENT = </span><span class="sr">/^(?:\n[^\n\S]*)+/</span>
|
|
|
|
<span class="nv">SIMPLESTR = </span><span class="sr">/^'[^\\']*(?:\\.[^\\']*)*'/</span>
|
|
|
|
<span class="nv">JSTOKEN = </span><span class="sr">/^`[^\\`]*(?:\\.[^\\`]*)*`/</span></pre></div> </td> </tr> <tr id="section-44"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-44">¶</a> </div> <p>Regex-matching-regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">REGEX = </span><span class="sr">/// ^</span>
|
|
<span class="sr"> (/ (?! [\s=] ) # disallow leading whitespace or equals signs</span>
|
|
<span class="sr"> [^ [ / \n \\ ]* # every other thing</span>
|
|
<span class="sr"> (?:</span>
|
|
<span class="sr"> (?: \\[\s\S] # anything escaped</span>
|
|
<span class="sr"> | \[ # character class</span>
|
|
<span class="sr"> [^ \] \n \\ ]*</span>
|
|
<span class="sr"> (?: \\[\s\S] [^ \] \n \\ ]* )*</span>
|
|
<span class="sr"> ]</span>
|
|
<span class="sr"> ) [^ [ / \n \\ ]*</span>
|
|
<span class="sr"> )*</span>
|
|
<span class="sr"> /) ([imgy]{0,4}) (?!\w)</span>
|
|
<span class="sr">///</span>
|
|
|
|
<span class="nv">HEREGEX = </span><span class="sr">/// ^ /{3} ([\s\S]+?) /{3} ([imgy]{0,4}) (?!\w) ///</span>
|
|
|
|
<span class="nv">HEREGEX_OMIT = </span><span class="sr">/\s+(?:#.*)?/g</span></pre></div> </td> </tr> <tr id="section-45"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-45">¶</a> </div> <p>Token cleaning regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">MULTILINER = </span><span class="sr">/\n/g</span>
|
|
|
|
<span class="nv">HEREDOC_INDENT = </span><span class="sr">/\n+([^\n\S]*)/g</span>
|
|
|
|
<span class="nv">HEREDOC_ILLEGAL = </span><span class="sr">/\*\//</span>
|
|
|
|
<span class="nv">LINE_CONTINUER = </span><span class="sr">/// ^ \s* (?: , | \??\.(?![.\d]) | :: ) ///</span>
|
|
|
|
<span class="nv">TRAILING_SPACES = </span><span class="sr">/\s+$/</span></pre></div> </td> </tr> <tr id="section-46"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-46">¶</a> </div> <p>Compound assignment tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COMPOUND_ASSIGN = </span><span class="p">[</span>
|
|
<span class="s">'-='</span><span class="p">,</span> <span class="s">'+='</span><span class="p">,</span> <span class="s">'/='</span><span class="p">,</span> <span class="s">'*='</span><span class="p">,</span> <span class="s">'%='</span><span class="p">,</span> <span class="s">'||='</span><span class="p">,</span> <span class="s">'&&='</span><span class="p">,</span> <span class="s">'?='</span><span class="p">,</span> <span class="s">'<<='</span><span class="p">,</span> <span class="s">'>>='</span><span class="p">,</span> <span class="s">'>>>='</span><span class="p">,</span> <span class="s">'&='</span><span class="p">,</span> <span class="s">'^='</span><span class="p">,</span> <span class="s">'|='</span>
|
|
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-47"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-47">¶</a> </div> <p>Unary tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">UNARY = </span><span class="p">[</span><span class="s">'!'</span><span class="p">,</span> <span class="s">'~'</span><span class="p">,</span> <span class="s">'NEW'</span><span class="p">,</span> <span class="s">'TYPEOF'</span><span class="p">,</span> <span class="s">'DELETE'</span><span class="p">,</span> <span class="s">'DO'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-48"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-48">¶</a> </div> <p>Logical tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LOGIC = </span><span class="p">[</span><span class="s">'&&'</span><span class="p">,</span> <span class="s">'||'</span><span class="p">,</span> <span class="s">'&'</span><span class="p">,</span> <span class="s">'|'</span><span class="p">,</span> <span class="s">'^'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-49">¶</a> </div> <p>Bit-shifting tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">SHIFT = </span><span class="p">[</span><span class="s">'<<'</span><span class="p">,</span> <span class="s">'>>'</span><span class="p">,</span> <span class="s">'>>>'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-50">¶</a> </div> <p>Comparison tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COMPARE = </span><span class="p">[</span><span class="s">'=='</span><span class="p">,</span> <span class="s">'!='</span><span class="p">,</span> <span class="s">'<'</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">'<='</span><span class="p">,</span> <span class="s">'>='</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-51"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-51">¶</a> </div> <p>Mathematical tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">MATH = </span><span class="p">[</span><span class="s">'*'</span><span class="p">,</span> <span class="s">'/'</span><span class="p">,</span> <span class="s">'%'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-52">¶</a> </div> <p>Relational tokens that are negatable with <code>not</code> prefix.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RELATION = </span><span class="p">[</span><span class="s">'IN'</span><span class="p">,</span> <span class="s">'OF'</span><span class="p">,</span> <span class="s">'INSTANCEOF'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-53">¶</a> </div> <p>Boolean tokens.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">BOOL = </span><span class="p">[</span><span class="s">'TRUE'</span><span class="p">,</span> <span class="s">'FALSE'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-54"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-54">¶</a> </div> <p>Tokens which a regular expression will never immediately follow, but which
|
|
a division operator might.</p>
|
|
|
|
<p>See: http://www.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions</p>
|
|
|
|
<p>Our list is shorter, due to sans-parentheses method calls.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_REGEX = </span><span class="p">[</span><span class="s">'NUMBER'</span><span class="p">,</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">'BOOL'</span><span class="p">,</span> <span class="s">'NULL'</span><span class="p">,</span> <span class="s">'UNDEFINED'</span><span class="p">,</span> <span class="s">'++'</span><span class="p">,</span> <span class="s">'--'</span><span class="p">,</span> <span class="s">']'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-55"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-55">¶</a> </div> <p>If the previous token is not spaced, there are more preceding tokens that
|
|
force a division parse:</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_SPACED_REGEX = </span><span class="nx">NOT_REGEX</span><span class="p">.</span><span class="nx">concat</span> <span class="s">')'</span><span class="p">,</span> <span class="s">'}'</span><span class="p">,</span> <span class="s">'THIS'</span><span class="p">,</span> <span class="s">'IDENTIFIER'</span><span class="p">,</span> <span class="s">'STRING'</span></pre></div> </td> </tr> <tr id="section-56"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-56">¶</a> </div> <p>Tokens which could legitimately be invoked or indexed. An opening
|
|
parentheses or bracket following these tokens will be recorded as the start
|
|
of a function invocation or indexing operation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CALLABLE = </span><span class="p">[</span><span class="s">'IDENTIFIER'</span><span class="p">,</span> <span class="s">'STRING'</span><span class="p">,</span> <span class="s">'REGEX'</span><span class="p">,</span> <span class="s">')'</span><span class="p">,</span> <span class="s">']'</span><span class="p">,</span> <span class="s">'}'</span><span class="p">,</span> <span class="s">'?'</span><span class="p">,</span> <span class="s">'::'</span><span class="p">,</span> <span class="s">'@'</span><span class="p">,</span> <span class="s">'THIS'</span><span class="p">,</span> <span class="s">'SUPER'</span><span class="p">]</span>
|
|
<span class="nv">INDEXABLE = </span><span class="nx">CALLABLE</span><span class="p">.</span><span class="nx">concat</span> <span class="s">'NUMBER'</span><span class="p">,</span> <span class="s">'BOOL'</span><span class="p">,</span> <span class="s">'NULL'</span><span class="p">,</span> <span class="s">'UNDEFINED'</span></pre></div> </td> </tr> <tr id="section-57"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-57">¶</a> </div> <p>Tokens that, when immediately preceding a <code>WHEN</code>, indicate that the <code>WHEN</code>
|
|
occurs at the start of a line. We disambiguate these from trailing whens to
|
|
avoid an ambiguity in the grammar.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LINE_BREAK = </span><span class="p">[</span><span class="s">'INDENT'</span><span class="p">,</span> <span class="s">'OUTDENT'</span><span class="p">,</span> <span class="s">'TERMINATOR'</span><span class="p">]</span>
|
|
|
|
</pre></div> </td> </tr> </tbody> </table> </div> </body> </html> |