mirror of
https://github.com/jashkenas/coffeescript.git
synced 2022-11-09 12:23:24 -05:00
rebuilding the docs with new pygments, no errors on '@'
This commit is contained in:
Jeremy Ashkenas
parent
44765907b3
commit
9d2c81ea54
7 changed files with 566 additions and 565 deletions
|
|
@ -32,31 +32,31 @@ of source.</p>
|
|||
unless explicitly asked not to.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tokenize: </span><span class="p">(</span><span class="nx">code</span><span class="p">,</span> <span class="nx">options</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="nx">code</span> <span class="o">:</span> <span class="nx">code</span><span class="p">.</span><span class="nx">replace</span> <span class="o">/</span><span class="p">(</span><span class="err">\</span><span class="nx">r</span><span class="o">|</span><span class="err">\</span><span class="nx">s</span><span class="o">+</span><span class="nx">$</span><span class="p">)</span><span class="o">/</span><span class="nx">g</span><span class="p">,</span> <span class="s1">''</span>
|
||||
<span class="nx">o</span> <span class="o">:</span> <span class="nx">options</span> <span class="o">or</span> <span class="p">{}</span>
|
||||
<span class="err">@</span><span class="nx">code</span> <span class="o">:</span> <span class="nx">code</span> <span class="c1"># The remainder of the source code.</span>
|
||||
<span class="err">@</span><span class="nx">i</span> <span class="o">:</span> <span class="mi">0</span> <span class="c1"># Current character position we're parsing.</span>
|
||||
<span class="err">@</span><span class="nx">line</span> <span class="o">:</span> <span class="nx">o</span><span class="p">.</span><span class="nx">line</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The current line.</span>
|
||||
<span class="err">@</span><span class="nx">indent</span> <span class="o">:</span> <span class="mi">0</span> <span class="c1"># The current indentation level.</span>
|
||||
<span class="err">@</span><span class="nx">indents</span> <span class="o">:</span> <span class="p">[]</span> <span class="c1"># The stack of all current indentation levels.</span>
|
||||
<span class="err">@</span><span class="nx">tokens</span> <span class="o">:</span> <span class="p">[]</span> <span class="c1"># Stream of parsed tokens in the form ['TYPE', value, line]</span>
|
||||
<span class="k">while</span> <span class="err">@</span><span class="nx">i</span> <span class="o"><</span> <span class="err">@</span><span class="nx">code</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="err">@</span><span class="nv">chunk: </span><span class="err">@</span><span class="nx">code</span><span class="p">.</span><span class="nx">slice</span> <span class="err">@</span><span class="nx">i</span>
|
||||
<span class="err">@</span><span class="nx">extract_next_token</span><span class="p">()</span>
|
||||
<span class="err">@</span><span class="nx">close_indentation</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">tokens</span> <span class="k">if</span> <span class="nx">o</span><span class="p">.</span><span class="nx">rewrite</span> <span class="o">is</span> <span class="kc">off</span>
|
||||
<span class="p">(</span><span class="k">new</span> <span class="nx">Rewriter</span><span class="p">()).</span><span class="nx">rewrite</span> <span class="err">@</span><span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-7">#</a> </div> <p>At every position, run through this list of attempted matches,
|
||||
<span class="nx">@code</span> <span class="o">:</span> <span class="nx">code</span> <span class="c1"># The remainder of the source code.</span>
|
||||
<span class="nx">@i</span> <span class="o">:</span> <span class="mi">0</span> <span class="c1"># Current character position we're parsing.</span>
|
||||
<span class="nx">@line</span> <span class="o">:</span> <span class="nx">o</span><span class="p">.</span><span class="nx">line</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The current line.</span>
|
||||
<span class="nx">@indent</span> <span class="o">:</span> <span class="mi">0</span> <span class="c1"># The current indentation level.</span>
|
||||
<span class="nx">@indents</span> <span class="o">:</span> <span class="p">[]</span> <span class="c1"># The stack of all current indentation levels.</span>
|
||||
<span class="nx">@tokens</span> <span class="o">:</span> <span class="p">[]</span> <span class="c1"># Stream of parsed tokens in the form ['TYPE', value, line]</span>
|
||||
<span class="k">while</span> <span class="nx">@i</span> <span class="o"><</span> <span class="nx">@code</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="vi">@chunk: </span><span class="nx">@code</span><span class="p">.</span><span class="nx">slice</span> <span class="nx">@i</span>
|
||||
<span class="nx">@extract_next_token</span><span class="p">()</span>
|
||||
<span class="nx">@close_indentation</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="nx">@tokens</span> <span class="k">if</span> <span class="nx">o</span><span class="p">.</span><span class="nx">rewrite</span> <span class="o">is</span> <span class="kc">off</span>
|
||||
<span class="p">(</span><span class="k">new</span> <span class="nx">Rewriter</span><span class="p">()).</span><span class="nx">rewrite</span> <span class="nx">@tokens</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-7">#</a> </div> <p>At every position, run through this list of attempted matches,
|
||||
short-circuiting if any of them succeed. Their order determines precedence:
|
||||
<code>@literal_token</code> is the fallback catch-all.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">extract_next_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">extension_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">identifier_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">number_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">heredoc_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">regex_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">comment_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">line_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">whitespace_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">js_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">string_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">literal_token</span><span class="p">()</span></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-8">#</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-9">#</a> </div> <p>Language extensions get the highest priority, first chance to tag tokens
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@extension_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@identifier_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@number_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@heredoc_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@regex_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@comment_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@line_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@whitespace_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@js_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@string_token</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="nx">@literal_token</span><span class="p">()</span></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-8">#</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-9">#</a> </div> <p>Language extensions get the highest priority, first chance to tag tokens
|
||||
as something else.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">extension_token: </span><span class="o">-></span>
|
||||
<span class="k">for</span> <span class="nx">extension</span> <span class="k">in</span> <span class="nx">Lexer</span><span class="p">.</span><span class="nx">extensions</span>
|
||||
<span class="k">return</span> <span class="kc">true</span> <span class="k">if</span> <span class="nx">extension</span><span class="p">.</span><span class="nx">call</span> <span class="k">this</span>
|
||||
|
|
@ -66,73 +66,73 @@ identifiers. Because CoffeeScript reserves a handful of keywords that are
|
|||
allowed in JavaScript, we're careful not to tag them as keywords when
|
||||
referenced as property names here, so you can still do <code>jQuery.is()</code> even
|
||||
though <code>is</code> means <code>===</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">identifier_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">id: </span><span class="err">@</span><span class="nx">match</span> <span class="nx">IDENTIFIER</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="err">@</span><span class="nx">name_access_type</span><span class="p">()</span>
|
||||
<span class="nv">accessed: </span><span class="nx">include</span> <span class="nx">ACCESSORS</span><span class="p">,</span> <span class="err">@</span><span class="nx">tag</span> <span class="mi">0</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">id: </span><span class="nx">@match</span> <span class="nx">IDENTIFIER</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nx">@name_access_type</span><span class="p">()</span>
|
||||
<span class="nv">accessed: </span><span class="nx">include</span> <span class="nx">ACCESSORS</span><span class="p">,</span> <span class="nx">@tag</span> <span class="mi">0</span>
|
||||
<span class="nv">tag: </span><span class="s1">'IDENTIFIER'</span>
|
||||
<span class="nv">tag: </span><span class="nx">id</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">()</span> <span class="k">if</span> <span class="o">not</span> <span class="nx">accessed</span> <span class="o">and</span> <span class="nx">include</span><span class="p">(</span><span class="nx">KEYWORDS</span><span class="p">,</span> <span class="nx">id</span><span class="p">)</span>
|
||||
<span class="err">@</span><span class="nx">identifier_error</span> <span class="nx">id</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">RESERVED</span><span class="p">,</span> <span class="nx">id</span>
|
||||
<span class="nv">tag: </span><span class="s1">'LEADING_WHEN'</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'WHEN'</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">LINE_BREAK</span><span class="p">,</span> <span class="err">@</span><span class="nx">tag</span><span class="p">()</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nx">@identifier_error</span> <span class="nx">id</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">RESERVED</span><span class="p">,</span> <span class="nx">id</span>
|
||||
<span class="nv">tag: </span><span class="s1">'LEADING_WHEN'</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'WHEN'</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">LINE_BREAK</span><span class="p">,</span> <span class="nx">@tag</span><span class="p">()</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="k">if</span> <span class="o">not</span> <span class="nx">accessed</span>
|
||||
<span class="nv">tag: id: </span><span class="nx">CONVERSIONS</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">COFFEE_ALIASES</span><span class="p">,</span> <span class="nx">id</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">tag_half_assignment</span> <span class="nx">tag</span> <span class="k">if</span> <span class="err">@</span><span class="nx">prev</span><span class="p">()</span> <span class="o">and</span> <span class="err">@</span><span class="nx">prev</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'ASSIGN'</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">HALF_ASSIGNMENTS</span><span class="p">,</span> <span class="nx">tag</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">id</span>
|
||||
<span class="k">return</span> <span class="nx">@tag_half_assignment</span> <span class="nx">tag</span> <span class="k">if</span> <span class="nx">@prev</span><span class="p">()</span> <span class="o">and</span> <span class="nx">@prev</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'ASSIGN'</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">HALF_ASSIGNMENTS</span><span class="p">,</span> <span class="nx">tag</span>
|
||||
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">id</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-11">#</a> </div> <p>Matches numbers, including decimals, hex, and exponential notation.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">number_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">number: </span><span class="err">@</span><span class="nx">match</span> <span class="nx">NUMBER</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'NUMBER'</span><span class="p">,</span> <span class="nx">number</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">number</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">number: </span><span class="nx">@match</span> <span class="nx">NUMBER</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nx">@token</span> <span class="s1">'NUMBER'</span><span class="p">,</span> <span class="nx">number</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">number</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-12">#</a> </div> <p>Matches strings, including multi-line strings. Ensures that quotation marks
|
||||
are balanced within the string's contents, and within nested interpolations.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">string_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">starts</span><span class="p">(</span><span class="err">@</span><span class="nx">chunk</span><span class="p">,</span> <span class="s1">'"'</span><span class="p">)</span> <span class="o">or</span> <span class="nx">starts</span><span class="p">(</span><span class="err">@</span><span class="nx">chunk</span><span class="p">,</span> <span class="s2">"'"</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">starts</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">,</span> <span class="s1">'"'</span><span class="p">)</span> <span class="o">or</span> <span class="nx">starts</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">,</span> <span class="s2">"'"</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">string:</span>
|
||||
<span class="err">@</span><span class="nx">balanced_token</span><span class="p">([</span><span class="s1">'"'</span><span class="p">,</span> <span class="s1">'"'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'${'</span><span class="p">,</span> <span class="s1">'}'</span><span class="p">])</span> <span class="o">or</span>
|
||||
<span class="err">@</span><span class="nx">balanced_token</span> <span class="p">[</span><span class="s2">"'"</span><span class="p">,</span> <span class="s2">"'"</span><span class="p">]</span>
|
||||
<span class="err">@</span><span class="nx">interpolate_string</span> <span class="nx">string</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">STRING_NEWLINES</span><span class="p">,</span> <span class="s2">" \\\n"</span>
|
||||
<span class="err">@</span><span class="nv">line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">string</span><span class="p">,</span> <span class="s2">"\n"</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nx">@balanced_token</span><span class="p">([</span><span class="s1">'"'</span><span class="p">,</span> <span class="s1">'"'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'${'</span><span class="p">,</span> <span class="s1">'}'</span><span class="p">])</span> <span class="o">or</span>
|
||||
<span class="nx">@balanced_token</span> <span class="p">[</span><span class="s2">"'"</span><span class="p">,</span> <span class="s2">"'"</span><span class="p">]</span>
|
||||
<span class="nx">@interpolate_string</span> <span class="nx">string</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">STRING_NEWLINES</span><span class="p">,</span> <span class="s2">" \\\n"</span>
|
||||
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">string</span><span class="p">,</span> <span class="s2">"\n"</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-13">#</a> </div> <p>Matches heredocs, adjusting indentation to the correct level, as heredocs
|
||||
preserve whitespace, but ignore indentation to the left.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">heredoc_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">match: </span><span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">HEREDOC</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">match: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">HEREDOC</span><span class="p">)</span>
|
||||
<span class="nv">quote: </span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">substr</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nv">doc: </span><span class="err">@</span><span class="nx">sanitize_heredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">or</span> <span class="nx">match</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span> <span class="nx">quote</span>
|
||||
<span class="err">@</span><span class="nx">interpolate_string</span> <span class="s2">"$quote$doc$quote"</span>
|
||||
<span class="err">@</span><span class="nv">line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s2">"\n"</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">length</span>
|
||||
<span class="nv">doc: </span><span class="nx">@sanitize_heredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">or</span> <span class="nx">match</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span> <span class="nx">quote</span>
|
||||
<span class="nx">@interpolate_string</span> <span class="s2">"$quote$doc$quote"</span>
|
||||
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s2">"\n"</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-14">#</a> </div> <p>Matches JavaScript interpolated directly into the source via backticks.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">js_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">starts</span> <span class="err">@</span><span class="nx">chunk</span><span class="p">,</span> <span class="s1">'`'</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">script: </span><span class="err">@</span><span class="nx">balanced_token</span> <span class="p">[</span><span class="s1">'`'</span><span class="p">,</span> <span class="s1">'`'</span><span class="p">]</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'JS'</span><span class="p">,</span> <span class="nx">script</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">JS_CLEANER</span><span class="p">,</span> <span class="s1">''</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">script</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">starts</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="s1">'`'</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">script: </span><span class="nx">@balanced_token</span> <span class="p">[</span><span class="s1">'`'</span><span class="p">,</span> <span class="s1">'`'</span><span class="p">]</span>
|
||||
<span class="nx">@token</span> <span class="s1">'JS'</span><span class="p">,</span> <span class="nx">script</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">JS_CLEANER</span><span class="p">,</span> <span class="s1">''</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">script</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-15">#</a> </div> <p>Matches regular expression literals. Lexing regular expressions is difficult
|
||||
to distinguish from division, so we borrow some basic heuristics from
|
||||
JavaScript and Ruby, borrow slash balancing from <code>@balanced_token</code>, and
|
||||
borrow interpolation from <code>@interpolate_string</code>.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">regex_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">REGEX_START</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">NOT_REGEX</span><span class="p">,</span> <span class="err">@</span><span class="nx">tag</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">regex: </span><span class="err">@</span><span class="nx">balanced_token</span> <span class="p">[</span><span class="s1">'/'</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">]</span>
|
||||
<span class="nv">regex: </span><span class="o">+</span> <span class="p">(</span><span class="nv">flags: </span><span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">substr</span><span class="p">(</span><span class="nx">regex</span><span class="p">.</span><span class="nx">length</span><span class="p">).</span><span class="nx">match</span> <span class="nx">REGEX_FLAGS</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">REGEX_START</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">NOT_REGEX</span><span class="p">,</span> <span class="nx">@tag</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">regex: </span><span class="nx">@balanced_token</span> <span class="p">[</span><span class="s1">'/'</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">]</span>
|
||||
<span class="nv">regex: </span><span class="o">+</span> <span class="p">(</span><span class="nv">flags: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">substr</span><span class="p">(</span><span class="nx">regex</span><span class="p">.</span><span class="nx">length</span><span class="p">).</span><span class="nx">match</span> <span class="nx">REGEX_FLAGS</span><span class="p">)</span>
|
||||
<span class="k">if</span> <span class="nx">regex</span><span class="p">.</span><span class="nx">match</span> <span class="nx">REGEX_INTERPOLATION</span>
|
||||
<span class="nv">str: </span><span class="nx">regex</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span><span class="nx">split</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="nv">str: </span><span class="nx">str</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">REGEX_ESCAPE</span><span class="p">,</span> <span class="p">(</span><span class="nx">escaped</span><span class="p">)</span> <span class="o">-></span> <span class="s1">'\\'</span> <span class="o">+</span> <span class="nx">escaped</span>
|
||||
<span class="err">@</span><span class="nv">tokens: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[[</span><span class="s1">'('</span><span class="p">,</span> <span class="s1">'('</span><span class="p">],</span> <span class="p">[</span><span class="s1">'NEW'</span><span class="p">,</span> <span class="s1">'new'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'RegExp'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'CALL_START'</span><span class="p">,</span> <span class="s1">'('</span><span class="p">]]</span>
|
||||
<span class="err">@</span><span class="nx">interpolate_string</span> <span class="s2">"\"$str\""</span><span class="p">,</span> <span class="kc">yes</span>
|
||||
<span class="err">@</span><span class="nv">tokens: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[[</span><span class="s1">','</span><span class="p">,</span> <span class="s1">','</span><span class="p">],</span> <span class="p">[</span><span class="s1">'STRING'</span><span class="p">,</span> <span class="s2">"\"$flags\""</span><span class="p">],</span> <span class="p">[</span><span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">],</span> <span class="p">[</span><span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">]]</span>
|
||||
<span class="vi">@tokens: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[[</span><span class="s1">'('</span><span class="p">,</span> <span class="s1">'('</span><span class="p">],</span> <span class="p">[</span><span class="s1">'NEW'</span><span class="p">,</span> <span class="s1">'new'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'RegExp'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'CALL_START'</span><span class="p">,</span> <span class="s1">'('</span><span class="p">]]</span>
|
||||
<span class="nx">@interpolate_string</span> <span class="s2">"\"$str\""</span><span class="p">,</span> <span class="kc">yes</span>
|
||||
<span class="vi">@tokens: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[[</span><span class="s1">','</span><span class="p">,</span> <span class="s1">','</span><span class="p">],</span> <span class="p">[</span><span class="s1">'STRING'</span><span class="p">,</span> <span class="s2">"\"$flags\""</span><span class="p">],</span> <span class="p">[</span><span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">],</span> <span class="p">[</span><span class="s1">')'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">]]</span>
|
||||
<span class="k">else</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="nx">regex</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">regex</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nx">@token</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="nx">regex</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">regex</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-16">#</a> </div> <p>Matches a token in which which the passed delimiter pairs must be correctly
|
||||
balanced (ie. strings, JS literals).</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">balanced_token: </span><span class="p">(</span><span class="nx">delimited</span><span class="p">...)</span> <span class="o">-></span>
|
||||
<span class="nx">balanced_string</span> <span class="err">@</span><span class="nx">chunk</span><span class="p">,</span> <span class="nx">delimited</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-17">#</a> </div> <p>Matches and conumes comments. We pass through comments into JavaScript,
|
||||
<span class="nx">balanced_string</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="nx">delimited</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-17">#</a> </div> <p>Matches and conumes comments. We pass through comments into JavaScript,
|
||||
so they're treated as real tokens, like any other part of the language.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">comment_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">comment: </span><span class="err">@</span><span class="nx">match</span> <span class="nx">COMMENT</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="err">@</span><span class="nv">line: </span><span class="o">+</span> <span class="p">(</span><span class="nx">comment</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">)</span> <span class="o">or</span> <span class="p">[]).</span><span class="nx">length</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">comment: </span><span class="nx">@match</span> <span class="nx">COMMENT</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="vi">@line: </span><span class="o">+</span> <span class="p">(</span><span class="nx">comment</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">)</span> <span class="o">or</span> <span class="p">[]).</span><span class="nx">length</span>
|
||||
<span class="nv">lines: </span><span class="nx">compact</span> <span class="nx">comment</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">COMMENT_CLEANER</span><span class="p">,</span> <span class="s1">''</span><span class="p">).</span><span class="nx">split</span> <span class="nx">MULTILINER</span>
|
||||
<span class="nv">i: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
||||
<span class="k">if</span> <span class="err">@</span><span class="nx">unfinished</span><span class="p">()</span>
|
||||
<span class="nv">i: </span><span class="o">-</span> <span class="mi">1</span> <span class="k">while</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">include</span> <span class="nx">LINE_BREAK</span><span class="p">,</span> <span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">'COMMENT'</span><span class="p">,</span> <span class="nx">lines</span><span class="p">,</span> <span class="err">@</span><span class="nx">line</span><span class="p">],</span> <span class="p">[</span><span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s1">'\n'</span><span class="p">,</span> <span class="err">@</span><span class="nx">line</span><span class="p">])</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">comment</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nv">i: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
||||
<span class="k">if</span> <span class="nx">@unfinished</span><span class="p">()</span>
|
||||
<span class="nv">i: </span><span class="o">-</span> <span class="mi">1</span> <span class="k">while</span> <span class="nx">@tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">include</span> <span class="nx">LINE_BREAK</span><span class="p">,</span> <span class="nx">@tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">'COMMENT'</span><span class="p">,</span> <span class="nx">lines</span><span class="p">,</span> <span class="nx">@line</span><span class="p">],</span> <span class="p">[</span><span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s1">'\n'</span><span class="p">,</span> <span class="nx">@line</span><span class="p">])</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">comment</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-18">#</a> </div> <p>Matches newlines, indents, and outdents, and determines which is which.
|
||||
If we can detect that the current line is continued onto the the next line,
|
||||
then the newline is suppressed:</p>
|
||||
|
|
@ -144,103 +144,103 @@ then the newline is suppressed:</p>
|
|||
|
||||
<p>Keeps track of the level of indentation, because a single outdent token
|
||||
can close multiple indents, so we need to know how far in we happen to be.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">line_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">indent: </span><span class="err">@</span><span class="nx">match</span> <span class="nx">MULTI_DENT</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="err">@</span><span class="nv">line: </span><span class="o">+</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">).</span><span class="nx">length</span>
|
||||
<span class="err">@</span><span class="nx">i</span> <span class="o">:</span> <span class="o">+</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nv">prev: </span><span class="err">@</span><span class="nx">prev</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">indent: </span><span class="nx">@match</span> <span class="nx">MULTI_DENT</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">).</span><span class="nx">length</span>
|
||||
<span class="nx">@i</span> <span class="o">:</span> <span class="o">+</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nv">prev: </span><span class="nx">@prev</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="nv">size: </span><span class="nx">indent</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">LAST_DENTS</span><span class="p">).</span><span class="nx">reverse</span><span class="p">()[</span><span class="mi">0</span><span class="p">].</span><span class="nx">match</span><span class="p">(</span><span class="nx">LAST_DENT</span><span class="p">)[</span><span class="mi">1</span><span class="p">].</span><span class="nx">length</span>
|
||||
<span class="nv">next_character: </span><span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTI_DENT</span><span class="p">)[</span><span class="mi">4</span><span class="p">]</span>
|
||||
<span class="nv">no_newlines: </span><span class="nx">next_character</span> <span class="o">is</span> <span class="s1">'.'</span> <span class="o">or</span> <span class="err">@</span><span class="nx">unfinished</span><span class="p">()</span>
|
||||
<span class="k">if</span> <span class="nx">size</span> <span class="o">is</span> <span class="err">@</span><span class="nx">indent</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">suppress_newlines</span><span class="p">()</span> <span class="k">if</span> <span class="nx">no_newlines</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">newline_token</span> <span class="nx">indent</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">size</span> <span class="o">></span> <span class="err">@</span><span class="nx">indent</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">suppress_newlines</span><span class="p">()</span> <span class="k">if</span> <span class="nx">no_newlines</span>
|
||||
<span class="nv">diff: </span><span class="nx">size</span> <span class="o">-</span> <span class="err">@</span><span class="nx">indent</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'INDENT'</span><span class="p">,</span> <span class="nx">diff</span>
|
||||
<span class="err">@</span><span class="nx">indents</span><span class="p">.</span><span class="nx">push</span> <span class="nx">diff</span>
|
||||
<span class="nv">next_character: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTI_DENT</span><span class="p">)[</span><span class="mi">4</span><span class="p">]</span>
|
||||
<span class="nv">no_newlines: </span><span class="nx">next_character</span> <span class="o">is</span> <span class="s1">'.'</span> <span class="o">or</span> <span class="nx">@unfinished</span><span class="p">()</span>
|
||||
<span class="k">if</span> <span class="nx">size</span> <span class="o">is</span> <span class="nx">@indent</span>
|
||||
<span class="k">return</span> <span class="nx">@suppress_newlines</span><span class="p">()</span> <span class="k">if</span> <span class="nx">no_newlines</span>
|
||||
<span class="k">return</span> <span class="nx">@newline_token</span> <span class="nx">indent</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">size</span> <span class="o">></span> <span class="nx">@indent</span>
|
||||
<span class="k">return</span> <span class="nx">@suppress_newlines</span><span class="p">()</span> <span class="k">if</span> <span class="nx">no_newlines</span>
|
||||
<span class="nv">diff: </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span>
|
||||
<span class="nx">@token</span> <span class="s1">'INDENT'</span><span class="p">,</span> <span class="nx">diff</span>
|
||||
<span class="nx">@indents</span><span class="p">.</span><span class="nx">push</span> <span class="nx">diff</span>
|
||||
<span class="k">else</span>
|
||||
<span class="err">@</span><span class="nx">outdent_token</span> <span class="err">@</span><span class="nx">indent</span> <span class="o">-</span> <span class="nx">size</span><span class="p">,</span> <span class="nx">no_newlines</span>
|
||||
<span class="err">@</span><span class="nv">indent: </span><span class="nx">size</span>
|
||||
<span class="nx">@outdent_token</span> <span class="nx">@indent</span> <span class="o">-</span> <span class="nx">size</span><span class="p">,</span> <span class="nx">no_newlines</span>
|
||||
<span class="vi">@indent: </span><span class="nx">size</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-19">#</a> </div> <p>Record an outdent token or multiple tokens, if we happen to be moving back
|
||||
inwards past several recorded indents.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">outdent_token: </span><span class="p">(</span><span class="nx">move_out</span><span class="p">,</span> <span class="nx">no_newlines</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">while</span> <span class="nx">move_out</span> <span class="o">></span> <span class="mi">0</span> <span class="o">and</span> <span class="err">@</span><span class="nx">indents</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nv">last_indent: </span><span class="err">@</span><span class="nx">indents</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'OUTDENT'</span><span class="p">,</span> <span class="nx">last_indent</span>
|
||||
<span class="k">while</span> <span class="nx">move_out</span> <span class="o">></span> <span class="mi">0</span> <span class="o">and</span> <span class="nx">@indents</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nv">last_indent: </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="nx">@token</span> <span class="s1">'OUTDENT'</span><span class="p">,</span> <span class="nx">last_indent</span>
|
||||
<span class="nv">move_out: </span><span class="o">-</span> <span class="nx">last_indent</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s2">"\n"</span> <span class="nx">unless</span> <span class="err">@</span><span class="nx">tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span> <span class="o">or</span> <span class="nx">no_newlines</span>
|
||||
<span class="nx">@token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s2">"\n"</span> <span class="nx">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span> <span class="o">or</span> <span class="nx">no_newlines</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-20">#</a> </div> <p>Matches and consumes non-meaningful whitespace. Tag the previous token
|
||||
as being "spaced", because there are some cases where it makes a difference.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">whitespace_token: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">space: </span><span class="err">@</span><span class="nx">match</span> <span class="nx">WHITESPACE</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nv">prev: </span><span class="err">@</span><span class="nx">prev</span><span class="p">()</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">space: </span><span class="nx">@match</span> <span class="nx">WHITESPACE</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nv">prev: </span><span class="nx">@prev</span><span class="p">()</span>
|
||||
<span class="nv">prev.spaced: </span><span class="kc">true</span> <span class="k">if</span> <span class="nx">prev</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">space</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">space</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-21">#</a> </div> <p>Generate a newline token. Consecutive newlines get merged together.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">newline_token: </span><span class="p">(</span><span class="nx">newlines</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s2">"\n"</span> <span class="nx">unless</span> <span class="err">@</span><span class="nx">tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span>
|
||||
<span class="nx">@token</span> <span class="s1">'TERMINATOR'</span><span class="p">,</span> <span class="s2">"\n"</span> <span class="nx">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'TERMINATOR'</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-22">#</a> </div> <p>Use a <code>\</code> at a line-ending to suppress the newline.
|
||||
The slash is removed here once its job is done.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">suppress_newlines: </span><span class="o">-></span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">if</span> <span class="err">@</span><span class="nx">value</span><span class="p">()</span> <span class="o">is</span> <span class="s2">"\\"</span>
|
||||
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s2">"\\"</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-23">#</a> </div> <p>We treat all other single characters as a token. Eg.: <code>( ) , . !</code>
|
||||
Multi-character operators are also literal tokens, so that Jison can assign
|
||||
the proper order of operations. There are some symbols that we tag specially
|
||||
here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish
|
||||
parentheses that indicate a method call from regular parentheses, and so on.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">literal_token: </span><span class="o">-></span>
|
||||
<span class="nv">match: </span><span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">OPERATOR</span>
|
||||
<span class="nv">match: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">OPERATOR</span>
|
||||
<span class="nv">value: </span><span class="nx">match</span> <span class="o">and</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
|
||||
<span class="nv">space: </span><span class="nx">match</span> <span class="o">and</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
|
||||
<span class="err">@</span><span class="nx">tag_parameters</span><span class="p">()</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">and</span> <span class="nx">value</span><span class="p">.</span><span class="nx">match</span> <span class="nx">CODE</span>
|
||||
<span class="nv">value: </span><span class="o">or</span> <span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">substr</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nv">prev_spaced: </span><span class="err">@</span><span class="nx">prev</span><span class="p">()</span> <span class="o">and</span> <span class="err">@</span><span class="nx">prev</span><span class="p">().</span><span class="nx">spaced</span>
|
||||
<span class="nx">@tag_parameters</span><span class="p">()</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">and</span> <span class="nx">value</span><span class="p">.</span><span class="nx">match</span> <span class="nx">CODE</span>
|
||||
<span class="nv">value: </span><span class="o">or</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">substr</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
|
||||
<span class="nv">prev_spaced: </span><span class="nx">@prev</span><span class="p">()</span> <span class="o">and</span> <span class="nx">@prev</span><span class="p">().</span><span class="nx">spaced</span>
|
||||
<span class="nv">tag: </span><span class="nx">value</span>
|
||||
<span class="k">if</span> <span class="nx">value</span><span class="p">.</span><span class="nx">match</span> <span class="nx">ASSIGNMENT</span>
|
||||
<span class="nv">tag: </span><span class="s1">'ASSIGN'</span>
|
||||
<span class="err">@</span><span class="nx">assignment_error</span><span class="p">()</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">JS_FORBIDDEN</span><span class="p">,</span> <span class="err">@</span><span class="nx">value</span>
|
||||
<span class="nx">@assignment_error</span><span class="p">()</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">JS_FORBIDDEN</span><span class="p">,</span> <span class="nx">@value</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">';'</span>
|
||||
<span class="nv">tag: </span><span class="s1">'TERMINATOR'</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'['</span> <span class="o">and</span> <span class="err">@</span><span class="nx">tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'?'</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev_spaced</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'['</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'?'</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev_spaced</span>
|
||||
<span class="nv">tag: </span><span class="s1">'SOAKED_INDEX_START'</span>
|
||||
<span class="err">@</span><span class="nv">soaked_index: </span><span class="kc">true</span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">']'</span> <span class="o">and</span> <span class="err">@</span><span class="nx">soaked_index</span>
|
||||
<span class="vi">@soaked_index: </span><span class="kc">true</span>
|
||||
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">']'</span> <span class="o">and</span> <span class="nx">@soaked_index</span>
|
||||
<span class="nv">tag: </span><span class="s1">'SOAKED_INDEX_END'</span>
|
||||
<span class="err">@</span><span class="nv">soaked_index: </span><span class="kc">false</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">include</span><span class="p">(</span><span class="nx">CALLABLE</span><span class="p">,</span> <span class="err">@</span><span class="nx">tag</span><span class="p">())</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev_spaced</span>
|
||||
<span class="vi">@soaked_index: </span><span class="kc">false</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">include</span><span class="p">(</span><span class="nx">CALLABLE</span><span class="p">,</span> <span class="nx">@tag</span><span class="p">())</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev_spaced</span>
|
||||
<span class="nv">tag: </span><span class="s1">'CALL_START'</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'('</span>
|
||||
<span class="nv">tag: </span><span class="s1">'INDEX_START'</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">'['</span>
|
||||
<span class="err">@</span><span class="nv">i: </span><span class="o">+</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="k">return</span> <span class="err">@</span><span class="nx">tag_half_assignment</span> <span class="nx">tag</span> <span class="k">if</span> <span class="nx">space</span> <span class="o">and</span> <span class="nx">prev_spaced</span> <span class="o">and</span> <span class="err">@</span><span class="nx">prev</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'ASSIGN'</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">HALF_ASSIGNMENTS</span><span class="p">,</span> <span class="nx">tag</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
|
||||
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="k">return</span> <span class="nx">@tag_half_assignment</span> <span class="nx">tag</span> <span class="k">if</span> <span class="nx">space</span> <span class="o">and</span> <span class="nx">prev_spaced</span> <span class="o">and</span> <span class="nx">@prev</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">'ASSIGN'</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">HALF_ASSIGNMENTS</span><span class="p">,</span> <span class="nx">tag</span>
|
||||
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-24">#</a> </div> <h2>Token Manipulators</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-25">#</a> </div> <p>As we consume a new <code>IDENTIFIER</code>, look at the previous token to determine
|
||||
if it's a special kind of accessor.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">name_access_type: </span><span class="o">-></span>
|
||||
<span class="err">@</span><span class="nx">tag</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'PROTOTYPE_ACCESS'</span><span class="p">)</span> <span class="k">if</span> <span class="err">@</span><span class="nx">value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'::'</span>
|
||||
<span class="k">if</span> <span class="err">@</span><span class="nx">value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'.'</span> <span class="o">and</span> <span class="o">not</span> <span class="p">(</span><span class="err">@</span><span class="nx">value</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'.'</span><span class="p">)</span>
|
||||
<span class="k">if</span> <span class="err">@</span><span class="nx">tag</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'?'</span>
|
||||
<span class="err">@</span><span class="nx">tag</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'SOAK_ACCESS'</span><span class="p">)</span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="nx">@tag</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'PROTOTYPE_ACCESS'</span><span class="p">)</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'::'</span>
|
||||
<span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">'.'</span> <span class="o">and</span> <span class="o">not</span> <span class="p">(</span><span class="nx">@value</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'.'</span><span class="p">)</span>
|
||||
<span class="k">if</span> <span class="nx">@tag</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">is</span> <span class="s1">'?'</span>
|
||||
<span class="nx">@tag</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'SOAK_ACCESS'</span><span class="p">)</span>
|
||||
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="k">else</span>
|
||||
<span class="err">@</span><span class="nx">tag</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'PROPERTY_ACCESS'</span></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-26">#</a> </div> <p>Sanitize a heredoc by escaping internal double quotes and erasing all
|
||||
<span class="nx">@tag</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'PROPERTY_ACCESS'</span></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-26">#</a> </div> <p>Sanitize a heredoc by escaping internal double quotes and erasing all
|
||||
external indentation on the left-hand side.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">sanitize_heredoc: </span><span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">quote</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="nv">indent: </span><span class="p">(</span><span class="nx">doc</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">HEREDOC_INDENT</span><span class="p">)</span> <span class="o">or</span> <span class="p">[</span><span class="s1">''</span><span class="p">]).</span><span class="nx">sort</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="k">new</span> <span class="nb">RegExp</span><span class="p">(</span><span class="s2">"^"</span> <span class="o">+</span><span class="nx">indent</span><span class="p">,</span> <span class="s1">'gm'</span><span class="p">),</span> <span class="s1">''</span><span class="p">)</span>
|
||||
<span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">,</span> <span class="s2">"\\n"</span><span class="p">)</span>
|
||||
<span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="k">new</span> <span class="nb">RegExp</span><span class="p">(</span><span class="nx">quote</span><span class="p">,</span> <span class="s1">'g'</span><span class="p">),</span> <span class="s1">'\\"'</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-27">#</a> </div> <p>Tag a half assignment.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag_half_assignment: </span><span class="p">(</span><span class="nx">tag</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="nv">last: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s2">"$tag="</span><span class="p">,</span> <span class="s2">"$tag="</span><span class="p">,</span> <span class="nx">last</span><span class="p">[</span><span class="mi">2</span><span class="p">]]</span>
|
||||
<span class="nv">last: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s2">"$tag="</span><span class="p">,</span> <span class="s2">"$tag="</span><span class="p">,</span> <span class="nx">last</span><span class="p">[</span><span class="mi">2</span><span class="p">]]</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-28">#</a> </div> <p>A source of ambiguity in our grammar used to be parameter lists in function
|
||||
definitions versus argument lists in function calls. Walk backwards, tagging
|
||||
parameters specially in order to make things easier for the parser.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag_parameters: </span><span class="o">-></span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="err">@</span><span class="nx">tag</span><span class="p">()</span> <span class="o">isnt</span> <span class="s1">')'</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">isnt</span> <span class="s1">')'</span>
|
||||
<span class="nv">i: </span><span class="mi">0</span>
|
||||
<span class="k">while</span> <span class="kc">true</span>
|
||||
<span class="nv">i: </span><span class="o">+</span> <span class="mi">1</span>
|
||||
<span class="nv">tok: </span><span class="err">@</span><span class="nx">prev</span> <span class="nx">i</span>
|
||||
<span class="nv">tok: </span><span class="nx">@prev</span> <span class="nx">i</span>
|
||||
<span class="k">return</span> <span class="k">if</span> <span class="o">not</span> <span class="nx">tok</span>
|
||||
<span class="k">switch</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="k">when</span> <span class="s1">'IDENTIFIER'</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">'PARAM'</span>
|
||||
<span class="k">when</span> <span class="s1">')'</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">'PARAM_END'</span>
|
||||
<span class="k">when</span> <span class="s1">'('</span> <span class="k">then</span> <span class="k">return</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">'PARAM_START'</span>
|
||||
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-29">#</a> </div> <p>Close up all remaining open blocks at the end of the file.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">close_indentation: </span><span class="o">-></span>
|
||||
<span class="err">@</span><span class="nx">outdent_token</span> <span class="err">@</span><span class="nx">indent</span></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-30">#</a> </div> <p>The error for when you try to use a forbidden word in JavaScript as
|
||||
<span class="nx">@outdent_token</span> <span class="nx">@indent</span></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-30">#</a> </div> <p>The error for when you try to use a forbidden word in JavaScript as
|
||||
an identifier.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">identifier_error: </span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s2">"SyntaxError: Reserved word \"$word\" on line ${@line + 1}"</span></pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-31">#</a> </div> <p>The error for when you try to assign to a reserved word in JavaScript,
|
||||
like "function" or "default".</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">assignment_error: </span><span class="o">-></span>
|
||||
|
|
@ -256,7 +256,7 @@ for substitution of bare variables as well as arbitrary expressions.</p>
|
|||
new Lexer, tokenize the interpolated contents, and merge them into the
|
||||
token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">interpolate_string: </span><span class="p">(</span><span class="nx">str</span><span class="p">,</span> <span class="nx">escape_quotes</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">if</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span> <span class="o"><</span> <span class="mi">3</span> <span class="o">or</span> <span class="o">not</span> <span class="nx">starts</span> <span class="nx">str</span><span class="p">,</span> <span class="s1">'"'</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="nx">str</span>
|
||||
<span class="nx">@token</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="nx">str</span>
|
||||
<span class="k">else</span>
|
||||
<span class="nv">lexer: </span> <span class="k">new</span> <span class="nx">Lexer</span><span class="p">()</span>
|
||||
<span class="nv">tokens: </span> <span class="p">[]</span>
|
||||
|
|
@ -276,7 +276,7 @@ token stream.</p> </td> <td class="code">
|
|||
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'STRING'</span><span class="p">,</span> <span class="s2">"$quote${ str.substring(pi, i) }$quote"</span><span class="p">]</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o"><</span> <span class="nx">i</span>
|
||||
<span class="nv">inner: </span><span class="nx">expr</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="k">if</span> <span class="nx">inner</span><span class="p">.</span><span class="nx">length</span>
|
||||
<span class="nv">nested: </span><span class="nx">lexer</span><span class="p">.</span><span class="nx">tokenize</span> <span class="s2">"($inner)"</span><span class="p">,</span> <span class="p">{</span><span class="nv">rewrite: </span><span class="kc">no</span><span class="p">,</span> <span class="nv">line: </span><span class="err">@</span><span class="nx">line</span><span class="p">}</span>
|
||||
<span class="nv">nested: </span><span class="nx">lexer</span><span class="p">.</span><span class="nx">tokenize</span> <span class="s2">"($inner)"</span><span class="p">,</span> <span class="p">{</span><span class="nv">rewrite: </span><span class="kc">no</span><span class="p">,</span> <span class="nv">line: </span><span class="nx">@line</span><span class="p">}</span>
|
||||
<span class="nx">nested</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
|
||||
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">'TOKENS'</span><span class="p">,</span> <span class="nx">nested</span><span class="p">]</span>
|
||||
<span class="k">else</span>
|
||||
|
|
@ -289,28 +289,28 @@ token stream.</p> </td> <td class="code">
|
|||
<span class="k">for</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">i</span> <span class="k">in</span> <span class="nx">tokens</span>
|
||||
<span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">]</span><span class="o">:</span> <span class="nx">token</span>
|
||||
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'TOKENS'</span>
|
||||
<span class="err">@</span><span class="nv">tokens: </span><span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">value</span>
|
||||
<span class="vi">@tokens: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">value</span>
|
||||
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">'STRING'</span> <span class="o">and</span> <span class="nx">escape_quotes</span>
|
||||
<span class="nv">escaped: </span><span class="nx">value</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/"/g</span><span class="p">,</span> <span class="s1">'\\"'</span><span class="p">)</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="nx">tag</span><span class="p">,</span> <span class="s2">"\"$escaped\""</span>
|
||||
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="s2">"\"$escaped\""</span>
|
||||
<span class="k">else</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
|
||||
<span class="err">@</span><span class="nx">token</span> <span class="s1">'+'</span><span class="p">,</span> <span class="s1">'+'</span> <span class="k">if</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
||||
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
|
||||
<span class="nx">@token</span> <span class="s1">'+'</span><span class="p">,</span> <span class="s1">'+'</span> <span class="k">if</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
|
||||
<span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-33">#</a> </div> <h2>Helpers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-34">#</a> </div> <p>Add a token to the results, taking note of the line number.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">token: </span><span class="p">(</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="err">@</span><span class="nx">line</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-35">#</a> </div> <p>Peek at a tag in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag: </span><span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">tag</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">return</span> <span class="nx">unless</span> <span class="nv">tok: </span><span class="err">@</span><span class="nx">prev</span> <span class="nx">index</span>
|
||||
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">@line</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-35">#</a> </div> <p>Peek at a tag in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag: </span><span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">tag</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">return</span> <span class="nx">unless</span> <span class="nv">tok: </span><span class="nx">@prev</span> <span class="nx">index</span>
|
||||
<span class="k">return</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="nx">tag</span> <span class="k">if</span> <span class="nx">tag</span><span class="o">?</span>
|
||||
<span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-36">#</a> </div> <p>Peek at a value in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">value: </span><span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">return</span> <span class="nx">unless</span> <span class="nv">tok: </span><span class="err">@</span><span class="nx">prev</span> <span class="nx">index</span>
|
||||
<span class="k">return</span> <span class="nx">unless</span> <span class="nv">tok: </span><span class="nx">@prev</span> <span class="nx">index</span>
|
||||
<span class="k">return</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">:</span> <span class="nx">val</span> <span class="k">if</span> <span class="nx">val</span><span class="o">?</span>
|
||||
<span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-37">#</a> </div> <p>Peek at a previous token, entire.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">prev: </span><span class="p">(</span><span class="nx">index</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="err">@</span><span class="nx">tokens</span><span class="p">[</span><span class="err">@</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="p">(</span><span class="nx">index</span> <span class="o">or</span> <span class="mi">1</span><span class="p">)]</span></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-38">#</a> </div> <p>Attempt to match a string against the current chunk, returning the indexed
|
||||
<span class="nx">@tokens</span><span class="p">[</span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="p">(</span><span class="nx">index</span> <span class="o">or</span> <span class="mi">1</span><span class="p">)]</span></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-38">#</a> </div> <p>Attempt to match a string against the current chunk, returning the indexed
|
||||
match if successful, and <code>false</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">match: </span><span class="p">(</span><span class="nx">regex</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">-></span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">m: </span><span class="err">@</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">regex</span>
|
||||
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">m: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">regex</span>
|
||||
<span class="k">if</span> <span class="nx">m</span> <span class="k">then</span> <span class="nx">m</span><span class="p">[</span><span class="nx">index</span><span class="p">]</span> <span class="k">else</span> <span class="kc">false</span></pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-39">#</a> </div> <p>Are we in the midst of an unfinished expression?</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">unfinished: </span><span class="o">-></span>
|
||||
<span class="nv">prev: </span><span class="err">@</span><span class="nx">prev</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="err">@</span><span class="nx">value</span><span class="p">()</span> <span class="o">and</span> <span class="err">@</span><span class="nx">value</span><span class="p">().</span><span class="nx">match</span> <span class="o">and</span> <span class="err">@</span><span class="nx">value</span><span class="p">().</span><span class="nx">match</span><span class="p">(</span><span class="nx">NO_NEWLINE</span><span class="p">)</span> <span class="o">and</span>
|
||||
<span class="nx">prev</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">'.'</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="err">@</span><span class="nx">value</span><span class="p">().</span><span class="nx">match</span><span class="p">(</span><span class="nx">CODE</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-40">#</a> </div> <h2>Lexer Properties</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-41">#</a> </div> <p>There are no exensions to the core lexer by default.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="err">@</span><span class="nv">extensions: </span><span class="p">[]</span></pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-42">#</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-43">#</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_KEYWORDS: </span><span class="p">[</span>
|
||||
<span class="nv">prev: </span><span class="nx">@prev</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="nx">@value</span><span class="p">()</span> <span class="o">and</span> <span class="nx">@value</span><span class="p">().</span><span class="nx">match</span> <span class="o">and</span> <span class="nx">@value</span><span class="p">().</span><span class="nx">match</span><span class="p">(</span><span class="nx">NO_NEWLINE</span><span class="p">)</span> <span class="o">and</span>
|
||||
<span class="nx">prev</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">'.'</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">@value</span><span class="p">().</span><span class="nx">match</span><span class="p">(</span><span class="nx">CODE</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-40">#</a> </div> <h2>Lexer Properties</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-41">#</a> </div> <p>There are no exensions to the core lexer by default.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="vi">@extensions: </span><span class="p">[]</span></pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-42">#</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-43">#</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_KEYWORDS: </span><span class="p">[</span>
|
||||
<span class="s2">"if"</span><span class="p">,</span> <span class="s2">"else"</span><span class="p">,</span>
|
||||
<span class="s2">"true"</span><span class="p">,</span> <span class="s2">"false"</span><span class="p">,</span>
|
||||
<span class="s2">"new"</span><span class="p">,</span> <span class="s2">"return"</span><span class="p">,</span>
|
||||
|
|
@ -345,7 +345,7 @@ be used as identifiers or properties.</p> </td> <td clas
|
|||
<span class="nx">LAST_DENTS</span> <span class="o">:</span> <span class="sr">/\n([ \t]*)/g</span>
|
||||
<span class="nx">LAST_DENT</span> <span class="o">:</span> <span class="sr">/\n([ \t]*)/</span>
|
||||
<span class="nx">ASSIGNMENT</span> <span class="o">:</span> <span class="sr">/^(:|=)$/</span></pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-49">#</a> </div> <p>Regex-matching-regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nx">REGEX_START</span> <span class="o">:</span> <span class="sr">/^\/[^\/ ]/</span>
|
||||
<span class="nv">REGEX_INTERPOLATION: </span><span class="sr">/([^\\]\$[a-zA-Z_@]|[^\\]\$\{.*[^\\]\})/</span>
|
||||
<span class="nv">REGEX_INTERPOLATION: </span><span class="o">/</span><span class="p">([</span><span class="o">^</span><span class="err">\\</span><span class="p">]</span><span class="err">\</span><span class="nx">$</span><span class="p">[</span><span class="nx">a</span><span class="o">-</span><span class="nx">zA</span><span class="o">-</span><span class="nx">Z_</span><span class="err">@</span><span class="p">]</span><span class="o">|</span><span class="p">[</span><span class="o">^</span><span class="err">\\</span><span class="p">]</span><span class="err">\</span><span class="nx">$</span><span class="err">\</span><span class="p">{.</span><span class="o">*</span><span class="p">[</span><span class="o">^</span><span class="err">\\</span><span class="p">]</span><span class="err">\</span><span class="p">})</span><span class="o">/</span>
|
||||
<span class="nx">REGEX_FLAGS</span> <span class="o">:</span> <span class="sr">/^[imgy]{0,4}/</span>
|
||||
<span class="nx">REGEX_ESCAPE</span> <span class="o">:</span> <span class="sr">/\\[^\$]/g</span></pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-50">#</a> </div> <p>Token cleaning regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nx">JS_CLEANER</span> <span class="o">:</span> <span class="sr">/(^`|`$)/g</span>
|
||||
<span class="nx">MULTILINER</span> <span class="o">:</span> <span class="sr">/\n/g</span>
|
||||
|
|
@ -361,7 +361,7 @@ a division operator might.</p>
|
|||
<span class="s1">'NUMBER'</span><span class="p">,</span> <span class="s1">'REGEX'</span><span class="p">,</span> <span class="s1">'++'</span><span class="p">,</span> <span class="s1">'--'</span><span class="p">,</span> <span class="s1">'FALSE'</span><span class="p">,</span> <span class="s1">'NULL'</span><span class="p">,</span> <span class="s1">'TRUE'</span>
|
||||
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-52">#</a> </div> <p>Tokens which could legitimately be invoked or indexed. A opening
|
||||
parentheses or bracket following these tokens will be recorded as the start
|
||||
of a function invocation or indexing operation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CALLABLE: </span><span class="p">[</span><span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'SUPER'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">']'</span><span class="p">,</span> <span class="s1">'}'</span><span class="p">,</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'@'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-53">#</a> </div> <p>Tokens that indicate an access -- keywords immediately following will be
|
||||
of a function invocation or indexing operation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CALLABLE: </span><span class="p">[</span><span class="s1">'IDENTIFIER'</span><span class="p">,</span> <span class="s1">'SUPER'</span><span class="p">,</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">']'</span><span class="p">,</span> <span class="s1">'}'</span><span class="p">,</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'@'</span><span class="p">,</span> <span class="s1">'THIS'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-53">#</a> </div> <p>Tokens that indicate an access -- keywords immediately following will be
|
||||
treated as identifiers.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">ACCESSORS: </span><span class="p">[</span><span class="s1">'PROPERTY_ACCESS'</span><span class="p">,</span> <span class="s1">'PROTOTYPE_ACCESS'</span><span class="p">,</span> <span class="s1">'SOAK_ACCESS'</span><span class="p">,</span> <span class="s1">'@'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-54"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-54">#</a> </div> <p>Tokens that, when immediately preceding a <code>WHEN</code>, indicate that the <code>WHEN</code>
|
||||
occurs at the start of a line. We disambiguate these from trailing whens to
|
||||
avoid an ambiguity in the grammar.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LINE_BREAK: </span><span class="p">[</span><span class="s1">'INDENT'</span><span class="p">,</span> <span class="s1">'OUTDENT'</span><span class="p">,</span> <span class="s1">'TERMINATOR'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-55"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-55">#</a> </div> <p>Half-assignments...</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">HALF_ASSIGNMENTS: </span><span class="p">[</span><span class="s1">'-'</span><span class="p">,</span> <span class="s1">'+'</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">,</span> <span class="s1">'%'</span><span class="p">,</span> <span class="s1">'||'</span><span class="p">,</span> <span class="s1">'&&'</span><span class="p">,</span> <span class="s1">'?'</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-56"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-56">#</a> </div> <p>Conversions from CoffeeScript operators into JavaScript ones.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CONVERSIONS: </span><span class="p">{</span>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue