kotovalexarian-likes-github
/
jashkenas--coffeescript
mirror of https://github.com/jashkenas/coffeescript.git
1
0
Fork 0
Code Releases Activity
jashkenas--coffeescript/documentation/docs/lexer.html

383 lines
91 KiB
HTML
Raw Blame History

<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To &hellip; <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> lexer.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-1">#</a> </div> <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt
matches against the beginning of the source code. When a match is found,
a token is produced, we consume the match, and start again. Tokens are in the
form:</p>
<pre><code>[tag, value, line_number]
</code></pre>
<p>Which is a format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>.</p> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-2">#</a> </div> <p>Set up the Lexer for both Node.js and the browser, depending on where we are.</p> </td> <td class="code"> <div class="highlight"><pre><span class="k">if</span> <span class="nx">process</span><span class="o">?</span>
<span class="p">{</span><span class="nx">Rewriter</span><span class="p">}</span><span class="o">:</span> <span class="nx">require</span><span class="p">(</span><span class="s1">&#39;./rewriter&#39;</span><span class="p">)</span>
<span class="p">{</span><span class="nx">helpers</span><span class="p">}</span><span class="o">:</span> <span class="nx">require</span><span class="p">(</span><span class="s1">&#39;./helpers&#39;</span><span class="p">)</span>
<span class="k">else</span>
<span class="k">this</span><span class="p">.</span><span class="nv">exports: </span><span class="k">this</span>
<span class="nv">Rewriter: </span> <span class="k">this</span><span class="p">.</span><span class="nx">Rewriter</span>
<span class="nv">helpers: </span> <span class="k">this</span><span class="p">.</span><span class="nx">helpers</span></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-3">#</a> </div> <p>Import the helpers we need.</p> </td> <td class="code"> <div class="highlight"><pre><span class="p">{</span><span class="nx">include</span><span class="p">,</span> <span class="nx">count</span><span class="p">,</span> <span class="nx">starts</span><span class="p">,</span> <span class="nx">compact</span><span class="p">,</span> <span class="nx">balanced_string</span><span class="p">}</span><span class="o">:</span> <span class="nx">helpers</span></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-4">#</a> </div> <h2>The Lexer Class</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-5">#</a> </div> <p>The Lexer class reads a stream of CoffeeScript and divvys it up into tagged
tokens. Some potential ambiguity in the grammar has been avoided by
pushing some extra smarts into the Lexer.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">exports.Lexer: </span><span class="nx">class</span> <span class="nx">Lexer</span></pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-6">#</a> </div> <p><strong>tokenize</strong> is the Lexer's main method. Scan by attempting to match tokens
one at a time, using a regular expression anchored at the start of the
remaining code, or a custom recursive token-matching method
(for interpolations). When the next token has been recorded, we move forward
within the code past the token, and begin again.</p>
<p>Each tokenizing method is responsible for incrementing <code>@i</code> by the number of
characters it has consumed. <code>@i</code> can be thought of as our finger on the page
of source.</p>
<p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>
unless explicitly asked not to.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tokenize: </span><span class="p">(</span><span class="nx">code</span><span class="p">,</span> <span class="nx">options</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nv">code : </span><span class="nx">code</span><span class="p">.</span><span class="nx">replace</span> <span class="sr">/(\r|\s+$)/g</span><span class="p">,</span> <span class="s1">&#39;&#39;</span>
<span class="nv">o : </span><span class="nx">options</span> <span class="o">or</span> <span class="p">{}</span>
<span class="vi">@code : </span><span class="nx">code</span> <span class="c1"># The remainder of the source code.</span>
<span class="vi">@i : </span><span class="mi">0</span> <span class="c1"># Current character position we&#39;re parsing.</span>
<span class="vi">@line : </span><span class="nx">o</span><span class="p">.</span><span class="nx">line</span> <span class="o">or</span> <span class="mi">0</span> <span class="c1"># The current line.</span>
<span class="vi">@indent : </span><span class="mi">0</span> <span class="c1"># The current indentation level.</span>
<span class="vi">@indents : </span><span class="p">[]</span> <span class="c1"># The stack of all current indentation levels.</span>
<span class="vi">@tokens : </span><span class="p">[]</span> <span class="c1"># Stream of parsed tokens in the form [&#39;TYPE&#39;, value, line]</span>
<span class="k">while</span> <span class="nx">@i</span> <span class="o">&lt;</span> <span class="nx">@code</span><span class="p">.</span><span class="nx">length</span>
<span class="vi">@chunk: </span><span class="nx">@code</span><span class="p">.</span><span class="nx">slice</span> <span class="nx">@i</span>
<span class="nx">@extract_next_token</span><span class="p">()</span>
<span class="nx">@close_indentation</span><span class="p">()</span>
<span class="k">return</span> <span class="nx">@tokens</span> <span class="k">if</span> <span class="nx">o</span><span class="p">.</span><span class="nx">rewrite</span> <span class="o">is</span> <span class="kc">off</span>
<span class="p">(</span><span class="k">new</span> <span class="nx">Rewriter</span><span class="p">()).</span><span class="nx">rewrite</span> <span class="nx">@tokens</span></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-7">#</a> </div> <p>At every position, run through this list of attempted matches,
short-circuiting if any of them succeed. Their order determines precedence:
<code>@literal_token</code> is the fallback catch-all.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">extract_next_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@extension_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@identifier_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@number_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@heredoc_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@regex_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@comment_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@line_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@whitespace_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@js_token</span><span class="p">()</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@string_token</span><span class="p">()</span>
<span class="k">return</span> <span class="nx">@literal_token</span><span class="p">()</span></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-8">#</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-9">#</a> </div> <p>Language extensions get the highest priority, first chance to tag tokens
as something else.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">extension_token: </span><span class="o">-&gt;</span>
<span class="k">for</span> <span class="nx">extension</span> <span class="k">in</span> <span class="nx">Lexer</span><span class="p">.</span><span class="nx">extensions</span>
<span class="k">return</span> <span class="kc">true</span> <span class="k">if</span> <span class="nx">extension</span><span class="p">.</span><span class="nx">call</span> <span class="k">this</span>
<span class="kc">false</span></pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-10">#</a> </div> <p>Matches identifying literals: variables, keywords, method names, etc.
Check to ensure that JavaScript reserved words aren't being used as
identifiers. Because CoffeeScript reserves a handful of keywords that are
allowed in JavaScript, we're careful not to tag them as keywords when
referenced as property names here, so you can still do <code>jQuery.is()</code> even
though <code>is</code> means <code>===</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">identifier_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">id: </span><span class="nx">@match</span> <span class="nx">IDENTIFIER</span><span class="p">,</span> <span class="mi">1</span>
<span class="nx">@name_access_type</span><span class="p">()</span>
<span class="nv">accessed: </span><span class="nx">include</span> <span class="nx">ACCESSORS</span><span class="p">,</span> <span class="nx">@tag</span> <span class="mi">0</span>
<span class="nv">tag: </span><span class="s1">&#39;IDENTIFIER&#39;</span>
<span class="nv">tag: </span><span class="nx">id</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">()</span> <span class="k">if</span> <span class="o">not</span> <span class="nx">accessed</span> <span class="o">and</span> <span class="nx">include</span><span class="p">(</span><span class="nx">KEYWORDS</span><span class="p">,</span> <span class="nx">id</span><span class="p">)</span>
<span class="nx">@identifier_error</span> <span class="nx">id</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">RESERVED</span><span class="p">,</span> <span class="nx">id</span>
<span class="nv">tag: </span><span class="s1">&#39;LEADING_WHEN&#39;</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;WHEN&#39;</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">LINE_BREAK</span><span class="p">,</span> <span class="nx">@tag</span><span class="p">()</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">id</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">unless</span> <span class="nx">accessed</span>
<span class="nv">tag: id: </span><span class="nx">CONVERSIONS</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">COFFEE_ALIASES</span><span class="p">,</span> <span class="nx">id</span>
<span class="k">return</span> <span class="nx">@tag_half_assignment</span> <span class="nx">tag</span> <span class="k">if</span> <span class="nx">@prev</span><span class="p">()</span> <span class="o">and</span> <span class="nx">@prev</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;ASSIGN&#39;</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">HALF_ASSIGNMENTS</span><span class="p">,</span> <span class="nx">tag</span>
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">id</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-11">#</a> </div> <p>Matches numbers, including decimals, hex, and exponential notation.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">number_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">number: </span><span class="nx">@match</span> <span class="nx">NUMBER</span><span class="p">,</span> <span class="mi">1</span>
<span class="nx">@token</span> <span class="s1">&#39;NUMBER&#39;</span><span class="p">,</span> <span class="nx">number</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">number</span><span class="p">.</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-12">#</a> </div> <p>Matches strings, including multi-line strings. Ensures that quotation marks
are balanced within the string's contents, and within nested interpolations.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">string_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">starts</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">,</span> <span class="s1">&#39;&quot;&#39;</span><span class="p">)</span> <span class="o">or</span> <span class="nx">starts</span><span class="p">(</span><span class="nx">@chunk</span><span class="p">,</span> <span class="s2">&quot;&#39;&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">string:</span>
<span class="nx">@balanced_token</span><span class="p">([</span><span class="s1">&#39;&quot;&#39;</span><span class="p">,</span> <span class="s1">&#39;&quot;&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;${&#39;</span><span class="p">,</span> <span class="s1">&#39;}&#39;</span><span class="p">])</span> <span class="o">or</span>
<span class="nx">@balanced_token</span> <span class="p">[</span><span class="s2">&quot;&#39;&quot;</span><span class="p">,</span> <span class="s2">&quot;&#39;&quot;</span><span class="p">]</span>
<span class="nx">@interpolate_string</span> <span class="nx">string</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">STRING_NEWLINES</span><span class="p">,</span> <span class="s2">&quot; \\\n&quot;</span>
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">string</span><span class="p">,</span> <span class="s2">&quot;\n&quot;</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">string</span><span class="p">.</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-13">#</a> </div> <p>Matches heredocs, adjusting indentation to the correct level, as heredocs
preserve whitespace, but ignore indentation to the left.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">heredoc_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">match: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">HEREDOC</span><span class="p">)</span>
<span class="nv">quote: </span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">substr</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
<span class="nv">doc: </span><span class="nx">@sanitize_heredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">or</span> <span class="nx">match</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span> <span class="p">{</span><span class="nx">quote</span><span class="p">}</span>
<span class="nx">@interpolate_string</span> <span class="s2">&quot;$quote$doc$quote&quot;</span>
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s2">&quot;\n&quot;</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-14">#</a> </div> <p>Matches and conumes comments. We pass through comments into JavaScript,
so they're treated as real tokens, like any other part of the language.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">comment_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">match: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">COMMENT</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">match</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="nv">comment: </span><span class="nx">@sanitize_heredoc</span> <span class="nx">match</span><span class="p">[</span><span class="mi">3</span><span class="p">],</span> <span class="p">{</span><span class="nv">herecomment: </span><span class="kc">true</span><span class="p">}</span>
<span class="nx">@token</span> <span class="s1">&#39;HERECOMMENT&#39;</span><span class="p">,</span> <span class="nx">comment</span><span class="p">.</span><span class="nx">split</span> <span class="nx">MULTILINER</span>
<span class="k">else</span>
<span class="nv">lines: </span><span class="nx">compact</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">replace</span><span class="p">(</span><span class="nx">COMMENT_CLEANER</span><span class="p">,</span> <span class="s1">&#39;&#39;</span><span class="p">).</span><span class="nx">split</span> <span class="nx">MULTILINER</span>
<span class="nv">i: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="k">if</span> <span class="nx">@unfinished</span><span class="p">()</span>
<span class="nv">i: </span><span class="o">-</span> <span class="mi">1</span> <span class="k">while</span> <span class="nx">@tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">include</span> <span class="nx">LINE_BREAK</span><span class="p">,</span> <span class="nx">@tokens</span><span class="p">[</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;COMMENT&#39;</span><span class="p">,</span> <span class="nx">lines</span><span class="p">,</span> <span class="nx">@line</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;TERMINATOR&#39;</span><span class="p">,</span> <span class="s1">&#39;\n&#39;</span><span class="p">,</span> <span class="nx">@line</span><span class="p">])</span>
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s2">&quot;\n&quot;</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-15">#</a> </div> <p>Matches JavaScript interpolated directly into the source via backticks.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">js_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">starts</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="s1">&#39;`&#39;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">script: </span><span class="nx">@balanced_token</span> <span class="p">[</span><span class="s1">&#39;`&#39;</span><span class="p">,</span> <span class="s1">&#39;`&#39;</span><span class="p">]</span>
<span class="nx">@token</span> <span class="s1">&#39;JS&#39;</span><span class="p">,</span> <span class="nx">script</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">JS_CLEANER</span><span class="p">,</span> <span class="s1">&#39;&#39;</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">script</span><span class="p">.</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-16">#</a> </div> <p>Matches regular expression literals. Lexing regular expressions is difficult
to distinguish from division, so we borrow some basic heuristics from
JavaScript and Ruby, borrow slash balancing from <code>@balanced_token</code>, and
borrow interpolation from <code>@interpolate_string</code>.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">regex_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">REGEX_START</span>
<span class="k">return</span> <span class="kc">false</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">NOT_REGEX</span><span class="p">,</span> <span class="nx">@tag</span><span class="p">()</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">regex: </span><span class="nx">@balanced_token</span> <span class="p">[</span><span class="s1">&#39;/&#39;</span><span class="p">,</span> <span class="s1">&#39;/&#39;</span><span class="p">]</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">end: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">substr</span><span class="p">(</span><span class="nx">regex</span><span class="p">.</span><span class="nx">length</span><span class="p">).</span><span class="nx">match</span> <span class="nx">REGEX_END</span>
<span class="nv">regex: </span><span class="o">+</span> <span class="nv">flags: </span><span class="nx">end</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="k">if</span> <span class="nx">end</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="k">if</span> <span class="nx">regex</span><span class="p">.</span><span class="nx">match</span> <span class="nx">REGEX_INTERPOLATION</span>
<span class="nv">str: </span><span class="nx">regex</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span><span class="nx">split</span><span class="p">(</span><span class="s1">&#39;/&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="nv">str: </span><span class="nx">str</span><span class="p">.</span><span class="nx">replace</span> <span class="nx">REGEX_ESCAPE</span><span class="p">,</span> <span class="p">(</span><span class="nx">escaped</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="s1">&#39;\\&#39;</span> <span class="o">+</span> <span class="nx">escaped</span>
<span class="vi">@tokens: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[[</span><span class="s1">&#39;(&#39;</span><span class="p">,</span> <span class="s1">&#39;(&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;NEW&#39;</span><span class="p">,</span> <span class="s1">&#39;new&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;IDENTIFIER&#39;</span><span class="p">,</span> <span class="s1">&#39;RegExp&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;CALL_START&#39;</span><span class="p">,</span> <span class="s1">&#39;(&#39;</span><span class="p">]]</span>
<span class="nx">@interpolate_string</span> <span class="s2">&quot;\&quot;$str\&quot;&quot;</span><span class="p">,</span> <span class="kc">yes</span>
<span class="vi">@tokens: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[[</span><span class="s1">&#39;,&#39;</span><span class="p">,</span> <span class="s1">&#39;,&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s2">&quot;\&quot;$flags\&quot;&quot;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">],</span> <span class="p">[</span><span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">]]</span>
<span class="k">else</span>
<span class="nx">@token</span> <span class="s1">&#39;REGEX&#39;</span><span class="p">,</span> <span class="nx">regex</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">regex</span><span class="p">.</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-17">#</a> </div> <p>Matches a token in which which the passed delimiter pairs must be correctly
balanced (ie. strings, JS literals).</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">balanced_token: </span><span class="p">(</span><span class="nx">delimited</span><span class="p">...)</span> <span class="o">-&gt;</span>
<span class="nx">balanced_string</span> <span class="nx">@chunk</span><span class="p">,</span> <span class="nx">delimited</span></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-18">#</a> </div> <p>Matches newlines, indents, and outdents, and determines which is which.
If we can detect that the current line is continued onto the the next line,
then the newline is suppressed:</p>
<pre><code>elements
.each( ... )
.map( ... )
</code></pre>
<p>Keeps track of the level of indentation, because a single outdent token
can close multiple indents, so we need to know how far in we happen to be.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">line_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">indent: </span><span class="nx">@match</span> <span class="nx">MULTI_DENT</span><span class="p">,</span> <span class="mi">1</span>
<span class="vi">@line: </span><span class="o">+</span> <span class="nx">count</span> <span class="nx">indent</span><span class="p">,</span> <span class="s2">&quot;\n&quot;</span>
<span class="vi">@i : </span><span class="o">+</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">prev: </span><span class="nx">@prev</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="nv">size: </span><span class="nx">indent</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">LAST_DENTS</span><span class="p">).</span><span class="nx">reverse</span><span class="p">()[</span><span class="mi">0</span><span class="p">].</span><span class="nx">match</span><span class="p">(</span><span class="nx">LAST_DENT</span><span class="p">)[</span><span class="mi">1</span><span class="p">].</span><span class="nx">length</span>
<span class="nv">next_character: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="nx">MULTI_DENT</span><span class="p">)[</span><span class="mi">4</span><span class="p">]</span>
<span class="nv">no_newlines: </span><span class="nx">next_character</span> <span class="o">is</span> <span class="s1">&#39;.&#39;</span> <span class="o">or</span> <span class="nx">@unfinished</span><span class="p">()</span>
<span class="k">if</span> <span class="nx">size</span> <span class="o">is</span> <span class="nx">@indent</span>
<span class="k">return</span> <span class="nx">@suppress_newlines</span><span class="p">()</span> <span class="k">if</span> <span class="nx">no_newlines</span>
<span class="k">return</span> <span class="nx">@newline_token</span> <span class="nx">indent</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">size</span> <span class="o">&gt;</span> <span class="nx">@indent</span>
<span class="k">return</span> <span class="nx">@suppress_newlines</span><span class="p">()</span> <span class="k">if</span> <span class="nx">no_newlines</span>
<span class="nv">diff: </span><span class="nx">size</span> <span class="o">-</span> <span class="nx">@indent</span>
<span class="nx">@token</span> <span class="s1">&#39;INDENT&#39;</span><span class="p">,</span> <span class="nx">diff</span>
<span class="nx">@indents</span><span class="p">.</span><span class="nx">push</span> <span class="nx">diff</span>
<span class="k">else</span>
<span class="nx">@outdent_token</span> <span class="nx">@indent</span> <span class="o">-</span> <span class="nx">size</span><span class="p">,</span> <span class="nx">no_newlines</span>
<span class="vi">@indent: </span><span class="nx">size</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-19">#</a> </div> <p>Record an outdent token or multiple tokens, if we happen to be moving back
inwards past several recorded indents.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">outdent_token: </span><span class="p">(</span><span class="nx">move_out</span><span class="p">,</span> <span class="nx">no_newlines</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">while</span> <span class="nx">move_out</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="o">and</span> <span class="nx">@indents</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">last_indent: </span><span class="nx">@indents</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nx">@token</span> <span class="s1">&#39;OUTDENT&#39;</span><span class="p">,</span> <span class="nx">last_indent</span>
<span class="nv">move_out: </span><span class="o">-</span> <span class="nx">last_indent</span>
<span class="nx">@token</span> <span class="s1">&#39;TERMINATOR&#39;</span><span class="p">,</span> <span class="s2">&quot;\n&quot;</span> <span class="nx">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">&#39;TERMINATOR&#39;</span> <span class="o">or</span> <span class="nx">no_newlines</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-20">#</a> </div> <p>Matches and consumes non-meaningful whitespace. Tag the previous token
as being "spaced", because there are some cases where it makes a difference.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">whitespace_token: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">space: </span><span class="nx">@match</span> <span class="nx">WHITESPACE</span><span class="p">,</span> <span class="mi">1</span>
<span class="nv">prev: </span><span class="nx">@prev</span><span class="p">()</span>
<span class="nv">prev.spaced: </span><span class="kc">true</span> <span class="k">if</span> <span class="nx">prev</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">space</span><span class="p">.</span><span class="nx">length</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-21">#</a> </div> <p>Generate a newline token. Consecutive newlines get merged together.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">newline_token: </span><span class="p">(</span><span class="nx">newlines</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nx">@token</span> <span class="s1">&#39;TERMINATOR&#39;</span><span class="p">,</span> <span class="s2">&quot;\n&quot;</span> <span class="nx">unless</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">&#39;TERMINATOR&#39;</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-22">#</a> </div> <p>Use a <code>\</code> at a line-ending to suppress the newline.
The slash is removed here once its job is done.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">suppress_newlines: </span><span class="o">-&gt;</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s2">&quot;\\&quot;</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-23">#</a> </div> <p>We treat all other single characters as a token. Eg.: <code>( ) , . !</code>
Multi-character operators are also literal tokens, so that Jison can assign
the proper order of operations. There are some symbols that we tag specially
here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish
parentheses that indicate a method call from regular parentheses, and so on.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">literal_token: </span><span class="o">-&gt;</span>
<span class="nv">match: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">OPERATOR</span>
<span class="nv">value: </span><span class="nx">match</span> <span class="o">and</span> <span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="nv">space: </span><span class="nx">match</span> <span class="o">and</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="nx">@tag_parameters</span><span class="p">()</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">and</span> <span class="nx">value</span><span class="p">.</span><span class="nx">match</span> <span class="nx">CODE</span>
<span class="nv">value: </span><span class="o">or</span> <span class="nx">@chunk</span><span class="p">.</span><span class="nx">substr</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
<span class="nv">prev_spaced: </span><span class="nx">@prev</span><span class="p">()</span> <span class="o">and</span> <span class="nx">@prev</span><span class="p">().</span><span class="nx">spaced</span>
<span class="nv">tag: </span><span class="nx">value</span>
<span class="k">if</span> <span class="nx">value</span><span class="p">.</span><span class="nx">match</span> <span class="nx">ASSIGNMENT</span>
<span class="nv">tag: </span><span class="s1">&#39;ASSIGN&#39;</span>
<span class="nx">@assignment_error</span><span class="p">()</span> <span class="k">if</span> <span class="nx">include</span> <span class="nx">JS_FORBIDDEN</span><span class="p">,</span> <span class="nx">@value</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">&#39;;&#39;</span>
<span class="nv">tag: </span><span class="s1">&#39;TERMINATOR&#39;</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">&#39;[&#39;</span> <span class="o">and</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">is</span> <span class="s1">&#39;?&#39;</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev_spaced</span>
<span class="nv">tag: </span><span class="s1">&#39;SOAKED_INDEX_START&#39;</span>
<span class="vi">@soaked_index: </span><span class="kc">true</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">&#39;]&#39;</span> <span class="o">and</span> <span class="nx">@soaked_index</span>
<span class="nv">tag: </span><span class="s1">&#39;SOAKED_INDEX_END&#39;</span>
<span class="vi">@soaked_index: </span><span class="kc">false</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">include</span><span class="p">(</span><span class="nx">CALLABLE</span><span class="p">,</span> <span class="nx">@tag</span><span class="p">())</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">prev_spaced</span>
<span class="nv">tag: </span><span class="s1">&#39;CALL_START&#39;</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">&#39;(&#39;</span>
<span class="nv">tag: </span><span class="s1">&#39;INDEX_START&#39;</span> <span class="k">if</span> <span class="nx">value</span> <span class="o">is</span> <span class="s1">&#39;[&#39;</span>
<span class="vi">@i: </span><span class="o">+</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span>
<span class="k">return</span> <span class="nx">@tag_half_assignment</span> <span class="nx">tag</span> <span class="k">if</span> <span class="nx">space</span> <span class="o">and</span> <span class="nx">prev_spaced</span> <span class="o">and</span> <span class="nx">@prev</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;ASSIGN&#39;</span> <span class="o">and</span> <span class="nx">include</span> <span class="nx">HALF_ASSIGNMENTS</span><span class="p">,</span> <span class="nx">tag</span>
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-24">#</a> </div> <h2>Token Manipulators</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-25">#</a> </div> <p>As we consume a new <code>IDENTIFIER</code>, look at the previous token to determine
if it's a special kind of accessor.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">name_access_type: </span><span class="o">-&gt;</span>
<span class="nx">@tag</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;PROTOTYPE_ACCESS&#39;</span><span class="p">)</span> <span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">&#39;::&#39;</span>
<span class="k">if</span> <span class="nx">@value</span><span class="p">()</span> <span class="o">is</span> <span class="s1">&#39;.&#39;</span> <span class="o">and</span> <span class="o">not</span> <span class="p">(</span><span class="nx">@value</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">is</span> <span class="s1">&#39;.&#39;</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">@tag</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">is</span> <span class="s1">&#39;?&#39;</span>
<span class="nx">@tag</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;SOAK_ACCESS&#39;</span><span class="p">)</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">else</span>
<span class="nx">@tag</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;PROPERTY_ACCESS&#39;</span></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-26">#</a> </div> <p>Sanitize a heredoc or herecomment by escaping internal double quotes and
erasing all external indentation on the left-hand side.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">sanitize_heredoc: </span><span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">options</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">while</span> <span class="nv">match: </span><span class="nx">HEREDOC_INDENT</span><span class="p">.</span><span class="nx">exec</span> <span class="nx">doc</span>
<span class="nv">attempt: </span><span class="k">if</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="o">?</span> <span class="k">then</span> <span class="nx">match</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="k">else</span> <span class="nx">match</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="nv">indent: </span><span class="nx">attempt</span> <span class="k">if</span> <span class="o">not</span> <span class="nx">indent</span> <span class="o">or</span> <span class="nx">attempt</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="nx">indent</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">doc: </span><span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="k">new</span> <span class="nb">RegExp</span><span class="p">(</span><span class="s2">&quot;^&quot;</span> <span class="o">+</span><span class="nx">indent</span><span class="p">,</span> <span class="s1">&#39;gm&#39;</span><span class="p">),</span> <span class="s1">&#39;&#39;</span><span class="p">)</span>
<span class="k">return</span> <span class="nx">doc</span> <span class="k">if</span> <span class="nx">options</span><span class="p">.</span><span class="nx">herecomment</span>
<span class="nx">doc</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">MULTILINER</span><span class="p">,</span> <span class="s2">&quot;\\n&quot;</span><span class="p">)</span>
<span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="k">new</span> <span class="nb">RegExp</span><span class="p">(</span><span class="nx">options</span><span class="p">.</span><span class="nx">quote</span><span class="p">,</span> <span class="s1">&#39;g&#39;</span><span class="p">),</span> <span class="s1">&#39;\\&quot;&#39;</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-27">#</a> </div> <p>Tag a half assignment.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag_half_assignment: </span><span class="p">(</span><span class="nx">tag</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nv">last: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s2">&quot;$tag=&quot;</span><span class="p">,</span> <span class="s2">&quot;$tag=&quot;</span><span class="p">,</span> <span class="nx">last</span><span class="p">[</span><span class="mi">2</span><span class="p">]]</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-28">#</a> </div> <p>A source of ambiguity in our grammar used to be parameter lists in function
definitions versus argument lists in function calls. Walk backwards, tagging
parameters specially in order to make things easier for the parser.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag_parameters: </span><span class="o">-&gt;</span>
<span class="k">return</span> <span class="k">if</span> <span class="nx">@tag</span><span class="p">()</span> <span class="o">isnt</span> <span class="s1">&#39;)&#39;</span>
<span class="nv">i: </span><span class="mi">0</span>
<span class="k">while</span> <span class="kc">true</span>
<span class="nv">i: </span><span class="o">+</span> <span class="mi">1</span>
<span class="nv">tok: </span><span class="nx">@prev</span> <span class="nx">i</span>
<span class="k">return</span> <span class="k">if</span> <span class="o">not</span> <span class="nx">tok</span>
<span class="k">switch</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">when</span> <span class="s1">&#39;IDENTIFIER&#39;</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">&#39;PARAM&#39;</span>
<span class="k">when</span> <span class="s1">&#39;)&#39;</span> <span class="k">then</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">&#39;PARAM_END&#39;</span>
<span class="k">when</span> <span class="s1">&#39;(&#39;</span><span class="p">,</span> <span class="s1">&#39;CALL_START&#39;</span> <span class="k">then</span> <span class="k">return</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">&#39;PARAM_START&#39;</span>
<span class="kc">true</span></pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-29">#</a> </div> <p>Close up all remaining open blocks at the end of the file.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">close_indentation: </span><span class="o">-&gt;</span>
<span class="nx">@outdent_token</span> <span class="nx">@indent</span></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-30">#</a> </div> <p>The error for when you try to use a forbidden word in JavaScript as
an identifier.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">identifier_error: </span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s2">&quot;SyntaxError: Reserved word \&quot;$word\&quot; on line ${@line + 1}&quot;</span></pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-31">#</a> </div> <p>The error for when you try to assign to a reserved word in JavaScript,
like "function" or "default".</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">assignment_error: </span><span class="o">-&gt;</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span> <span class="s2">&quot;SyntaxError: Reserved word \&quot;${@value()}\&quot; on line ${@line + 1} can&#39;t be assigned&quot;</span></pre></div> </td> </tr> <tr id="section-32"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-32">#</a> </div> <p>Expand variables and expressions inside double-quoted strings using
<a href="http://wiki.ecmascript.org/doku.php?id=strawman:string_interpolation">ECMA Harmony's interpolation syntax</a>
for substitution of bare variables as well as arbitrary expressions.</p>
<pre><code>"Hello $name."
"Hello ${name.capitalize()}."
</code></pre>
<p>If it encounters an interpolation, this method will recursively create a
new Lexer, tokenize the interpolated contents, and merge them into the
token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">interpolate_string: </span><span class="p">(</span><span class="nx">str</span><span class="p">,</span> <span class="nx">escape_quotes</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">if</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="mi">3</span> <span class="o">or</span> <span class="o">not</span> <span class="nx">starts</span> <span class="nx">str</span><span class="p">,</span> <span class="s1">&#39;&quot;&#39;</span>
<span class="nx">@token</span> <span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="nx">str</span>
<span class="k">else</span>
<span class="nv">lexer: </span> <span class="k">new</span> <span class="nx">Lexer</span><span class="p">()</span>
<span class="nv">tokens: </span> <span class="p">[]</span>
<span class="nv">quote: </span> <span class="nx">str</span><span class="p">.</span><span class="nx">substring</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
<span class="p">[</span><span class="nx">i</span><span class="p">,</span> <span class="nx">pi</span><span class="p">]</span><span class="o">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="k">while</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="k">if</span> <span class="nx">starts</span> <span class="nx">str</span><span class="p">,</span> <span class="s1">&#39;\\&#39;</span><span class="p">,</span> <span class="nx">i</span>
<span class="nv">i: </span><span class="o">+</span> <span class="mi">1</span>
<span class="k">else</span> <span class="k">if</span> <span class="nv">match: </span><span class="nx">str</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="nx">i</span><span class="p">).</span><span class="nx">match</span> <span class="nx">INTERPOLATION</span>
<span class="p">[</span><span class="nx">group</span><span class="p">,</span> <span class="nx">interp</span><span class="p">]</span><span class="o">:</span> <span class="nx">match</span>
<span class="nv">interp: </span><span class="s2">&quot;this.${ interp.substring(1) }&quot;</span> <span class="k">if</span> <span class="nx">starts</span> <span class="nx">interp</span><span class="p">,</span> <span class="s1">&#39;@&#39;</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s2">&quot;$quote${ str.substring(pi, i) }$quote&quot;</span><span class="p">]</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o">&lt;</span> <span class="nx">i</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">&#39;IDENTIFIER&#39;</span><span class="p">,</span> <span class="nx">interp</span><span class="p">]</span>
<span class="nv">i: </span><span class="o">+</span> <span class="nx">group</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="nv">pi: </span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
<span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nv">expr: </span><span class="nx">balanced_string</span> <span class="nx">str</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="nx">i</span><span class="p">),</span> <span class="p">[[</span><span class="s1">&#39;${&#39;</span><span class="p">,</span> <span class="s1">&#39;}&#39;</span><span class="p">]])</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s2">&quot;$quote${ str.substring(pi, i) }$quote&quot;</span><span class="p">]</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o">&lt;</span> <span class="nx">i</span>
<span class="nv">inner: </span><span class="nx">expr</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">inner</span><span class="p">.</span><span class="nx">length</span>
<span class="nv">nested: </span><span class="nx">lexer</span><span class="p">.</span><span class="nx">tokenize</span> <span class="s2">&quot;($inner)&quot;</span><span class="p">,</span> <span class="p">{</span><span class="nv">line: </span><span class="nx">@line</span><span class="p">}</span>
<span class="p">(</span><span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="s1">&#39;)&#39;</span><span class="p">)</span> <span class="k">for</span> <span class="nx">tok</span><span class="p">,</span> <span class="nx">idx</span> <span class="k">in</span> <span class="nx">nested</span> <span class="k">when</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;CALL_END&#39;</span>
<span class="nx">nested</span><span class="p">.</span><span class="nx">pop</span><span class="p">()</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">&#39;TOKENS&#39;</span><span class="p">,</span> <span class="nx">nested</span><span class="p">]</span>
<span class="k">else</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s2">&quot;$quote$quote&quot;</span><span class="p">]</span>
<span class="nv">i: </span><span class="o">+</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="nv">pi: </span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span>
<span class="nv">i: </span><span class="o">+</span> <span class="mi">1</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s2">&quot;$quote${ str.substring(pi, i) }$quote&quot;</span><span class="p">]</span> <span class="k">if</span> <span class="nx">pi</span> <span class="o">&lt;</span> <span class="nx">i</span> <span class="o">and</span> <span class="nx">pi</span> <span class="o">&lt;</span> <span class="nx">str</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="nx">tokens</span><span class="p">.</span><span class="nx">unshift</span> <span class="p">[</span><span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s1">&#39;&quot;&quot;&#39;</span><span class="p">]</span> <span class="nx">unless</span> <span class="nx">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">is</span> <span class="s1">&#39;STRING&#39;</span>
<span class="nv">interpolated: </span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">&gt;</span> <span class="mi">1</span>
<span class="nx">@token</span> <span class="s1">&#39;(&#39;</span><span class="p">,</span> <span class="s1">&#39;(&#39;</span> <span class="k">if</span> <span class="nx">interpolated</span>
<span class="k">for</span> <span class="nx">token</span><span class="p">,</span> <span class="nx">i</span> <span class="k">in</span> <span class="nx">tokens</span>
<span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">]</span><span class="o">:</span> <span class="nx">token</span>
<span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;TOKENS&#39;</span>
<span class="vi">@tokens: </span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">value</span>
<span class="k">else</span> <span class="k">if</span> <span class="nx">tag</span> <span class="o">is</span> <span class="s1">&#39;STRING&#39;</span> <span class="o">and</span> <span class="nx">escape_quotes</span>
<span class="nv">escaped: </span><span class="nx">value</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/&quot;/g</span><span class="p">,</span> <span class="s1">&#39;\\&quot;&#39;</span><span class="p">)</span>
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="s2">&quot;\&quot;$escaped\&quot;&quot;</span>
<span class="k">else</span>
<span class="nx">@token</span> <span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span>
<span class="nx">@token</span> <span class="s1">&#39;+&#39;</span><span class="p">,</span> <span class="s1">&#39;+&#39;</span> <span class="k">if</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span>
<span class="nx">@token</span> <span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span> <span class="k">if</span> <span class="nx">interpolated</span>
<span class="nx">tokens</span></pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-33">#</a> </div> <h2>Helpers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-34">#</a> </div> <p>Add a token to the results, taking note of the line number.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">token: </span><span class="p">(</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nx">@tokens</span><span class="p">.</span><span class="nx">push</span> <span class="p">[</span><span class="nx">tag</span><span class="p">,</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">@line</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-35">#</a> </div> <p>Peek at a tag in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">tag: </span><span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">new_tag</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">return</span> <span class="nx">unless</span> <span class="nv">tok: </span><span class="nx">@prev</span> <span class="nx">index</span>
<span class="k">return</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="nx">new_tag</span> <span class="k">if</span> <span class="nx">new_tag</span><span class="o">?</span>
<span class="nx">tok</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-36">#</a> </div> <p>Peek at a value in the current token stream.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">value: </span><span class="p">(</span><span class="nx">index</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">return</span> <span class="nx">unless</span> <span class="nv">tok: </span><span class="nx">@prev</span> <span class="nx">index</span>
<span class="k">return</span> <span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">:</span> <span class="nx">val</span> <span class="k">if</span> <span class="nx">val</span><span class="o">?</span>
<span class="nx">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-37">#</a> </div> <p>Peek at a previous token, entire.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">prev: </span><span class="p">(</span><span class="nx">index</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="nx">@tokens</span><span class="p">[</span><span class="nx">@tokens</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="p">(</span><span class="nx">index</span> <span class="o">or</span> <span class="mi">1</span><span class="p">)]</span></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-38">#</a> </div> <p>Attempt to match a string against the current chunk, returning the indexed
match if successful, and <code>false</code> otherwise.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">match: </span><span class="p">(</span><span class="nx">regex</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">-&gt;</span>
<span class="k">return</span> <span class="kc">false</span> <span class="nx">unless</span> <span class="nv">m: </span><span class="nx">@chunk</span><span class="p">.</span><span class="nx">match</span> <span class="nx">regex</span>
<span class="k">if</span> <span class="nx">m</span> <span class="k">then</span> <span class="nx">m</span><span class="p">[</span><span class="nx">index</span><span class="p">]</span> <span class="k">else</span> <span class="kc">false</span></pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-39">#</a> </div> <p>Are we in the midst of an unfinished expression?</p> </td> <td class="code"> <div class="highlight"><pre> <span class="nv">unfinished: </span><span class="o">-&gt;</span>
<span class="nv">prev: </span><span class="nx">@prev</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="nx">@value</span><span class="p">()</span> <span class="o">and</span> <span class="nx">@value</span><span class="p">().</span><span class="nx">match</span> <span class="o">and</span> <span class="nx">@value</span><span class="p">().</span><span class="nx">match</span><span class="p">(</span><span class="nx">NO_NEWLINE</span><span class="p">)</span> <span class="o">and</span>
<span class="nx">prev</span> <span class="o">and</span> <span class="p">(</span><span class="nx">prev</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">isnt</span> <span class="s1">&#39;.&#39;</span><span class="p">)</span> <span class="o">and</span> <span class="o">not</span> <span class="nx">@value</span><span class="p">().</span><span class="nx">match</span><span class="p">(</span><span class="nx">CODE</span><span class="p">)</span></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-40">#</a> </div> <h2>Lexer Properties</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-41">#</a> </div> <p>There are no exensions to the core lexer by default.</p> </td> <td class="code"> <div class="highlight"><pre> <span class="vi">@extensions: </span><span class="p">[]</span></pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-42">#</a> </div> <h2>Constants</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-43">#</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_KEYWORDS: </span><span class="p">[</span>
<span class="s2">&quot;if&quot;</span><span class="p">,</span> <span class="s2">&quot;else&quot;</span><span class="p">,</span>
<span class="s2">&quot;true&quot;</span><span class="p">,</span> <span class="s2">&quot;false&quot;</span><span class="p">,</span>
<span class="s2">&quot;new&quot;</span><span class="p">,</span> <span class="s2">&quot;return&quot;</span><span class="p">,</span>
<span class="s2">&quot;try&quot;</span><span class="p">,</span> <span class="s2">&quot;catch&quot;</span><span class="p">,</span> <span class="s2">&quot;finally&quot;</span><span class="p">,</span> <span class="s2">&quot;throw&quot;</span><span class="p">,</span>
<span class="s2">&quot;break&quot;</span><span class="p">,</span> <span class="s2">&quot;continue&quot;</span><span class="p">,</span>
<span class="s2">&quot;for&quot;</span><span class="p">,</span> <span class="s2">&quot;in&quot;</span><span class="p">,</span> <span class="s2">&quot;while&quot;</span><span class="p">,</span>
<span class="s2">&quot;delete&quot;</span><span class="p">,</span> <span class="s2">&quot;instanceof&quot;</span><span class="p">,</span> <span class="s2">&quot;typeof&quot;</span><span class="p">,</span>
<span class="s2">&quot;switch&quot;</span><span class="p">,</span> <span class="s2">&quot;super&quot;</span><span class="p">,</span> <span class="s2">&quot;extends&quot;</span><span class="p">,</span> <span class="s2">&quot;class&quot;</span><span class="p">,</span>
<span class="s2">&quot;this&quot;</span><span class="p">,</span> <span class="s2">&quot;null&quot;</span>
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-44"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-44">#</a> </div> <p>CoffeeScript-only keywords, which we're more relaxed about allowing. They can't
be used standalone, but you can reference them as an attached property.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">COFFEE_ALIASES: </span> <span class="p">[</span><span class="s2">&quot;and&quot;</span><span class="p">,</span> <span class="s2">&quot;or&quot;</span><span class="p">,</span> <span class="s2">&quot;is&quot;</span><span class="p">,</span> <span class="s2">&quot;isnt&quot;</span><span class="p">,</span> <span class="s2">&quot;not&quot;</span><span class="p">]</span>
<span class="nv">COFFEE_KEYWORDS: </span><span class="nx">COFFEE_ALIASES</span><span class="p">.</span><span class="nx">concat</span> <span class="p">[</span>
<span class="s2">&quot;then&quot;</span><span class="p">,</span> <span class="s2">&quot;unless&quot;</span><span class="p">,</span> <span class="s2">&quot;until&quot;</span><span class="p">,</span>
<span class="s2">&quot;yes&quot;</span><span class="p">,</span> <span class="s2">&quot;no&quot;</span><span class="p">,</span> <span class="s2">&quot;on&quot;</span><span class="p">,</span> <span class="s2">&quot;off&quot;</span><span class="p">,</span>
<span class="s2">&quot;of&quot;</span><span class="p">,</span> <span class="s2">&quot;by&quot;</span><span class="p">,</span> <span class="s2">&quot;where&quot;</span><span class="p">,</span> <span class="s2">&quot;when&quot;</span>
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-45"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-45">#</a> </div> <p>The combined list of keywords is the superset that gets passed verbatim to
the parser.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">KEYWORDS: </span><span class="nx">JS_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">COFFEE_KEYWORDS</span></pre></div> </td> </tr> <tr id="section-46"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-46">#</a> </div> <p>The list of keywords that are reserved by JavaScript, but not used, or are
used by CoffeeScript internally. We throw an error when these are encountered,
to avoid having a JavaScript error at runtime.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">RESERVED: </span><span class="p">[</span>
<span class="s2">&quot;case&quot;</span><span class="p">,</span> <span class="s2">&quot;default&quot;</span><span class="p">,</span> <span class="s2">&quot;do&quot;</span><span class="p">,</span> <span class="s2">&quot;function&quot;</span><span class="p">,</span> <span class="s2">&quot;var&quot;</span><span class="p">,</span> <span class="s2">&quot;void&quot;</span><span class="p">,</span> <span class="s2">&quot;with&quot;</span>
<span class="s2">&quot;const&quot;</span><span class="p">,</span> <span class="s2">&quot;let&quot;</span><span class="p">,</span> <span class="s2">&quot;enum&quot;</span><span class="p">,</span> <span class="s2">&quot;export&quot;</span><span class="p">,</span> <span class="s2">&quot;import&quot;</span><span class="p">,</span> <span class="s2">&quot;native&quot;</span>
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-47"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-47">#</a> </div> <p>The superset of both JavaScript keywords and reserved words, none of which may
be used as identifiers or properties.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_FORBIDDEN: </span><span class="nx">JS_KEYWORDS</span><span class="p">.</span><span class="nx">concat</span> <span class="nx">RESERVED</span></pre></div> </td> </tr> <tr id="section-48"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-48">#</a> </div> <p>Token matching regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">IDENTIFIER : </span><span class="sr">/^([a-zA-Z\$_](\w|\$)*)/</span>
<span class="nv">NUMBER : </span><span class="sr">/^(\b((0(x|X)[0-9a-fA-F]+)|([0-9]+(\.[0-9]+)?(e[+\-]?[0-9]+)?)))\b/i</span>
<span class="nv">HEREDOC : </span><span class="sr">/^(&quot;{6}|&#39;{6}|&quot;{3}\n?([\s\S]*?)\n?([ \t]*)&quot;{3}|&#39;{3}\n?([\s\S]*?)\n?([ \t]*)&#39;{3})/</span>
<span class="nv">INTERPOLATION : </span><span class="sr">/^\$([a-zA-Z_@]\w*(\.\w+)*)/</span>
<span class="nv">OPERATOR : </span><span class="sr">/^([+\*&amp;|\/\-%=&lt;&gt;:!?]+)([ \t]*)/</span>
<span class="nv">WHITESPACE : </span><span class="sr">/^([ \t]+)/</span>
<span class="nv">COMMENT : </span><span class="sr">/^((\n?[ \t]*)?#{3}(?!#)\n*([\s\S]*?)\n*([ \t]*)#{3}|((\n?[ \t]*)?#[^\n]*)+)/</span>
<span class="nv">CODE : </span><span class="sr">/^((-|=)&gt;)/</span>
<span class="nv">MULTI_DENT : </span><span class="sr">/^((\n([ \t]*))+)(\.)?/</span>
<span class="nv">LAST_DENTS : </span><span class="sr">/\n([ \t]*)/g</span>
<span class="nv">LAST_DENT : </span><span class="sr">/\n([ \t]*)/</span>
<span class="nv">ASSIGNMENT : </span><span class="sr">/^(:|=)$/</span></pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-49">#</a> </div> <p>Regex-matching-regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">REGEX_START : </span><span class="sr">/^\/[^\/ ]/</span>
<span class="nv">REGEX_INTERPOLATION: </span><span class="sr">/([^\\]\$[a-zA-Z_@]|[^\\]\$\{.*[^\\]\})/</span>
<span class="nv">REGEX_END : </span><span class="sr">/^(([imgy]{1,4})\b|\W)/</span>
<span class="nv">REGEX_ESCAPE : </span><span class="sr">/\\[^\$]/g</span></pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-50">#</a> </div> <p>Token cleaning regexes.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">JS_CLEANER : </span><span class="sr">/(^`|`$)/g</span>
<span class="nv">MULTILINER : </span><span class="sr">/\n/g</span>
<span class="nv">STRING_NEWLINES : </span><span class="sr">/\n[ \t]*/g</span>
<span class="nv">COMMENT_CLEANER : </span><span class="sr">/(^[ \t]*#|\n[ \t]*$)/mg</span>
<span class="nv">NO_NEWLINE : </span><span class="sr">/^([+\*&amp;|\/\-%=&lt;&gt;:!.\\][&lt;&gt;=&amp;|]*|and|or|is|isnt|not|delete|typeof|instanceof)$/</span>
<span class="nv">HEREDOC_INDENT : </span><span class="sr">/(\n+([ \t]*)|^([ \t]+))/g</span></pre></div> </td> </tr> <tr id="section-51"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-51">#</a> </div> <p>Tokens which a regular expression will never immediately follow, but which
a division operator might.</p>
<p>See: http://www.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions</p>
<p>Our list is shorter, due to sans-parentheses method calls.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">NOT_REGEX: </span><span class="p">[</span>
<span class="s1">&#39;NUMBER&#39;</span><span class="p">,</span> <span class="s1">&#39;REGEX&#39;</span><span class="p">,</span> <span class="s1">&#39;++&#39;</span><span class="p">,</span> <span class="s1">&#39;--&#39;</span><span class="p">,</span> <span class="s1">&#39;FALSE&#39;</span><span class="p">,</span> <span class="s1">&#39;NULL&#39;</span><span class="p">,</span> <span class="s1">&#39;TRUE&#39;</span><span class="p">,</span> <span class="s1">&#39;]&#39;</span>
<span class="p">]</span></pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-52">#</a> </div> <p>Tokens which could legitimately be invoked or indexed. A opening
parentheses or bracket following these tokens will be recorded as the start
of a function invocation or indexing operation.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CALLABLE: </span><span class="p">[</span><span class="s1">&#39;IDENTIFIER&#39;</span><span class="p">,</span> <span class="s1">&#39;SUPER&#39;</span><span class="p">,</span> <span class="s1">&#39;)&#39;</span><span class="p">,</span> <span class="s1">&#39;]&#39;</span><span class="p">,</span> <span class="s1">&#39;}&#39;</span><span class="p">,</span> <span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s1">&#39;@&#39;</span><span class="p">,</span> <span class="s1">&#39;THIS&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-53">#</a> </div> <p>Tokens that indicate an access -- keywords immediately following will be
treated as identifiers.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">ACCESSORS: </span><span class="p">[</span><span class="s1">&#39;PROPERTY_ACCESS&#39;</span><span class="p">,</span> <span class="s1">&#39;PROTOTYPE_ACCESS&#39;</span><span class="p">,</span> <span class="s1">&#39;SOAK_ACCESS&#39;</span><span class="p">,</span> <span class="s1">&#39;@&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-54"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-54">#</a> </div> <p>Tokens that, when immediately preceding a <code>WHEN</code>, indicate that the <code>WHEN</code>
occurs at the start of a line. We disambiguate these from trailing whens to
avoid an ambiguity in the grammar.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">LINE_BREAK: </span><span class="p">[</span><span class="s1">&#39;INDENT&#39;</span><span class="p">,</span> <span class="s1">&#39;OUTDENT&#39;</span><span class="p">,</span> <span class="s1">&#39;TERMINATOR&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-55"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-55">#</a> </div> <p>Half-assignments...</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">HALF_ASSIGNMENTS: </span><span class="p">[</span><span class="s1">&#39;-&#39;</span><span class="p">,</span> <span class="s1">&#39;+&#39;</span><span class="p">,</span> <span class="s1">&#39;/&#39;</span><span class="p">,</span> <span class="s1">&#39;*&#39;</span><span class="p">,</span> <span class="s1">&#39;%&#39;</span><span class="p">,</span> <span class="s1">&#39;||&#39;</span><span class="p">,</span> <span class="s1">&#39;&amp;&amp;&#39;</span><span class="p">,</span> <span class="s1">&#39;?&#39;</span><span class="p">]</span></pre></div> </td> </tr> <tr id="section-56"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-56">#</a> </div> <p>Conversions from CoffeeScript operators into JavaScript ones.</p> </td> <td class="code"> <div class="highlight"><pre><span class="nv">CONVERSIONS: </span><span class="p">{</span>
<span class="s1">&#39;and&#39;</span><span class="o">:</span> <span class="s1">&#39;&amp;&amp;&#39;</span>
<span class="s1">&#39;or&#39;</span><span class="o">:</span> <span class="s1">&#39;||&#39;</span>
<span class="s1">&#39;is&#39;</span><span class="o">:</span> <span class="s1">&#39;==&#39;</span>
<span class="s1">&#39;isnt&#39;</span><span class="o">:</span> <span class="s1">&#39;!=&#39;</span>
<span class="s1">&#39;not&#39;</span><span class="o">:</span> <span class="s1">&#39;!&#39;</span>
<span class="p">}</span>
</pre></div> </td> </tr> </tbody> </table> </div> </body> </html>
View Git Blame Copy Permalink