twbs--bootstrap/site/layouts/robots.txt

# www.robotstxt.org

{{- $isProduction := eq (getenv "HUGO_ENV") "production" -}}
{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}
{{- $allowCrawling := and (not $isNetlify) $isProduction -}}

{{ if $allowCrawling }}
# Allow crawling of all content
{{- end }}
User-agent: *
Disallow:{{ if not $allowCrawling }} /{{ end }}
Sitemap: {{ .Site.BaseURL }}/sitemap.xml
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 16:33:28 +00:00			`# www.robotstxt.org`

robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 14:26:25 +00:00			`{{- $isProduction := eq (getenv "HUGO_ENV") "production" -}}`
			`{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}`
			`{{- $allowCrawling := and (not $isNetlify) $isProduction -}}`

			`{{ if $allowCrawling }}`
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 16:33:28 +00:00			`# Allow crawling of all content`
			`{{- end }}`
			`User-agent: *`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 14:26:25 +00:00			`Disallow:{{ if not $allowCrawling }} /{{ end }}`
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 16:33:28 +00:00			`Sitemap: {{ .Site.BaseURL }}/sitemap.xml`