kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

allow crawling of commit page but not patch/diffs

Browse source 
Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
This commit is contained in:
Ben Bodenmiller 2015-10-04 23:39:24 -07:00
parent 0de7c83a78
commit abf18abb52
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
public/robots.txt
View file

@ -38,7 +38,8 @@ Disallow: /*/*/edit
Disallow: /*/*/raw Disallow: /*/*/raw
Disallow: /*/*/blame Disallow: /*/*/blame
Disallow: /*/*/commits/*/* Disallow: /*/*/commits/*/*
Disallow: /*/*/commit Disallow: /*/*/commit/*.patch
Disallow: /*/*/commit/*.diff
Disallow: /*/*/compare Disallow: /*/*/compare
Disallow: /*/*/branches/new Disallow: /*/*/branches/new
Disallow: /*/*/tags/new Disallow: /*/*/tags/new