Commit graph

3 commits

Author SHA1 Message Date
Ben Bodenmiller
abf18abb52 allow crawling of commit page but not patch/diffs
Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
2015-10-04 23:39:24 -07:00
Ben Bodenmiller
595a93ee2c disallow irrelevant pages by default in robots
Update default robots.txt rules to disallow irrelevant pages that search
engines should not care about. This will still allow important pages
like the files, commit details, merge requests, issues, comments, etc.
to be crawled.
2015-08-17 23:11:16 -07:00
gitlabhq
9ba1224867 init commit 2011-10-09 00:36:38 +03:00