From: Art Cancro <ajc@citadel.org>
Date: Thu, 20 Jan 2011 03:37:30 +0000 (-0500)
Subject: Disallow certain pages in robots.txt which would mislead crawlers.
X-Git-Tag: v8.11~896
X-Git-Url: https://code.citadel.org/?a=commitdiff_plain;h=ef02cfe2893f823a4d37eb5c90bcd1e014e7041c;p=citadel.git

Disallow certain pages in robots.txt which would mislead crawlers.
/printmsg
/msgheaders
/groupdav
/do_template
/static
---

diff --git a/webcit/static.c b/webcit/static.c
index 971def890..71ca5641a 100644
--- a/webcit/static.c
+++ b/webcit/static.c
@@ -283,7 +283,11 @@ void robots_txt(void) {
 	begin_burst();
 
 	wc_printf("User-agent: *\r\n"
-		"Disallow:\r\n"
+		"Disallow: /printmsg\r\n"
+		"Disallow: /msgheaders\r\n"
+		"Disallow: /groupdav\r\n"
+		"Disallow: /do_template\r\n"
+		"Disallow: /static\r\n"
 		"Sitemap: %s/sitemap.xml\r\n"
 		"\r\n"
 		,