Disallow certain pages in robots.txt which would mislead crawlers.