No more robots.txt, we now welcome spiders