Hi can anyone give me more information on robots.txt file. I dont currently use it and i want to know exactly why and if i should start using it. Also i hear people telling how they know a bot has crawld there site how can i tell if mine has been crawld and how often.