# Tells 'robots' (eg Google search crawler, Panoptic crawler, etc) where they should not index # # User-agent: can also specify by name; "*" is for everyone # Disallow: disallow if this matches first part of requested path (blank allows all) # User-agent: * Disallow: