Blog Post by Murat Yaşar

Posted on Tuesday, December 29, 2009 8:19:56 AM and it has been read 2468 times since then.

Syntax not understood error for robots.txt file in Google Webmaster Tools

I came across a strange problem while I was reviewing webmaster tools.
I signed in with my google account and started reviewing my settings.
I chose the website I validated before and clicked Site configuration>Crawler access respectively.

In the Test robots.txt tab, I saw interesting errors that was saying:

Line1: ?User-agent: *    Syntax not understood
Line2: Sitemap:    Valid Sitemap reference detected
Line3: No user-agent specified


I checked my robots.txt file again and again. There was no question mark in Line1 of the robots.txt file which google claims there is.

I create a new robots.txt file by using notepad in Vista Home Premium operating system and uploaded this new file into my production server. I waited for about a day and checked again. The same error was still there.

I googled for this and found a link which was talking about exactly the same issue I was having. After creating my new robots.txt file by using ANSI encoding type and checking it from this link problem solved. Well at least seems like that. I have been checking google and it is still showing me the same error. I think it is going to take some time to reflect new changes.

This is the link I found the solution for this specific problem:

This is the link of the web site which gave me the opportunity to check my robots.txt file:

Have a great day.

(In order to use this feature, you have to register.)