Blog Post by Murat Yaşar


Posted on Tuesday, December 29, 2009 8:19:56 AM and it has been read 2468 times since then.


Syntax not understood error for robots.txt file in Google Webmaster Tools

I came across a strange problem while I was reviewing webmaster tools.
I signed in with my google account and started reviewing my settings.
I chose the website I validated before and clicked Site configuration>Crawler access respectively.

In the Test robots.txt tab, I saw interesting errors that was saying:

Line1: ?User-agent: *    Syntax not understood
Line2: Sitemap: http://www.yasarmurat.com/SiteMap.aspx    Valid Sitemap reference detected
Line3: No user-agent specified

/BlogEntryImages/19/robotstxterror.jpg

I checked my robots.txt file again and again. There was no question mark in Line1 of the robots.txt file which google claims there is.

I create a new robots.txt file by using notepad in Vista Home Premium operating system and uploaded this new file into my production server. I waited for about a day and checked again. The same error was still there.

I googled for this and found a link which was talking about exactly the same issue I was having. After creating my new robots.txt file by using ANSI encoding type and checking it from this link problem solved. Well at least seems like that. I have been checking google and it is still showing me the same error. I think it is going to take some time to reflect new changes.

This is the link I found the solution for this specific problem:
http://www.google.com/support/forum/p/Webmasters/thread?tid=28d012e70d5fcdc8&hl=en

This is the link of the web site which gave me the opportunity to check my robots.txt file:
http://www.asymptoticdesign.co.uk/cgi-bin/check-url.pl


Have a great day.


(In order to use this feature, you have to register.)