Blog Post by Murat Yaşar
Posted on Tuesday, December 29, 2009 8:19:56 AM and it has been read 5298 times since then.
Syntax not understood error for robots.txt file in Google Webmaster Tools
I came across a strange problem while I was reviewing webmaster tools.
I signed in with my google account and started reviewing my settings.
I chose the website I validated before and clicked Site configuration>Crawler access respectively.
In the Test robots.txt tab, I saw interesting errors that was saying:
Line1: ?User-agent: * Syntax not understood
Line2: Sitemap: http://www.yasarmurat.com/SiteMap.aspx Valid Sitemap reference detected
Line3: No user-agent specified
I checked my robots.txt file again and again. There was no question mark in Line1 of the robots.txt file which google claims there is.
I create a new robots.txt file by using notepad in Vista Home Premium operating system and uploaded this new file into my production server. I waited for about a day and checked again. The same error was still there.
I googled for this and found a link which was talking about exactly the same issue I was having. After creating my new robots.txt file by using ANSI encoding type and checking it from this link problem solved. Well at least seems like that. I have been checking google and it is still showing me the same error. I think it is going to take some time to reflect new changes.
This is the link I found the solution for this specific problem:
http://www.google.com/support/forum/p/Webmasters/thread?tid=28d012e70d5fcdc8&hl=en
This is the link of the web site which gave me the opportunity to check my robots.txt file:
http://www.asymptoticdesign.co.uk/cgi-bin/check-url.pl
Have a great day.
(In order to use this feature, you have to register.)
Comments (2) Leave a Comment
-
-
Murat Yasar | Reply
Wednesday, December 30, 2009 8:58:01 AM
Here are another links to check your robots.txt file
http://www.rexswain.com/httpview.html
http://tool.motoricerca.info/robots-checker.phtml
Murat Yasar | Reply
Tuesday, December 29, 2009 8:35:30 AMI have just checked webmaster tools again and saw the problem has gone. There is no syntax not understood error. Everything seems ok for now.