- Visual Surf Engine - Visual Surf EngineThe IconSurf robot is a specialized internet program which searches for website icons. You may have found this page after noticing that the IconSurf robot visited your website. The purpose of IconSurf is to explore the variety of icon artwork and enable the unique and fun method of web surfing found at iconsurf.com.
Below you will find answers to the frequently asked questions about web robots and website icon files. You will also find instructions on how to prevent IconSurf or other robots from visiting your site or indexing your icons.
Website icons are the small images you may have seen to the left of the address bar or in the bookmarks section of your web browser for certain websites. These icons are usually named favicon.ico, but may have an entirely different name.
IconSurf searches all over the internet for website icons, so if you have a website you are likely to be visited by the IconSurf robot at least once. For most of the global tlds (biz, com, edu, info, int, museum, name, net and org), IconSurf works by scanning the authoritative lists of domain names. For the country-code top level domains, IconSurf performs a brute-force alphabetic or dictionary search of domain names or uses an authoritative list where available. IconSurf also has an icon submission page where anyone can request that a website or webpage be checked for an icon. Once IconSurf finds an icon at a given location, it will check back regularly to make sure it is still available.
Like most well-behaved web robots, IconSurf adheres to the Robot Exclusion Standard. By placing a properly formatted file called robots.txt in the root directory of your website, you can instruct individual or all web robots not to download individual pages, sections, or all parts of your website. IconSurf looks in the robots.txt file for a user agent identifier of "iconsurf" (case insensitive) or failing that looks for the default "*" user agent. IconSurf will not download any pages which are disallowed for it.
To forbid only IconSurf from downloading any pages on your website (other than robots.txt), place the following lines in your robots.txt file:
User-agent: IconSurf Disallow: /
To forbid only IconSurf from downloading any pages starting with the string sample or from a directory called sample, place the following lines in your robots.txt file:
User-agent: IconSurf Disallow: /sample
To forbid all robots from downloading any pages on your website (other than robots.txt), place the following lines in your robots.txt file:
User-agent: * Disallow: /
Note that the following robots.txt file will NOT stop IconSurf from downloading arbitrary pages on your website, but will only prevent IconSurf from downloading the single file called favicon.ico:
User-agent: IconSurf Disallow: /favicon.ico
The robots.txt file must be placed in the root directory of your website. For example, if your website is called "mysite.net", your robots.txt file must be available at the location "http://mysite.net/robots.txt". The robots.txt file should be plain text or it will not be understood.
Most web robots do not provide this feature, but I will be happy to honor your request if you send me an email at pa@iconsurf.com stating the name of your websites. I may require proof of your authority over the website, so please place a file called "iconsurf.txt" in the root directory of your website containing the text "remove please" that I can check after reading your email. Once I process your email, your site should never be visited again by the IconSurf robot and your icon link (if any) will be permanently removed from our main site.
I know of no cases where this has happened, but programming is an art rather than a science. Maybe there is a problem with my code or your robots file. Drop me a line at pa@iconsurf.com and I will fix the problem with my robot or help you fix the problem with your robots.txt file.
IconSurf will usually check a given website at most once every few months for the presence of an icon. In this case, you will see the string "finder" in the IconSurf robot user agent string. Some websites are quite popular and are checked manually for icons by visitors to IconSurf on our icon submission page. In this case, you will see the string "submitter" in the IconSurf robot user agent string. If an icon was previously found by IconSurf at your website, your icon will be re-checked roughly once per week. In this case, you will see the string "monitor" in the IconSurf robot user agent string.
If your icon is being monitored but is no longer present or is protected by a robots.txt file, be assured that IconSurf will eventually give up monitoring. IconSurf will remove your icon link from the database when two times the number of icon monitoring failures is greater than the number of successes.
If your site was visited by the IconSurf robot but your icon was not found, there are many possible reasons for the failure. Try to manually submit your icon at the icon submission page, and if that fails drop me a line after reading the diagnosis on the submit page and attempting to fix any problems mentioned there. Please note that your icon will be automatically removed from the the main IconSurf page within 24 hours of the first unsuccessful monitoring download and will not be re-listed until the next successful monitoring download.
If the file favicon.ico is not found, IconSurf will check the default webpage for an icon link tag. An icon link tag is a way to indicate a non-default icon location from within a webpage. Below is a listing of a simple html file available at http://pupbun.com/test/index.html which uses icon link tags to specify an icon name other than favicon.ico. In reality, a webpage should have a maximum of one icon link tag.
<html> <head> <link rel="shortcut icon" href="http://pupbun.com/icon.ico" type="image/x-icon" /> <link rel="shortcut icon" href="/test/icon.ico" type="image/x-icon" /> <link rel="shortcut icon" href="icon.ico" type="image/x-icon" /> <link rel="shortcut icon" href="./icon.ico" type="image/x-icon" /> <link rel="shortcut icon" href="subdir/icon.ico" type="image/x-icon" /> <link rel="shortcut icon" href="../icon.ico" type="image/x-icon" /> </head> <body> <h1>Sample icon link tag page</h1>. </body> </html>
In the above example, I have included six types of icon link tags that IconSurf will understand. The first uses an absolute address, which IconSurf will only follow if the domain is the same domain where the webpage was downloaded. The second type is interpreted as a filename relative to the server root in the test directory. The third and fourth types specify an icon in the test directory (same as the webpage). The fifth type references an icon located at /test/subdir/icon.ico. The sixth type references an icon one directory up from the test directory, or the server root.
Most websites come in two forms - the form with and without a "www." prefix. But some websites have only one or the other. IconSurf checks both varieties so that it will not miss any icons. IconSurf will skip checking the www version if an icon, a robots.txt file or a webpage is found on the non-www website.
There are many places that will design icons for a price, and there are many websites with do-it-yourself instructions. Do a web search for "favicon" to see the possibilities or search for domain names containing the string icon at IconSurf and follow the icon links found there. But PLEASE do not copy icons found at iconsurf.com, as most icons listed are copyrighted and not in the public domain.
If you would like more information, or would like to ask a question not covered in this FAQ, please email me at pa@iconsurf.com.
Site design and content plus database contents © 2004 by Peter Andrews of SkySci LTD.
Make sure to see the links at the top of the page to learn more about favicon.ico files and other aspects of this website, including how you can submit your own icons.
This page was generated at Sun Oct 29 01:50:59 2006 EST.
