How Google Crawls (googlebot) a website - Click here to read
Google Webmaster Guide - Click here to read (don't take this too seriously but you'll know what they consider spam)
Yahoo's Spider (slurp) - Click here to read
Ask.com Spider info - Click here to read (owner of Ask also owns; Expedia.com, Ticketmaster.com and Match.com - this will bear fruit in the future)
Microsofts Live info - Click here to read
Search Engine Tools and Resources
Word Tracker Free keyword tool - Use this tool (this is what you can use after you've completed the real research - for that task purchased a week of their service with the above link - the Free service simply will not give you what you need)
Microsoft Keyword Forecast - Use this tool
See how competitive your site already is in the search engines - Use this tool
GoRank’s Related Keyword Lookup tool - Use this tool
Google Sets – Automatically create groups of sets - Use this tool
Google Keyword suggestion tool - Use this tool
Yahoo - Overture Keyword suggestion tool - Use this tool
l3xicon.com 'related words' - Use this tool
Givemebackmygoogle.com - Use this tool - (will show you the results of Google without the paid adwords and affiliate links - returns only organic results)
Microsoft search funnel - Use this tool (will show you the search phrases that surround targeted keywords in regards to the actual search cycle)
Microsoft Adlab - Click on the links below to use these tools
http://adlab.msn.com/contextSim/default.aspx - discover groups of related keywords
http://adlab.msn.com/keyMut/default.aspx - detect frequently used alternative spellings
http://adlab.msn.com/Forecast/ - predict seasonality trends in search volume
http://adlab.msn.com/DPUI/DPUI.aspx - collect demographic information related to a particular search phrase
Dependency-based Word Similarity - Use this tool - allows you to find dependency based word relatedness. Very useful for determining the appropriateness of contextual synonymy.
Search Engine Spider Simulators - These will show you what a seach engine spider captures from your website.
http://www.spidersimulator.net/
Competitive Analysis Tools
Yahoo Site explorer – see all pages indexed by Yahoo - Use this tool
Spyfu.com – Competitive Intelligence (enter a domain name or key phrase) - Use this tool
Google Trends - Use this tool
Check the History of a Domain name - Use this tool
The Wayback Machine (tracks a websites life) - Use this tool
Google Linking Commands
Google Search Commands to view how Google views your link infomation (per Google);
1. Type 'link:domainname.com' in the search field to see who is linking to a site.
2. Type 'related:domainname.com' in the search field to see a sites link neighborhood.
Purchase a 1 Week Subscription to the Wordtracker service ($30.00) - Click here to Subscribe