Crawler ID: ID211
Name: mbneutrino
Creator: mb
Start: Search for neutrino on google

Crawling Method: 1
Stopping Method: 1
(Amount of links to request: 400)

Get attributes and content of: ABBR 
(file all_tags_attributes.dat and all_tags_content.dat)

Leave trace in server request log: 'nutrino visit'

Request same page more than once

Look for and store url index number of status codes: 302 (Found),
307 (Temporary Redirect),404 (Not Found),410 (Gone) 
(file total_code_indices.dat)

Store these headers for every request: Client-Date^Client-Peer^
Title^Age^ETag^Location^Expires^Last-Modified (file headers_arrays.dat)

Get attributes: bgcolor,color,version (file all_attributes.dat)
Get comments. (file all_comments.dat)

Allow cookies to be set and retrived.