Interessado em saber como funciona o Google, Bing ou Yahoo? Quer saber o que é necessário para rastrear a web e como seria um rastreador da web simples? Em menos de 50 linhas de código Python (versão 3), aqui está um rastreador da web simples! (A fonte completa com comentários está no final deste artigo).

Vamos examinar o código com mais detalhes!

O código a seguir deve ser totalmente funcional para Python 3.x. Ele foi escrito e testado com Python 3.2.2 em setembro de 2011. Vá em frente e copie + cole em seu IDE Python e execute-o ou modifique-o .

de html.parser import HTMLParser

de urllib.request import urlopen

de urllib import parse

Vamos criar uma classe chamada LinkParser que herda alguns

métodos do HTMLParser, que é o motivo pelo qual é passado para a definição

classe LinkParser (HTMLParser):

# This is a function that HTMLParser normally has
# but we are adding some functionality to it
def handle_starttag(self, tag, attrs):
    # We are looking for the begining of a link. Links normally look
    # like <a href="www.someurl.com"></a>
    if tag  'a':
        for (key, value) in attrs:
            if key  'href':
                # We are grabbing the new URL. We are also adding the
                # base URL to it. For example:
                # www.fixithere.net is the base and
                # somepage.html is the new URL (a relative URL)
                #
                # We combine a relative URL with the base URL to create
                # an absolute URL like:
                # www.fixithere.net/sky-customer-service/
                newUrl = parse.urljoin(self.baseUrl, value)
                # And add it to our colection of links:
                self.links = self.links + [newUrl]

# This is a new function that we are creating to get links
# that our spider() function will call
def getLinks(self, url):
    self.links = []
    # Remember the base URL which will be important when creating
    # absolute URLs
    self.baseUrl = url
    # Use the urlopen function from the standard Python 3 library
    response = urlopen(url)
    # Make sure that we are looking at HTML and not other things that
    # are floating around on the internet (such as
    # JavaScript files, CSS, or .PDFs for example)
    if response.getheader('Content-Type')=='text/html':
        htmlBytes = response.read()
        # Note that feed() handles Strings well, but not bytes
        # (A change from Python 2.x to Python 3.x)
        htmlString = htmlBytes.decode("utf-8")
        self.feed(htmlString)
        return htmlString, self.links
    else:
        return "",[]

# And finally here is our spider. It takes in an URL, a word to find,
# and the number of pages to search through before giving up

def spider(url, word, maxPages):  
pagesToVisit = [url]
numberVisited = 0
foundWord = False
# The main loop. Create a LinkParser and get all the links on the page.
# Also search the page for the word or string
# In our getLinks function we return the web page
# (this is useful for searching for the word)
# and we return a set of links from that web page
# (this is useful for where to go next)
while numberVisited < maxPages and pagesToVisit != [] and not foundWord:
    numberVisited = numberVisited +1
    # Start from the beginning of our collection of pages to visit:
    url = pagesToVisit[0]
    pagesToVisit = pagesToVisit[1:]
    try:
        print(numberVisited, "Visiting:", url)
        parser = LinkParser()
        data, links = parser.getLinks(url)
        if data.find(word)>-1:
            foundWord = True
            # Add the pages that we visited to the end of our collection
            # of pages to visit:
            pagesToVisit = pagesToVisit + links
            print(" **Success!**")
    except:
        print(" **Failed!**")
if foundWord:
    print("The word", word, "was found at", url)
else:
    print("Word never found")

Magia!