(Jan 15, 2017 09:48 PM)0NUR0UX Wrote:  So far, I can scrap data from the IMDb website.

Code

def scrapData(url):
    import urllib.request
    from bs4 import BeautifulSoup
    req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    open = urllib.request.urlopen(req)
    readHtml = open.read()

    soup = BeautifulSoup(readHtml, "html.parser")

    tags = soup.find_all("tbody")
    for movies in tags:
        tdHeader = movies.findAll("td",attrs = {'class':"titleColumn"})
        for link in tdHeader:
            titleLink = link.find_all("a")
            for title in titleLink:

                theTitle = str(title)
                start = theTitle.index(">")
                end = theTitle[start:].index("<")
                print(theTitle[start:(start+end)])

scrapData("http://www."+"imdb.com/chart/top?ref_=nv_mv_250_6")

(Jan 16, 2017 05:29 PM)Goodies Wrote:  If that's the main purpose of the program, there's no need to import the module within a function. You're not doing optional paths with different libraries, so you should just move the imports to the top.

Also, you should follow the SRP. Having a function that does too many things (gets data and parses it) is not ideal. I recommend you separate them out into two functions and perhaps have one that combines them.

This will let you, for example, easily change out parsers.

PHP Code


<?php 
def parse(url, parser):
    try:
        req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
        open = urllib.request.urlopen(req)
        response = open.read()
    except Exception as e:
        print(e)
    else:
        return parser(response)

def imdb_parser(html):
    # do the parsing here

def other_parser(html):
    # parse some other site

data = parse("http://imdb.com/whatever", imdb_parser)
print(data)