Python Downloading source code automatically with python

[Python] Downloading source code automatically with python - Printable Version

+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Python] Downloading source code automatically with python (/showthread.php?tid=6090111)

Downloading source code automatically with python - 0u1law - Jun 25, 2020

hey guys.

i need some help in the following case:

i want to write a program that uses my browser. i have a list of links i want to visit and download the source code automatically. thats it.

which packages do i need for this ?

if someone have a piece of code i would be glad!

PSEUDO-CODE:

list_of_links = [... ... ... ...]
for each in list_of_links:
-----open browser
-----show sourcecode
-----download sourcecode
-----close browser

and if there are another ways to do this in a more efficient way please let me know

and just for the curiosity is there a method to do this with bash scripting?

best regards

0u1law

RE: Downloading source code automatically with python - Sickness - Jun 25, 2020

To automate it better you can use Selenium, just initialize a driver, save the sourcecode on a string and save it on a file and done.

RE: Downloading source code automatically with python - Harley Quinn - Jun 25, 2020

(Jun 25, 2020 05:42 AM)0u1law Wrote: hey guys.

i need some help in the following case:

i want to write a program that uses my browser. i have a list of links i want to visit and download the source code automatically. thats it.

which packages do i need for this ?

if someone have a piece of code i would be glad!

PSEUDO-CODE:

list_of_links = [... ... ... ...]
for each in list_of_links:
-----open browser
-----show sourcecode
-----download sourcecode
-----close browser

and if there are another ways to do this in a more efficient way please let me know

and just for the curiosity is there a method to do this with bash scripting?

best regards

0u1law

Hi 🙃

Yes You can use Selenium, or Requests w/ BeautifulSoup.

Hit me up with a DM and i'll see what i can do for you. I'll show you how to do it also so you know and can show someone else one day!

Harley

RE: Downloading source code automatically with python - 4Txe4y33C9zvNhMj - Jun 25, 2020

(Jun 25, 2020 05:52 AM)Sickness. Wrote: To automate it better you can use Selenium, just initialize a driver, save the sourcecode on a string and save it on a file and done.

Selenium is extremely inefficient.

Selenium was made for automated quality assurance, not for datascraping. Since OP just wants the actual source code, he can just use the Python.requests module to download that code.

This should work:

Code 
import requests

list_of_links = ["https://hackforums.net/", "https://google.com/", "https://wikipedia.org/"]

for link in links:

    result = requests.get(link)

    print("downloading link...")

    #do whatever you want with the content

    with f as open(link + "_download.txt", "w+")

        f.write(result)

RE: Downloading source code automatically with python - h0lyl337 - Jun 25, 2020

(Jun 25, 2020 11:31 AM)4Txe4y33C9zvNhMj Wrote: Selenium is extremely inefficient.

Selenium was made for automated quality assurance, not for datascraping. Since OP just wants the actual source code, he can just use the Python.requests module to download that code.

This should work:

Code
import requests list_of_links = ["https://hackforums.net/", "https://google.com/", "https://wikipedia.org/"] for link in links: result = requests.get(link) print("downloading link...") #do whatever you want with the content with f as open(link + "_download.txt", "w+") f.write(result)

i agree, requests would be perfect, bypassing the web browser and use less resource.

RE: Downloading source code automatically with python - Sickness - Jun 25, 2020

(Jun 25, 2020 11:31 AM)4Txe4y33C9zvNhMj Wrote: Selenium is extremely inefficient.

Selenium was made for automated quality assurance, not for datascraping. Since OP just wants the actual source code, he can just use the Python.requests module to download that code.

This should work:

Code
import requests list_of_links = ["https://hackforums.net/", "https://google.com/", "https://wikipedia.org/"] for link in links: result = requests.get(link) print("downloading link...") #do whatever you want with the content with f as open(link + "_download.txt", "w+") f.write(result)

You are right, but for pages that got protection for bots requests wont work always.

RE: Downloading source code automatically with python - 4Txe4y33C9zvNhMj - Jun 25, 2020

(Jun 25, 2020 11:44 AM)Sickness. Wrote: You are right, but for pages that got protection for bots requests wont work always.

In the end you can always fake the actual HTTP request your browser sends. Even if they dynamically get content from an API endpoint you can scrape that too.

RE: Downloading source code automatically with python - Pawnhub - Jun 25, 2020

(Jun 25, 2020 11:44 AM)Sickness. Wrote: You are right, but for pages that got protection for bots requests wont work always.

Selenium wouldn't do any better in this case. I get more sites blocking selenium when cracking then request's.
But if you could 100% be able to mimic a proper request (which can be outstandingly hard with hidden variables and cookies etc) request's would still be the most viable.
Although, selenium can read JS unlike request's. So if the form he wants to scrape from loads from a Iframe then yes selenium is most viable since request's can't.

RE: Downloading source code automatically with python - SIM-4771460 - Jul 4, 2020

Requests or Scrapy can do your Job.

I suggest selenium because It handles Javascript easily, and you can use headless chrome/firefox driver for speed,

RE: Downloading source code automatically with python - 0u1law - Jul 9, 2020

(Jul 4, 2020 04:48 PM)Gosyne ♦ Wrote: Requests or Scrapy can do your Job.

I suggest selenium because It handles Javascript easily, and you can use headless chrome/firefox driver for speed,

how can i use it headless?

RE: Downloading source code automatically with python - SIM-4771460 - Jul 9, 2020

(Jul 9, 2020 12:46 PM)0u1law Wrote: how can i use it headless?

you can use headless chrome/firefox, just put --headless argument in code, or some headless browser.

RE: Downloading source code automatically with python - 0u1law - Jul 9, 2020

(Jul 9, 2020 12:47 PM)Gosyne ♦ Wrote: you can use headless chrome/firefox, just put --headless argument in code, or some headless browser.

thanks a lot!