+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Shell] how to copy website pages url links (/showthread.php?tid=5669649)
how to copy website pages url links - mralihaider - Jul 14, 2017
hello, i want to copy all pages url links from shutterstock website in the food category
www.shutterstock.com/search?page=1&searchterm=Food&sort=popular&image_type=photo&safe=true&search_source=base_search_form&language=en
note that the number in red is a page number variable and it is from 1 until 151018
so {1..151018}
what tool can i use, if you could tell me a tool
or write me bash script i would appreciate it.
* - SIM-3642159 - Jul 14, 2017
im not going to write the script for you but, wget the url, sed to remove the urls and output them in a txt file and loop my dude.
* - Hackoo - Jul 14, 2017
Hi ✌ if you mean a batch file 😅
Just a little Loop For /L and you will got it like this script 😉
Get_shutterstock_Urls.bat
Get_shutterstock_Urls.bat (Click to View)
Edit : You can also do it, with a vbscript file, it's much faster then the batch file 😉 😝
Get_shutterstock_Urls.vbs
Get_shutterstock_Urls.vbs (Click to View)
👍 😉 👍
* - mralihaider - Jul 16, 2017
thank you Hackoo but i use linux mint
i can't use your script
still need this question if you could help i would appreciate it
* - oneghost - Jul 16, 2017
lt's sound for me as a scraper your questions if I understood it wrong please send me more details on PM ( what info do you want to gather , url pictures description?)
you should try to find which one will be the last page before increment the page number and when the last page will be = with the value from loop to make the script to stop.
with curl , sed , grep and a loop you may be able to do it.
or if you didn't decide yet how to do it, I may recommend you to use php (with dom or directly CURL with some functions ) or perl scripting , as you may randomize the information's about your browser.