+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Shell] Pulling links from source code with grep? (/showthread.php?tid=5633045)
Pulling links from source code with grep? - Tesseract.mk - May 27, 2017
Hey guys, first post
I've been taking online courses online for security+ and some other subjects, it's pretty much just video tutorials, I looked at the source code and the links to the videos aren't hidden so I can pull them down and save them, but there's loads of videos. I'm trying to run grep to pull the links from the source code but not having much luck.
cat source.php | grep *.mp4 > links.txt
Thats the command I tried but no luck, not sure if the command is wrong or if it's because the links aren't on their own line. Any help would be much appreciated :)
* - Dialog - May 27, 2017
Code
cat source.php | grep -Eo '(http|https):\/\/[^"]+\.mp4' > links.txt* - Tesseract.mk - May 27, 2017
That's awesome thanks a lot :)
Better get my head back in the man page, I'm not as good as I'd like to be in the terminal yet
* - Dialog - May 27, 2017
Save them all to the current working directory:
Code
for m in $(cat source.php | grep -Eo '(http|https):\/\/[^"]+\.mp4'); do
curl -O $m;
done* - Tesseract.mk - May 27, 2017
Is that perl in there too? The more you reply the less I feel I know, what would you recommend as a good resource to master the shell?
I've got The Linux Command Line by No starch press and some RHCSA video tutorials but they don't seem to cover this to your extent
* - Dialog - May 27, 2017
(May 27, 2017 12:26 PM)Tesseract.mk Wrote: Is that perl in there too? The more you reply the less I feel I know, what would you recommend as a good resource to master the shell?
I've got The Linux Command Line by No starch press and some RHCSA video tutorials but they don't seem to cover this to your extent
No - it's simply Bash, with Regex for matching the URLs.
Got used to it by doing, never read an actual book on it: just man pages and Google.
I've worked more seriously with other scripting languages though, so that probably makes it easier.
* - Hackoo - May 27, 2017
If your are intersted with batch too; here is some examples : How to extract data from website ?