![]() |
|
[C#] ProxyGRB - Web Based Proxy Scraper - Printable Version +- Hack Forums (https://hackforums.net) +-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444) +--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151) +---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118) +---- Thread: [C#] ProxyGRB - Web Based Proxy Scraper (/showthread.php?tid=5649520) |
ProxyGRB - Web Based Proxy Scraper - w4zt3d - Jun 18, 2017 I was able to pull 30K proxies with that Theory It's simple. 1. The tool searches for some querys like "proxylist" on Bing 2. Then it visits all the pages. 3. It grabs all proxies on each page using Regular Expressions (Regex). Done! NOTE: WE HAVEN'T CHECKED THE PROXYLIST YET! Source PHP Code * - Rageline - Jun 18, 2017 Dont use regex to parse proxies. Its slow, it causes shit ton of memory allocation and GC fragmentation. Also dont use foreach since it calls GetEnumerator on calls. Use for instead :) Also why is there while(true) {} on end? U can just call Console.ReadLine(); and then program will exit when user presses enter or w\e You should set a useragent because bing will block you for scraping Also you should use HashSet<string> instead of List because it has O(1) lookup and can't contain duplicate items. Also all collections in C# have a member called Count which returns number of elements in it. So you dont need your own counter variable * - Ninja - Jun 18, 2017 (Jun 18, 2017 12:18 AM)w4zt3d Wrote: I was able to pull 30K proxies with that This means nothing more than you put in a lot of sources or sources with lots of proxies. This does not mean anything more. Anyways, as always the code is pure cancer. * - SL x TnT - Jun 18, 2017 (Jun 18, 2017 12:35 AM)Rageline Wrote: Dont use regex to parse proxies. Its slow, it causes shit ton of memory allocation and GC fragmentation. I'm curious as to what you'd recommend for grabbing text from an unknown string with a known pattern. You'd also need an alternate solution before calling it slow. Compared to using Contains on a List and http requests, Regex isn't slow. * - Rageline - Jun 19, 2017 (Jun 18, 2017 02:36 PM)SL x TnT Wrote: I'm curious as to what you'd recommend for grabbing text from an unknown string with a known pattern. You'd also need an alternate solution before calling it slow. Write your own grasshopper algo for proxy parsing, it takes ~ 30mins to write it. I'd share my own but im using it in commercial products so I don't want to. Also OP your regex for proxies is wrong, u are capturing \d{1,5} for port which can capture all digits from 0-9 but ports are ushort size meaning 65535 is max and you will capture all invalid proxies above that :) * - SL x TnT - Jun 19, 2017 (Jun 19, 2017 02:55 AM)Rageline Wrote: Write your own grasshopper algo for proxy parsing, it takes ~ 30mins to write it. I'd share my own but im using it in commercial products so I don't want to. Still going on about this "grasshopper algo," that's not really an algorithm, is well known, yet has no public information? I'd rather spend the 10 seconds writing "slow" code than spend days searching for the elusive grasshopper that will have no noticeable optimizations. Optimizing everything from the start isn't the best route to go. As for the port, it would be easier to check the value after you grab it. * - +Haze - Jun 19, 2017 (Jun 19, 2017 02:55 AM)Rageline Wrote: Write your own grasshopper algo for proxy parsing, it takes ~ 30mins to write it. I'd share my own but im using it in commercial products so I don't want to. See, this is where you're going wrong, but you're too worried about insignificant optimisations. I've said this before and I'll say it again, premature optimisation is the root of all evil. Just like you said, it will take you ~30 minutes to code the algorithm for someone who understands how it works. Who wants to waste those 30 minutes? In the context of modern computing, this performance difference is not noticeable, fair enough if this was the 90s where regex very slow for computers then, but this is 2017 son, it will make no difference to my i7 8 core laptop. Unless, the application is time sensitive then you're just shooting yourself in the foot with your non sense. I can understand if you were debating about using a quick sort algo vs bubble sort or an O(log n) vs O(N^2) algorithm, but that's not the case. Take a look at this https://softwareengineering.stackexchange.com/questions/80084/is-premature-optimization-really-the-root-of-all-evil Spoiler (Click to View) So take a seat son. (Jun 19, 2017 04:29 AM)SL x TnT Wrote: x I'd have to agree with you, and just for future references, the term is called premature optimisation. * - HFDone - Jun 19, 2017 (Jun 19, 2017 02:55 AM)Rageline Wrote: Write your own grasshopper algo for proxy parsing, it takes ~ 30mins to write it. I'd share my own but im using it in commercial products so I don't want to. 🙃 \d numeric digit 0-9 {1,5} 1 to 5 times * - Rageline - Jun 19, 2017 (Jun 19, 2017 07:16 AM)HFDone Wrote: 🙃 5 times means 99999 which is not a valid port. * - Ninja - Jun 19, 2017 (Jun 19, 2017 07:35 AM)Rageline Wrote: 5 times means 99999 which is not a valid port. I love how you always ignore every single argument that you can't disprove. :) * - Owl - Jun 19, 2017 On the subject of regex, combine your delimiters into a charset/separate group and reduce capture work. * - DBots - Jun 19, 2017 Ever heard of IPAddress.TryParse? * - HFDone - Jun 19, 2017 (Jun 19, 2017 07:35 AM)Rageline Wrote: 5 times means 99999 which is not a valid port. 🤦♂ you match not generate > 49999 is a valid port * - SL x TnT - Jun 19, 2017 (Jun 19, 2017 06:36 AM)+Haze Wrote: I'd have to agree with you, and just for future references, the term is called premature optimisation. Thanks. (Jun 19, 2017 07:35 AM)Rageline Wrote: 5 times means 99999 which is not a valid port. It doesn't matter if it's valid or not when finding the proxies. You should try to parse the IP as an IPAddress and the port as a ushort. If either fails, don't use the proxy. * - Ninja - Jun 19, 2017 (Jun 19, 2017 04:29 AM)SL x TnT Wrote: spend days searching for the elusive grasshopper Gave me a good chuckle. 🤣 |