[Proioxis]

I am just learning httpwebrequest. I never had much of a use for it but I want to learn it now. I looked at some tutorials online and such and figured what I wanted to do was make a proxy grabber. I figured it wouldn't be so difficult all I would have to do is grab the text off the website.

Here is what I have so far

Code

Dim webStream As Stream
  Dim webResponse As String
  Dim req As HttpWebRequest
  Dim res As HttpWebResponse
  req = WebRequest.Create("http://www.hidemyass.com/proxy-list/")
  req.UserAgent = ("Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1")
  req.Method = "GET"
  res = req.GetResponse()
  webStream = res.GetResponseStream()
  Dim webStreamReader As New StreamReader(webStream)
  While webStreamReader.Peek >= 0
    webResponse = webStreamReader.ReadToEnd()
  End While
  frmWb.Show()
  frmWb.WebBrowser1.DocumentText = webResponse

It opens another form i made real quick that has a webbrowser in it to display the response. It pretty much brings up the page in all text, no css or pictures. But I'm wondering how I can get the proxies off there? I'm not asking to be spoonfed I'm just wondering where I can find a decent reference on how to get the text I'm looking for off the page

[Cody8295]

If you're just doing a GET request, perhaps the WebClient is a better option.

http://msdn.microsoft.com/en-us/library/...lient.aspx

[Proioxis]

If you're just doing a GET request, perhaps the WebClient is a better option.

http://msdn.microsoft.com/en-us/library/...lient.aspx

thanks for showing me this. I feel this will be a lot easier with a webclient but I've run into a snag. Not sure if you can help but the html code for the website is... confusing to say the least

this html code here

Spoiler (Click to View)

Code

</style><span class="DjBk">61</span><span class="55">.</span><span class="EV5C">22</span><div style="display:none">22</div><span class="Wl4p">49</span><span style="display:none">100</span><span style="display:none">122</span><div style="display:none">122</div><span style="display:none">124</span><span class="EV5C">124</span><span></span><span style="display:none">207</span><span class="129">.</span><span style="display:none">10</span><span></span><span class="Wl4p">40</span><span class="zBUq">99</span><div style="display:none">99</div><span style="display:none">109</span><span class="V7CG">109</span><span></span><span style="display:none">217</span><span style="display:none">219</span><span class="EV5C">219</span><span></span><span style="display:none">232</span><span></span><span class="zBUq">234</span><div style="display:none">234</div><span class="7">.</span><span style="display: inline">28</span></span></td>    
  <td>
8080</td>

seems to just output this


(either that or I'm way to dumb and just missing the obvious)

this is my new code here in replacement of the httpwebrequest

Code

Dim wc As New WebClient
  wc.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)")
  Dim html As Byte() = wc.DownloadData("http://www.hidemyass.com/proxy-list/")
  Dim utf As New UTF8Encoding()
  frmWb.Show()
  frmWb.WebBrowser1.DocumentText = utf.GetString(html)
  RichTextBox1.Text = utf.GetString(html)

[Cody8295]

this is my new code here in replacement of the httpwebrequest

Code

Dim wc As New WebClient
  wc.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)")
  Dim html As Byte() = wc.DownloadData("http://www.hidemyass.com/proxy-list/")
  Dim utf As New UTF8Encoding()
  frmWb.Show()
  frmWb.WebBrowser1.DocumentText = utf.GetString(html)
  RichTextBox1.Text = utf.GetString(html)

You don't even need to do that much, you can minimize all that into

Code

wc.DownloadString("url")

[Proioxis]

You don't even need to do that much, you can minimize all that into

Code

wc.DownloadString("url")

Doing so results in the source still showing up...



However I peeked at some source for a program that does the same thing mine is trying to do (on the same website) and there was this

Code

Dim r As New System.Text.RegularExpressions.Regex("(\d{1,4}\.\d{1,4}\.\d{1,4}\.\d{1,4})</span></td>\D+(\d{1,5})</td>\D+<img src=""\S+""\D+""flag"" /> (\w+)\D+\S+\D+(\d+%)\D+\S+\D+(\d+%)""\D+<td>(\w+)\D+"">(\w+)")

except that seems to have changed because that program doesn't work now. I'd say this project is a bust I'll just try something else because I can't find any sites with a free frequently updating proxy list

[Tidas]

Doing so results in the source still showing up...



However I peeked at some source for a program that does the same thing mine is trying to do (on the same website) and there was this

Code

Dim r As New System.Text.RegularExpressions.Regex("(\d{1,4}\.\d{1,4}\.\d{1,4}\.\d{1,4})</span></td>\D+(\d{1,5})</td>\D+<img src=""\S+""\D+""flag"" /> (\w+)\D+\S+\D+(\d+%)\D+\S+\D+(\d+%)""\D+<td>(\w+)\D+"">(\w+)")

except that seems to have changed because that program doesn't work now. I'd say this project is a bust I'll just try something else because I can't find any sites with a free frequently updating proxy list

Because it downloads the entire string of the website. It doesn't magically know what you want to get from it.

Use RegEx to get the info.

[Proioxis]

Because it downloads the entire string of the website. It doesn't magically know what you want to get from it.

Use RegEx to get the info.

after spending a bit of time researching how to use regex i understand what you mean. Thanks for telling me that

[★☆✰ Matto ✰☆★]

If you struggle with RegEx a quick and dirty solution is to use the public GetBetweenAll and GetBetween functions.

[Cody8295]

If you struggle with RegEx a quick and dirty solution is to use the public GetBetweenAll and GetBetween functions.

I believe GetBetween uses Split(), which is, if anything, less dirty than Regex, especially for HTML.

[★☆✰ Matto ✰☆★]

You'll need to set up loops and cleanup the results, with a good RegEX string you can easily get all the matches in one search.

[TrackPro+]

You'll need to set up loops and cleanup the results, with a good RegEX string you can easily get all the matches in one search.

That's what most are suggesting, but instead of suggesting, help?

[★☆✰ Matto ✰☆★]

I don't know how to write good RegEX strings? Instead of criticising my posts and getting off-topic, help?

OP: You can find loads of Regular Expressions to match IP addresses on Google, here are some (untested), obviously these will only get the IP not the port, this is a starting point for you if you want to use RegEX

\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
\b(?:\d{1,3}\.){3}\d{1,3}\b

[Cody8295]

I don't know how to write good RegEX strings? Instead of criticising my posts and getting off-topic, help?

OP: You can find loads of Regular Expressions to match IP addresses on Google, here are some (untested), obviously these will only get the IP not the port, this is a starting point for you if you want to use RegEX

\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
\b(?:\d{1,3}\.){3}\d{1,3}\b

RegEx isn't even appropriate for HTML anyways...

[★☆✰ Matto ✰☆★]

I thought you can use RegEX for any kind of text?

[Cody8295]

I thought you can use RegEX for any kind of text?

You can, but HTML tags aren't regular expressions, therefor RegEx isn't the best choice.