[TeachMe1]

I have been working on a small application that can crawl Google to show some of the search engine results pages and the WebClient normally connects fine, but after trying to connect to Google thru the WebClient proxy property, it just isn't connecting properly. Here is the code that I'm working with:

Code

Try  
           Dim wc As New System.Net.WebClient()  
          'This line below will bring in the "User Agent" to make the HTTP request:  
           wc.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36")                
           Dim wp As New System.Net.WebProxy(TextBox1.Text)  
           wc.Proxy = wp  
           Dim HTML As String = wc.DownloadString("Google's Home Page URL /search?q=dog+training")  
 
       Catch ex As Exception  
           MsgBox(ex.Message)  
End Try  

I know that I have found proxies that should work with Google as I've clearly tested them in my Google Chrome browser to see if they're dead or alive, but when I try to connect in my own application through the WebClient proxy setup, I keep getting the following error messages:

"Unable to connect to the remote server"


OR


"(503) Server Unavailable"


What am I missing? Could it be the headers are not fully configured or some other properties? Would appreciate any ideas.

[Soldiercrimes]

Why do you set useragent as a header?

Code

wc.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"

should do the job

and about the proxies:

Code

            Dim Proxies As New Queue(Of String)(File.ReadAllLines(directory))
                                                     Dim usedProxy As String = ""

                                                     usedProxy = Proxies(New Random().Next(1, Proxies.Count - 1))
                                                     wc.Proxy = New WebProxy(usedProxy)

something like this should do the trick.

[Ninja]

Why do you set useragent as a header?

Code

wc.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"

should do the job

and about the proxies:

Code

            Dim Proxies As New Queue(Of String)(File.ReadAllLines(directory))
                                                     Dim usedProxy As String = ""

                                                     usedProxy = Proxies(New Random().Next(1, Proxies.Count - 1))
                                                     wc.Proxy = New WebProxy(usedProxy)

something like this should do the trick.

No. Don't declare a new random instance just for getting a random integer. Declare the random once outside of the method, or as a constant inside the method, and then reference that. Also, validate the input. Read the text file line by line, check if they are valid proxies, then enqueue them to the queue.

Also, your answer does not answer his question.

---

@OP are you sure the proxies are HTTP and that you use the exact same proxy in your program as in your browser? If you are trying to use socks proxies you are out of luck as .NET framework has no support for socks proxies on it's own.

[Sn0wX]

"user-agent" is a header. The webclient class has no direct "user-agent" property. It would of taken you a matter of minutes to check this before correcting a member's correct code to something not even provided. Remember the webclient class is only a wrapper. For more flexibility, useother classes made ready for you.

Ninja:

As i know you know the web client interface uses Internet explorers settings. At a guess, i would say, even when specifying a proxy the property would still be needed to be overridden. My guess without looking, BypassProxyOnLocal needs to be set to true.

Maybe, maybe not. Have no intention looking. I am sure you would have done more research if your own project. But why do others work?

:)