+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [.NET] Webclient Overload? (/showthread.php?tid=5566675)
Webclient Overload? - Lexxi. - Mar 4, 2017
Hi, so i have a list of around 1 mil lines.
So i tried webbrowsers etc but they take long, best way i found was using webclient.downloadstring, before it starts, the list is opened into a listbox, then one at a time the webclient goes through the list, when it gets between 4200-4500 it crashes. So i thought it was maybe too much data, so i added wc.dispose which still did not work. I thought maybe due to amount of lines in listbox it goes through, then it doesnt define why each time it crashes between 4200-4500, usually something like 4306 or so.
the listbox selecteditem, which the first item is removed once it loads, can this be the issue? Maybe if i just make it go to next item instead of removing, can it fix it possibly?
Code
Public sub work()
ListBox1.SelectedIndex = 0
Dim wc As New Net.WebClient
wc.DownloadString(ListBox1.SelectedItem)
ListBox1.Items.Remove(ListBox1.SelectedItem)
wc.Dispose()
Call work()
End Sub* - Lucasheer - Mar 4, 2017
Do you have a specific error?
Are you making requests to one specific website?
Does still go to the same 4200-4500 if you restart immediately after?
Does the applications memory usage go up?
Also, have you tried this?
Code
Using wc As New WebClient
wc.DownloadString("url")
wc.Dispose()
End UsingI have worked quite a bit with WebClient, skype me : Lucasheer715, if you want direct help 🙂
Also, it might be a better idea to make a string array and just loop that (Not deleting items)
* - Lexxi. - Mar 4, 2017
(Mar 4, 2017 08:16 PM)Lucasheer Wrote: Do you have a specific error?
Are you making requests to one specific website?
Does still go to the same 4200-4500 if you restart immediately after?
Does the applications memory usage go up?
Also, have you tried this?
CodeUsing wc As New WebClient
wc.DownloadString("url")
wc.Dispose()
End Using
I have worked quite a bit with WebClient, skype me : Lucasheer715, if you want direct help 🙂
1. Specific error not really, works fine, but tends to crash around 4.3k or so.
2. same website but link is changed, ex: google.com.au, google.com.ru, google.com.xx.
3. when i restart it has to start over
4. After looking at it, very slowly the memory usage rises, so i believe when it gets around 4.x thousand the usage is very high which causes the crash, im letting it run currently, see what happens when it gets around 4k, mem usage number etc.
But from current looks, seems to be memory which keeps rising very slowly as it keeps going.
options i currently have: Make file lists to 4k lines each, when it finishes file 1, saves setting to file2, then restarts so when it restarts loads setting on file2 which goes through and so on.
* - Lucasheer - Mar 4, 2017
Well if the listbox is causing the memory issue, that is one thing. An easy way to fix that, is by just copying all the items into a string array and going through string array without deleting. Grabbing text from listbox item like that is bad anyways, it takes alot of time when done over and over again.
However, if its caused by webclient, then try the method i told you. It will cause everything to be deleted in memory, dont even have to dispose ;)
* - Lexxi. - Mar 4, 2017
(Mar 4, 2017 08:37 PM)Lucasheer Wrote: Well if the listbox is causing the memory issue, that is one thing. An easy way to fix that, is by just copying all the items into a string array and going through string array without deleting. Grabbing text from listbox item like that is bad anyways, it takes alot of time when done over and over again.
However, if its caused by webclient, then try the method i told you. It will cause everything to be deleted in memory, dont even have to dispose ;)
thanks, i currently added extra line to clear cookies, cache every 100 lines, so far memory staying under 97k for past 5 minutes, so far its going good.
* - Lucasheer - Mar 4, 2017
Glad to hear! Tell me if you have any other problems. BTW, not sure, but you might be interested in multi-threaded requests? If you're really doing thousands of requests, it doesn't make sense not to make several requests at once. Unless you are wanting to keep the resources used on the low side.
* - Lexxi. - Mar 4, 2017
(Mar 4, 2017 09:02 PM)Lucasheer Wrote: Glad to hear! Tell me if you have any other problems. BTW, not sure, but you might be interested in multi-threaded requests? If you're really doing thousands of requests, it doesn't make sense not to make several requests at once. Unless you are wanting to keep the resources used on the low side.
Would be great learning multithreads, i watched some tutorials but still not 100% familiar. Before i just ran the program 10x times, each with a diff file assigned which yes, does multiple but why bother when can do in one program with multithreads, issue is i never really do multi due to lack of knowledge on multi threading.
* - Lucasheer - Mar 4, 2017
Do you have the knowledge to make this program multi-threaded? If not, i can teach you, if you'd like. I've made many applications multi-threaded, but never actually like studied different methods and tokens and what not, so i can definitely teach you how to do something like that.
* - Lexxi. - Mar 4, 2017
(Mar 4, 2017 10:42 PM)Lucasheer Wrote: Do you have the knowledge to make this program multi-threaded? If not, i can teach you, if you'd like. I've made many applications multi-threaded, but never actually like studied different methods and tokens and what not, so i can definitely teach you how to do something like that.
Would be great! I like making bot like applications, sometimes not being able to multi thread them just sucks.
* - SL x TnT - Mar 4, 2017
Code
Public sub work()
ListBox1.SelectedIndex = 0
Dim wc As New Net.WebClient
wc.DownloadString(ListBox1.SelectedItem)
ListBox1.Items.Remove(ListBox1.SelectedItem)
wc.Dispose()
Call work()
End SubYou really should have the items stored in a list instead of the control. It's either on the UI thread and freezing, or you have CheckForIllegalCrossThreadCalls set to false.
Webclient can be in a Using statement instead of manually disposing.
You're using a recursive call to the method. If you're doing that for 1m lines, I'm not surprised that it crashes.
You're assuming there are Items in the control.
If you don't care about it removing the items from the control ...
Store all items in a list.
Call method while passing list of items.
Use Parallel.ForEach for a basic way of multithreading it. Due to how it works, it's not great when you get to the last few requests.
Try/Catch. You may already have this though.
If you want to have it remove after each one (not recommended), I'd use events. I guess you could Invoke it in the foreach loop too to make it easy (but messy).
* - Lucasheer - Mar 4, 2017
Do you have a skype? Mine is Lucasheer715
I was thinking of having a certain amount of threads, and whenever ones finished, assign it the next job in the list or array. I agree that a Parallel.ForEach or Parallel.For a great way for him to do it. He can just set the MaxDegreeOfParallelism for the amount of threads he wants to utilize, correct? However, can you just use a cancellation token for a timeout?
* - uint32 - Mar 5, 2017
I recommend you switch over to the HttpClient for it's easier asynchronous operations and such.
Nonetheless, I suggest you declare your client outside of your work scope; it's useless to create and dispose a new one whenever you can reuse a global one.
Example:
Code
private static HttpClient _globalClient = new HttpClient();
static void Work()
{
_globalClient.GetStringAsync("link").Wait();
Work();
}