Home Upgrade Search Memberlist Extras Hacker Tools Award Goals Help Wiki Contact

C# 

Best place to learn ThreadPools?

All I can find is indians talking at 2 words per minute on 240p.

If anyone can share any deets on where they learned them and what was the best technique, would appreciate it very much so.

I would use them for HttpWebRequests, as moving on to WinForms for WebRequests from Console app means I need to learn this.

Thanks!
If you want to do a bunch of webrequests fast, I would recommend creating your own implementation with tcp sockets depending on the complexity of the action, i got it to ~30 requests per second that way (across 5 threads), and with webrequest objects i got it to around 7 requests per second

If you meant to make requests without blocking the main thread, then look into async tasks and http
[Image: Syr3DlV.png]
(Oct 17, 2017 08:16 AM)gigajew Wrote:  If you want to do a bunch of webrequests fast, I would recommend creating your own implementation with tcp sockets depending on the complexity of the action, i got it to ~30 requests per second that way (across 5 threads), and with webrequest objects i got it to around 7 requests per second

If you meant to make requests without blocking the main thread, then look into async tasks and http

All WebRequests are are a wrapper over tcp clients (or sockets I can't remember), so I highly doubt you got it that much faster. Anyways, the built in threadpool isn't that great in my opinion, I recommend looking into the TPL (Tasks and TaskFactories).
(Oct 17, 2017 08:16 AM)gigajew Wrote:  If you want to do a bunch of webrequests fast, I would recommend creating your own implementation with tcp sockets depending on the complexity of the action, i got it to ~30 requests per second that way (across 5 threads), and with webrequest objects i got it to around 7 requests per second

If you meant to make requests without blocking the main thread, then look into async tasks and http

I have no idea what you're doing, but I've managed ~900k requests every minute across 3 servers or 5000 requests per second per server without having any issues. This was relaying requests too.

Did you make sure to increase your limits?




Use an HttpClient. Don't dispose it either. They're not meant to be disposed even though they implement IDisposable. Other than that, I've only had 2 issues with them.
I should say that I’m using a shitty old system that I bought for $8 but it’s all I have rn. Also it’s connection is through a WiFi dongle (the system is incredibly unstable and slow)

But yeah got it much faster handling sockets myself, but I’d love to see sample code of your solution if you don’t mind

And to Ninja yeah i really did get it that much faster because when i implemented them manually I could force the socket to stay alive and re use the resources across each request
[Image: Syr3DlV.png]
(Oct 17, 2017 08:40 PM)gigajew Wrote:  I should say that I’m using a shitty old system that I bought for $8 but it’s all I have rn. Also it’s connection is through a WiFi dongle (the system is incredibly unstable and slow)

But yeah got it much faster handling sockets myself, but I’d love to see sample code of your solution if you don’t mind

And to Ninja yeah i really did get it that much faster because when i implemented them manually I could force the socket to stay alive and re use the resources across each request

I'm using an HttpClient.
If you're able to get "x" per second using your own implementation, it should be possible to get near "x" using .net's.

How are you doing it (not personal implementation)?
Are you using tasks?
Parallel?
Did you increase the connection limit?
(Oct 17, 2017 08:40 PM)gigajew Wrote:  re use the resources across each request

What do you mean by this?
(Oct 18, 2017 08:28 AM)Ninja Wrote:  What do you mean by this?

If the uri's I want to GET or POST to is to the same host/port then I can re-use certain resources and I won't have to re-create any httpwebrequest objects or re instance any streams


(Oct 17, 2017 09:03 PM)SL x TnT Wrote:  I'm using an HttpClient.
If you're able to get "x" per second using your own implementation, it should be possible to get near "x" using .net's.

How are you doing it (not personal implementation)?
Are you using tasks?
Parallel?
Did you increase the connection limit?

The first way I did it was using HttpWebRequest and WebClient and all of those things, but they were proving very slow for my needs, since I didn't need any complex actions
[Image: Syr3DlV.png]
(Oct 18, 2017 01:42 PM)gigajew Wrote:  If the uri's I want to GET or POST to is to the same host/port then I can re-use certain resources and I won't have to re-create any httpwebrequest objects or re instance any streams



The first way I did it was using HttpWebRequest and WebClient and all of those things, but they were proving very slow for my needs, since I didn't need any complex actions

Don't think you understood correctly.

Are you using Thread or Task/Threadpool to send the requests? Will make a difference in some cases.
Did you increase the connection limit?
(Oct 18, 2017 08:19 PM)SL x TnT Wrote:  Don't think you understood correctly.

Are you using Thread or Task/Threadpool to send the requests? Will make a difference in some cases.
Did you increase the connection limit?

Oh sorry, manual threads (5) and yeah I set the max connection limit
[Image: Syr3DlV.png]
(Oct 18, 2017 01:42 PM)gigajew Wrote:  If the uri's I want to GET or POST to is to the same host/port then I can re-use certain resources and I won't have to re-create any httpwebrequest objects or re instance any streams

I remember trying that and I remember I had to close the connection else it would not work. Mind posting/sending me some snippets? I am curious how you re-use the same socket/stream for multiple requests on a single host.


(Oct 18, 2017 09:09 PM)gigajew Wrote:  Oh sorry, manual threads (5) and yeah I set the max connection limit

Sometimes the threadpool can take some time to spawn the rest of the threads.
(Oct 19, 2017 08:00 AM)Ninja Wrote:  Sometimes the threadpool can take some time to spawn the rest of the threads.

That's why I asked. Manual threads shouldn't have that issue.

Last thing I can think of, without seeing the code, is that his connection or the website is slow.
It’s definitely my connection and my computer by itself, my USB ports have unstable power and my WiFi goes through that


Replaced my motherboard with a Socket T w/8gb RAM and now the USB should be stable, without a lot of testing I see ~130 requests per second across 5 threads (200 connections per thread). (and rising, but I won't keep it running that long) That's a little faster

And with somebody who is better at managing threads than me, they can probably get it to a whole lot more than I can
[Image: Syr3DlV.png]


Possibly Related Threads...
Thread / Author Replies Views Last Post
[C] by F-NukeJul 6, 2021 02:29 AM 16 1,395 Jul 6, 2021 02:29 AM
Last Post: BigDawgAustin
[C] by shadilayJun 30, 2019 03:52 PM 3 163 Jun 30, 2019 03:52 PM
Last Post: TGilb94
[C] by VoxiOct 31, 2018 07:53 AM 13 652 Oct 31, 2018 07:53 AM
Last Post: Moslaxx