[draxyee]

Hi everyone! I'm sending many requests in parallel via HttpClient, but eventually my software runs out of memory (OutOfMemoryException).

I've tried setting

PHP Code


<?php 
var sp = ServicePointManager.FindServicePoint(new Uri("http://googleapis.com/")); 
sp.ConnectionLeaseTimeout = 60 * 1000; 


But no luck. We then tried setting:

PHP Code


<?php 
Client.DefaultRequestHeaders.ConnectionClose = true; 


(Connection:close header)

And it seemed to work fine, but eventually gave us this error:

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

Anyone have experience with HttpClient & memory leaks / sending many requests at a time?

[Shadow_Cloud]

Hi everyone! I'm sending many requests in parallel via HttpClient, but eventually my software runs out of memory (OutOfMemoryException).

I've tried setting

PHP Code


<?php 
var sp = ServicePointManager.FindServicePoint(new Uri("http://googleapis.com/")); 
sp.ConnectionLeaseTimeout = 60 * 1000; 


But no luck. We then tried setting:

PHP Code


<?php 
Client.DefaultRequestHeaders.ConnectionClose = true; 


(Connection:close header)

And it seemed to work fine, but eventually gave us this _error_:


Anyone have experience with HttpClient & memory leaks / sending many requests at a time?

So this might sound silly, but are you using a single instance of the HttpClient in your app? Or are you using a new instance for every call?

I know you typically have a using statement for things that implement IDisposable, but this is actually bad for HttpClient. I suspect you might actually be hitting port exhaustion at the moment hence the error.

If you look at some third party libraries for Http calls, you'll see they typically implement a singleton for this too.

[draxyee]

So this might sound silly, but are you using a single instance of the HttpClient in your app?  Or are you using a new instance for every call?

I know you typically have a using statement for things that implement IDisposable, but this is actually bad for HttpClient.  I suspect you might actually be hitting port exhaustion at the moment hence the _error_.

If you look at some third party libraries for Http calls, you'll see they typically implement a singleton for this too.

Thanks for the message back. I'm not reusing one HttpClient because we can't modify the proxy or the cookies of it. At this point I think the only solution is to switch to HttpWebRequest 😭

[Shadow_Cloud]

Thanks for the message back. I'm not reusing one HttpClient because we can't modify the proxy or the cookies of it. At this point I think the only solution is to switch to HttpWebRequest 😭

Have you taken a look at RestSharp?

https://github.com/restsharp/RestSharp (https://restsharp.dev/)

Code

var client = new RestClient("https://api.twitter.com/1.1");
client.Authenticator = new HttpBasicAuthenticator("username", "password");
client.Proxy = new WebProxy("http://proxy.example.com")

var request = new RestRequest("statuses/home_timeline.json", DataFormat.Json);
request.AddParameter(_cookie_name, _cookie_value, ParameterType.Cookie);

var timeline = await client.GetAsync<HomeTimeline>(request, cancellationToken);

[SIM-4707068]

You can modify the cookie or using proxy using HttpClient. You need to use custom HttpClientHandler.

To use custom cookie, basically it goes like this:

Code

var cookieContainer = new CookieContainer(); // this is the object you will use to fill the cookie
var handler = new HttpClientHandler() { CookieContainer = cookieContainer }
var client = new HttpClient(handler);

and for proxy, you also use custom handler:

Code

var proxy = new WebProxy
{
   Address = uri,
   BypassProxyOnLocal = false,
   UseDefaultCredentials = false,

   Credentials = new NetworkCredential(
       userName: proxyUserName,
       password: proxyPassword)
};

var httpClientHandler = new HttpClientHandler
{
   Proxy = proxy,
};

---

and related to your error "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full." maybe it is because you open too many connection and not related to memory leak?

[Joel]

I don't know if this will help you in your current situation but its worth a look, It helped me with some issues I was having.
https://aspnetmonsters.com/2016/08/2016-...ientwrong/

[devvariety]

Sorry for grave digging this, but I'm having this exact issue. I'm only using one HttpClient with Parallel.For concurrently and receive this after 10s

[Shadow_Cloud]

Sorry for grave digging this, but I'm having this exact issue. I'm only using one HttpClient with Parallel.For concurrently and receive this after 10s

Are you seeing the same problem if you don't run them in Parallel? Maybe share some code?

[devvariety]

The problem doesn't exist in the Parallel.For loop, but then becomes slow. It's just a request to Google for now. I need concurrent fast requests and I don't want to use a non-async library.

[iolaka]

Without posting your code, I can't tell you why your application has a memory leak. Post the full code.
I recommend using Leaf.xNet instead of HttpClient.