(Jan 9, 2022 02:45 PM)evilgnome Wrote:  I've got a request made with "requests" library, and one with "HTTPX" library, I've tweaked the requests to be the same, byte for byte, literally re-arranging the headers and everything, yet the one made with "requests" succeed and the one with "HTTPX" gets detected as bot activity / scraper.

What is the next troubleshooting steps? Like how do I even diagnose something like this?

The reason I want to use HTTPX is because I am using FastAPI and when I use requests it's fast and all until I have like 10 concurrent users of the API, then it'll serve the first 10 and put everyone else in a queue. I don't know why, it's not a CPU/anything bottleneck, I've tried this "workaround" to no avail:

Code

session = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=100000, pool_maxsize=100000)
session.mount('https://', adapter)

If someone knows
a) how I can go deeper looking into requests from HTTPX/requests to where they both succeed
b) raise the limit from 10 to 10000 on concurrent users of the API

EDIT: Explanation of flow
User -> My FastAPI endpoint -> External requests (...takes 5-10 seconds...) -> Returns the response

Once there are more than ~10 user waiting for a response, subsequent requests to my API won't even start their external requests 'til those are delivered, etc. The external server is more than capable to receive 100x more concurrent requests than my API sends.