[vorticalbox]

So i am creating a proxy scraper and I am trying to remove not working

so all the proxies are scraped and stored in list called proxies I then try open a webpage (this case facebook.com) using the proxy and then I want it to remove if it doesn't work.

The proxy list is map for speed.

Code

def proxyCheck(p):
    global proxies
    try:
        session.get('https://facebook.com', proxies={'https' :p}, timeout=1)
    except:
        print "%s removed" % p
        proxies.remove(p)


print "removing down proxies (%d)" % len(proxies)
p = Pool(cpu_count())
p.map(proxyCheck, proxies[:]) #[:] to make a copy, I read you can't remove as your iterarting
print "using %d" % len(proxies)

but the length stays the same and nothing is removed, i've tried appends to a new list and that doesn't worth either.

[Riԁdle]

So i am creating a proxy scraper and I am trying to remove not working

so all the proxies are scraped and stored in list called proxies I then try open a webpage (this case facebook.com) using the proxy and then I want it to remove if it doesn't work.

The proxy list is map for speed.

Code

def proxyCheck(p):
    global proxies
    try:
        session.get('https://facebook.com', proxies={'https' :p}, timeout=1)
    except:
        print "%s removed" % p
        proxies.remove(p)


print "removing down proxies (%d)" % len(proxies)
p = Pool(cpu_count())
p.map(proxyCheck, proxies[:]) #[:] to make a copy, I read you can't remove as your iterarting
print "using %d" % len(proxies)

but the length stays the same and nothing is removed, i've tried appends to a new list and that doesn't worth either.

try removing the global proxies line

[vorticalbox]

try removing the global proxies line

That worked but some don't get remove I added a check to see if it is still in the list. I am guessing thats due to access of all the threads?

---

Just let it run it says "removed" but it still in the list :(

---

Just to up date i never did solve this but as a work around i saved working proxies to a file and the emptied the array then filled it up from the file and finally deleting the file.

Not the most efficient way to get the job dome but it works.

[icmp]

If you posted the full traceback we could maybe offer greater assistance.

P.s. to avoid duplicates and to improve general performance, perhaps consider using sets as opposed to arrays. You could load an existing array into a set using the Set(array) function

[vorticalbox]

If you posted the full traceback we could maybe offer greater assistance.

P.s. to avoid duplicates and to improve general performance, perhaps consider using sets as opposed to arrays. You could load an existing array into a set using the Set(array) function

Under up returning true/false and using compression like in my proxy post.