[Anidektwist]

fo some reason i am interested in a website database
no idea why but i just like the idea of having a big database of websites
so i saw that Yahoo uses bingbot instead of Slurp (their old webcrawler)
would it ever been released to the public? it doesnt get used anyway.

i have been looking around but the standard "consumer" crawler is shit.
most of them doesnt even save the database untill you stop the crawler.
any good crawlers out there that saves every link after it finds it and that can remove duplicates?

---

no one?
infocrawler works ok but it hasn't been updated since 2008 and dont like the web interface