[Pr0m3th3us]

Robin - A DarkWeb Scraper

We all know its hard enough to find your way around in the Dark. Fumbling around till you run into the doorknob. Even then your just reaching your hand out and using only a couple of your senses, mainly touch, you may even run into 7. So what about the DarkWeb? Well its very similar, only a couple senses are available to you. It's honestly a hassle unless you know somebody who already gave you links or pointed you in the right direction. Most search engines on the Dark that claim to be accurate are just...not. Even then you gotta be careful you don't step on a nail, get burned or...worse. In this TUT i will introduce you to a program that uses AI to scrape the DarkWeb for you and legit gives you links that actually work. Of course its not perfect, nothing is in this world. But it helps to have a tool no doubt. This is that tool. Robin.

First thing is first broskis....update your shit.

Code

sudo apt update
sudo apt upgrade

Then install and enable good ole government made Tor

Code

sudo apt install tor -y
sudo systemctl enable tor
sudo systemctl start tor

Check that Tor is running, you should see some green text saying ENABLED and ACTIVE.

Code

sudo systemctl status tor

Now your going to have to install Docker. We are using a Docker image which pulls directly from Docker Hub. No Git Hub clone needed. However if you want to use python method instead you can do that, i do not cover the steps in this TUT though.

Code

sudo apt install docker.io docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

This program requires us to setup a .env file with your LLM API keys, you can use Ollama for free which is what im doing in this TUT. This is the command to install Ollama

Code

curl -fsSL https://ollama.com/install.sh | sh

That's it, it installs the binary, creates a systemd service, and starts it. Verify it's running

Code

ollama serve &
ollama list

Now lets setup that .env file

Code

# Option 1 - Anthropic Claude API
# ANTHROPIC_API_KEY=your_key_here

# Option 2 - OpenAI
# OPENAI_API_KEY=your_key_here

# Option 3 - Ollama (free, local, no API key needed)
OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_MODEL=llama3.2

Make a directory and edit your file to add your AI LLM info according to the options above

Code

[code]mkdir -p ~/robin && cd ~/robin
nano .env[/code]

Code

OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_MODEL=llama3.2
EOF

Code

systemctl status ollama

Stop DOCKER_HOST if necessary, also add an exception to your firewall if needed

Code

unset DOCKER_HOST

Code

sudo ufw allow from 172.17.0.0/16 to any port 11434

Pull what we need, and run the program. This is going to take a couple minutes so let it run till it shows "bootstrapped 100%" but do not ctrl + c when it is done.

Code

cd ~/robin

docker pull apurvsg/robin:latest

docker run --rm \
  -v "$(pwd)/.env:/app/.env" \
  -v "$(pwd)/investigations:/app/investigations" \
  --add-host=host.docker.internal:host-gateway \
  -p 8501:8501 \
  apurvsg/robin:latest

Now go to your browser and use this to connect to the program http://localhost:8501 You should see this screen. Search for what you want. It will narrow it down for you and even check to see if the .onion links it scrapes are active or not.

When you get your links open up Tor Browser and paste them in. If you need any help at all just message me in convo please. Or leave a reply and ill hit you up. Thanks.

Entering the Brotherhood Tutorial Contest.
7 Mercury MadScientist Weekend Offender

Disclaimer:
The information provided on this tutorial is for general informational purposes only and does not constitute advice of any kind. While i strive to ensure the accuracy and reliability of the information presented, i make no warranties or representations, express or implied, regarding its completeness, accuracy, timeliness, or suitability for any particular purpose. Hackforums and i shall not be liable for any loss or damage of any kind arising from the use of or reliance on the information provided. We encourage you to consult with qualified professionals before making any decisions based on this information

[PedroG]

Nice write up and helpful for anyone whose never had experience finding link sites before!

[Vagueness]

Coming back to this later thanks fam

[Mastercarder]

Great work. Full of detailed information.