Whether you're interested in penetration testing, OSINT, or bug bounty, this quick and easy Golang-based crawler is an essential addition to your recon toolbox.
On a newly configured Ubuntu ARM64 virtual machine, we will install Hakrawler, run practical examples, and learn how to use it efficiently.
Setup
"Let's start by installing Hakrawler on a fresh Ubuntu installation."
Terminal Commands:
sudo apt update && sudo apt upgrade -y
sudo apt install golang git -y
"First, we must install Go and Git. We'll use go install to install Hakrawler after that is finished.
go install github.com/hakluke/hakrawler@latest
The binary can be found at ~/go/bin/hakrawler after installation. Let us permanently include it in our PATH.
echo 'export PATH=$PATH:~/go/bin' >> ~/.bashrc
source ~/.bashrc
"Now, let's see if it works."
hakrawler --help
"Awesome! The hakrawler is now set up and operational.
Usage
"Let's crawl our initial website. For testing, I'll use http://www.openexploit.in as a secure target.
echo http://www.openexploit.in | hakrawler
"Hakrawler begins crawling after reading URLs from standard input." Multiple URLs can also be entered from a file in the manner described here:
cat urls.txt | hakrawler
"To illustrate that, let's make a brief urls.txt file."
echo -e "http://httpbin.org\nhttp://openexploit.in" > urls.txt
cat urls.txt | hakrawler
Bug Bounty-Style Recon
An actual bug bounty recon chain might look like this. Assume that you are testing target.com. You use Hakrawler to crawl them after first gathering subdomains with Haktrails and filtering live hosts with httpx.
echo openexploit.in | haktrails subdomains | httpx | hakrawler
"You receive a list of all subdomains, determine which are active, and then send them all to Hakrawler in a single line for crawling!"
Options & Flgs
"Hakrawler provides a number of helpful flags." Let's review a few that I use frequently.
Subdomains:
echo http://www.openexploit.in | hakrawler -subs
Depth:
echo http://example.com | hakrawler -subs
HTTP Headers:
echo http://www.openexploit.in | hakrawler -h "User-Agent: OpenExploitBot;;Referer: http://google.com"
JSON output:
echo http://www.openexploit.in | hakrawler -json
Proxy(such as Burp Suite):
echo http://www.openexploit.in | hakrawler -proxy http://127.0.0.1:8080
"You have more control over how and what you crawl with each of these flags."
Typical Problem and Solutions
You may not always receive URLs back. This is typically the result of the domain redirecting to a subdomain, so don't panic. As an example:
echo http://www.openexploit.in | hakrawler
"You will not receive any results if this reroutes to http://www.openexploit.in unless you either add -subs to include subdomains or crawl the redirected URL directly."
Bonus: Using Docker
Would you rather use Docker or not install Go? Here's how to use Docker with Hakrawler.
DockerHub:
echo http://httpbin.org | docker run --rm -i hakluke/hakrawler:v2 -subs
Local:
git clone http://github.com/hakluke/hakrawler
cd hakrawler
sudo docker build -t hakluke/hakrawler .
echo http://httpbin.org | sudo docker run --rm -i hakluke/hakrawler -subs
Final Words
It is quick, easy, and very useful for reconfiguring and finding endpoints quickly during bug bounty hunts or pentests.
Until the next time, be safe and curious!
Prefer watching instead of reading? Here’s a quick video guide
