If you are interested in open source intelligence (OSINT), you might have heard of Spiderfoot which is a powerful tool that automates the intel process and helps analyzing data from various sources. It can help you gather domains, subdomains, IP addresses, email addresses, phone numbers, usernames, vulnerabilities, and other information related to the target.
In this article, We will show you how to get started with Spiderfoot and use it to perform OSINT tasks more efficiently and effectively.
Spiderfoot Installation
Before starting installation process, you need to have atleast Python version 3.7 (python3) or higher installed with pip (python3) package manager on your system. You also need wget (to download spiderfoot build) or git (to clone developement repository) to download Spiderfoot.
To install Spiderfoot, open a terminal window and run these commands:
# Clone latest Spiderfoot development brach
git clone https://github.com/smicallef/spiderfoot.git
# or get the stable build
wget https://github.com/smicallef/spiderfoot/archive/v4.0.tar.gz && tar zxvf v4.0.tar.gz && mv spiderfoot-4.0/ spiderfoot
# Change directory to spiderfoot
cd spiderfoot
# Install the dependencies
pip install -r requirements.txt
Alternatively, you can use this Dockerfile to run Spiderfoot in a containerized environment.
Spiderfoot Configuration and setup
The first thing you need to do is start spiderfoot instance on an interface using sf.py script. Enter the following command in your terminal window:
# Run the script and specify port
python3 sf.py -l 127.0.0.1:8080
Then open your browser and go to http://127.0.0.1:8080. You should see Spiderfoot default scan page. The web-based user interface (UI) that allows us to configure setting which create, run, and manage scans, as well as view and export the results.
If you want to use Spiderfoot in CLI mode, you can use the sfcli.py script after using sf.py to start the instance. The sfcli.py script allows you to create, run, and manage scans all from command line.For example:
# Connect to spiderfoot instance with URL (default http://127.0.0.1:5001)
python3 sfcli.py -s http://127.0.0.1:8080
# Show help and usage information
python3 sfcli.py -h
This opens an interactive shell which lets you control all scans and perform all sort of tasks through command line.
In the web interface, it is easy to add/import API keys for all the tools form the "Settings" tab in the top navbar. Click on any tool given in Left sidebar to configure it.
You can also configure some global settings from the "global" settings option for the Spiderfoot instance, such as scan DNS resolvers, frequency, proxy settings, TOR integration, etc.
Running Spiderfoot Scans
This is interesting part where we define the intel to gather after specifying the target. To create a new scan on web UI of Spiderfoot, click on the "New Scan" tab on top left corner of navbar. Enter a scan name for future reference. You need to enter a target and choose a scan profile.
A target can be an IP v4 or v6 address, a domain name, a username, a human-name, an email address, a phone number, a bitcoin address, a hostname, or a subnet. For example:
- 1.0.0.1
- 1.0.0.1/24
- user_name
- John Doe
- alice@example.com
- example.com
- abc.example.com
A scan profile determines how aggressive or passive your scan will be with the scope of the information you need. Spiderfoot offers three scan options:
- By Use Case: this contains three profiles for your use cases to get all peice of information. (Passive scan is recommended for beginners)
- By Required Data: This lets you specify the information you want to gather about the target. It will automatically use right tools for that.
- By Module: Specify the tools from the list of hundreds which will run for gathering information. (better when you have special API keys for a tool.)
After entering a target and choosing a scan profile, you can click on "Run Scan Now" to start your scan.
Monitor scans in Spiderfoot
The "Scans" tab allows you to on top navbar. There will be a table that shows the scan name, target, profile, start time, end time, status, elemets (data found) and actions.
You can pause, resume, stop, restart or even delete a scan at any time by clicking on the corresponding buttons under the "Actions" column.
Analyze Spiderfoot Scan Results
Spiderfoot allows us to view and filter scan results, you can go to the "Browse" tab. You can expand or collapse each node to see more or less details.
Also, use the search box on sub-navbar to search for specific data elements or keywords with help of regular expression.
The "Graph" tab on the sub-navbar is used to visualize scan results in a graphical spider-verse form. It creates a network graph that shows nodes and edges of the data elements and their relationships. You can zoom in or out, or click on each node or edge to see more information.
Spiderfoot also supports data export and visualization in CSV and Excel formats with the export (Download-like) button.
Conclusion
We now know a lot about Spiderfoot tool to improve recon/intel process and get more information about the target. Although, we discussed about setup and configuration but if you still face any error or neet any help feel free to drop a message over
Spiderfoot's Discord channel.
