Content Discovery
Created bymhmdiaa-trickest
Last updated11/8/2024

Input Parameters

​
hosts
file
required
List of hosts

Outputs

urlsurl-details

​
Discover Paths via OSINT Sources

​
Description

Search OSINT sources for a list of hosts to discover hidden paths and endpoints.

​
Features

  • Aggregates URLs from multiple OSINT sources, combining historical and recent data.
  • Cleans and normalizes URLs to consolidate semantic duplicates and ensure consistent formatting.
  • Can process thousands of hosts simulataneously.

​
Inputs

​
Required

  • hosts: A list of hostnames for which URLs are to be discovered
foo.example.com
bar.example.net

​
Outputs

  • urls: List of discovered URLs.
https://foo.example.com/login
https://bar.example.net/admin
https://bar.example.net/signup
  • url-details: JSONLines records of URL discovery details.
{"url": "https://foo.example.com/login", "hostname": "foo.example.com", "domain_name": "example.com", "data_source": "osint source"}
{"url": "https://bar.example.net/admin", "hostname": "bar.example.net", "domain_name": "example.com", "data_source": "osint source",}
{"url": "https://bar.example.net/signup", "hostname": "bar.example.net", "domain_name": "example.com", "data_source": "osint source"}

​
Changelog

  • v1.0.0
    • Initial release