Probe for Web Servers

Probe for web servers on a list of hostnames, IP addresses, or IP ranges

Attack Surface Management

Created by

trickest-mhmdiaa

Last updated

7/4/2024

Input Parameters

hosts · file · required

List of hostnames/IP addresses/IP ranges

header · string

Header(s) to include in HTTP requests

header-file · file

Header(s) to include in HTTP requests

Outputs

web-serversweb-server-detailssubdomainssubdomain-detailspotential-hostnamespotential-hostname-detailssubdomain-wildcardssubdomain-wildcard-detailspotential-hostname-wildcardspotential-hostname-wildcard-details

Probe for Web Servers

Description

Probe for web servers on several common HTTP ports and collect relevant details about each discovered server. This module checks a list of common HTTP ports and gathers information including HTML titles, redirects, content lengths, favicons, TLS data, and more. You can supply hostnames, IP addresses, or IP ranges.

Features

• Enriches discovered hosts with useful data to guide prioritization and help identify patterns.
• Capable of processing tens of thousands of hosts simultaneously.
• Supports hostnames, IP addresses, and IP ranges.
• Probes the most statistically likely ports to have HTTP servers, balancing speed and thoroughness.
• Discovers additional hostnames associated with the target organization.

Inputs

Required

• hosts: a list of hostnames, IP addresses, or IP ranges

dashboard.example.com
1.2.3.4
5.6.7.8/24

Optional

• header: Header(s) to include in HTTP requests
• header-file: File with header(s) to include in HTTP requests

Outputs

• web-servers: List of web server URLs.

https://dashboard.example.com
http://1.2.3.4
http://5.6.7.10:8080

• web-server-details: JSONLines records of web server details.

{"timestamp":"2024-01-01T11:11:11.111111111Z","port":443,"url":"https://dashboard.example.com","input":"dashboard.example.com","scheme":"https","webserver":"cloudflare","method":"GET","host":"1.2.3.4","path":"/","favicon_path":"/favicon.ico","time":"133.7ms","a":["1.2.3.4","1.2.3.4"],"words":7331,"lines":1337,"status_code":200,"cdn":true,"cdn_name":"cloudflare","tls_host":"dashboard.example.com","tls_port":"443","tls_probe_status":true,"tls_version":"tls13","tls_cipher":"TLS_AES_128_GCM_SHA256","tls_not_before":"2021-3-3T07:00:00Z","tls_not_after":"2027-3-3T07:00:00Z","tls_subject_dn":"CN=*.example.com, O=Example\\, Inc., L=San Francisco, ST=California, C=US","tls_subject_cn":"*.example.com","tls_subject_org":["Example, Inc."],"tls_subject_an":["*.example.com","example.com"],"tls_issuer_dn":"CN=Acme TLS RSA SHA256 2020 CA1, O=Acme Inc, C=US","tls_issuer_cn":"Acme TLS RSA SHA256 2020 CA1","tls_issuer_org":["Acme Inc"],"tls_fingerprint_hash_md5":"9ff41ab3d13b2386ad77fa0b1f058f4f","tls_fingerprint_hash_sha1":"9dc28cb216b46bee01eeaeb5d4ff5906bbba582b","tls_fingerprint_hash_sha256":"324db4e227d9b5fb5dc175d7b4bb984705b9f4ec07a36bfe2fd4df79a965961e","tls_wildcard_certificate":true,"tls_connection":"ctls","tls_sni":"dashboard.example.com","headers":["Cf-Cache-Status: DYNAMIC","Server: cloudflare"]}

• subdomains: List of discovered subdomains.

foo.example.com
bar.example.com
baz.example.com

• subdomain-details: JSONLines records of subdomain discovery details.

{"hostname": "foo.example.com", "domain_name": "example.com", "data_source": "http response", "context": "http redirecrt", "linked_asset": "http://1.2.3.4"}
{"hostname": "bar.example.com", "domain_name": "example.com", "data_source": "http response", "context": "content security policy (csp)", "linked_asset": "http://5.6.7.10:8080"}
{"hostname": "baz.example.com", "domain_name": "example.com", "data_source": "tls certificate", "context": "tls subject alternative name (san)", "linked_asset": "https://dashboard.example.com"}

• potential-hostnames: List of related hostnames outside the strict scope.

foo.example-internal.com
bar.example-subsidiary.com
baz.example-service-provider.net

• potential-hostname-details: JSONLines records of potential hostname discovery details.

{"hostname": "foo.example-internal.com", "domain_name": "example.com", "data_source": "http response", "context": "http redirecrt", "linked_asset": "http://1.2.3.4"}
{"hostname": "bar.example-subsidiary.com", "domain_name": "example.com", "data_source": "http response", "context": "content security policy (csp)", "linked_asset": "http://5.6.7.10:8080"}
{"hostname": "baz.example-service-provider.net", "domain_name": "example.com", "data_source": "tls certificate", "context": "tls subject alternative name (san)", "linked_asset": "https://dashboard.example.com"}

• subdomain-wildcards: List of discovered subdomain wildcards.

*.internal.example.com

• subdomain-wildcard-details: JSONLines records of subdomain wildcard discovery details.

{"hostname": "*.internal.example.com", "domain_name": "example.com", "data_source": "tls certificate", "context": "tls common name (cn)", "linked_asset": "https://dashboard.example.com"}

• potential-hostname-wildcards: List of related hostnames with wildcards outside the strict scope.

*.proxy.example-corp.com

• potential-hostname-wildcard-details: JSONLines records of potential hostname wildcard discovery details.

{"hostname": "*.proxy.example-corp.com", "domain_name": "example.com", "data_source": "tls certificate", "context": "tls common name (cn)", "linked_asset": "https://dashboard.example.com"}

Note: The *-details outputs may contain duplicates if a hostname was discovered through multiple sources.

Changelog

• v1.0.0
  • Initial release
• v1.1.0
  • Improved handling of cases where a live web server redirects to a non-existent or unreachable host.
  • Enhanced overall detection accuracy with a more advanced retry mechanism.
  • Adjusted port scan logic to scan only ports 80 and 443 for hosts behind cloud WAFs or CDNs.
  • Added filters to exclude responses with protocol mismatches.
• v1.1.1
  • Preserve existing paths and queries from the input, if present.
• v1.1.2
  • Added header and header-file inputs