OSINT (open source intelligence) means collecting and analysing information that anyone can lawfully access. Applied to a domain or website, it means reading registration data (WHOIS and RDAP), DNS records, Certificate Transparency logs, IP and network ownership, the site's technology stack and its archived history — enough to judge who runs a site and whether to trust it.
What OSINT means for websites
"Open source" here refers to the source of information, not to open-source software. The discipline comes from intelligence work, where anything obtained without covert means — press, public registries, databases, the web — counts as open. For a website, OSINT is a simple habit: don't take a site at its word, compare what it says about itself with what independent systems record about it. Registries, resolvers, certificate authorities and web archives have no reason to flatter anyone.
Every investigation follows the same loop: ask a precise question ("is this the brand's real shop or a clone?"), collect data, confirm each finding from a second source, then conclude. Skipping confirmation is the classic beginner's mistake — a single WHOIS field or one IP address proves very little on its own.
Two modes are worth keeping apart:
- Passive OSINT reads third-party data — registry records, CT logs, public resolvers, archives — without touching the target's servers. It is fine to run against any site.
- Active reconnaissance — port scans, subdomain brute-forcing, zone transfer attempts — interacts with someone else's infrastructure. Reserve it for systems you own or have written permission to test.
Legitimate reasons to investigate a domain
- Vetting a supplier or online shop. A vendor claiming "trusted since 2010" on a domain registered last week deserves a pause before you pay.
- Catching phishing and brand look-alikes. Attackers register similar names and obtain certificates for them; certificate logs often expose those domains before the campaign starts.
- Auditing your own attack surface. Attackers see your domain exactly the way OSINT does. Forgotten subdomains running old software, CNAMEs pointing at deleted cloud resources (a subdomain takeover risk) and chatty TXT records are better found by you.
- Due diligence before buying a domain. Check what the name hosted before: pharma spam, gambling, malware. Reputation travels with the name.
- Incident response. Find where a malicious page is hosted and whom to send the abuse report to.
What you can learn about a website from open sources
WHOIS and RDAP: registrar, dates, name servers
Even when the registrant is redacted — which is now the norm for gTLDs such as .com since GDPR — the record still shows the registrar, creation and expiry dates, name servers and status codes. RDAP is the structured successor to WHOIS: the same data as JSON with a defined schema (RFC 9083), and ICANN has made it the primary source of registration data for generic TLDs.
# Classic WHOIS (Linux/macOS)
whois example.com
# RDAP via the rdap.org bootstrap redirector (needs curl and jq)
curl -sL https://rdap.org/domain/example.com | jq '.events, .nameservers[].ldhName'Look at the creation date first (a young domain behind an "established company" is a red flag), then expiry (a lapsed domain may have changed hands) and the name servers, which reveal whether DNS sits with the registrar, a host or a CDN. Field-by-field reading is covered in our WHOIS lookup guide.
DNS records: mail, services, infrastructure
DNS answers anyone who asks. MX shows the mail provider; TXT records leak integrations — an SPF include:_spf.google.com points to Google Workspace, include:spf.protection.outlook.com to Microsoft 365, and verification strings such as google-site-verification show which consoles the owner has claimed.
# Linux/macOS
dig +short NS example.com
dig +short MX example.com
dig +short TXT example.com
# Windows PowerShell
Resolve-DnsName example.com -Type TXT
# Windows cmd
nslookup -type=mx example.comOn your own domain, confirm that name servers refuse full zone transfers: dig axfr example.com @ns1.example.com should end with Transfer failed. Getting the whole zone back means anyone can map your infrastructure. Only test servers you control.
Certificate Transparency: subdomains from crt.sh
Publicly trusted TLS certificates are recorded in append-only Certificate Transparency logs (RFC 6962), because browsers reject certificates that lack proof of logging. The result is a public history of every hostname ever certified — including internal ones like vpn., jira. or staging.. The easiest way in is crt.sh, where % is a wildcard (URL-encoded as %25):
curl -s "https://crt.sh/?q=%25.example.com&output=json" \
| jq -r '.[].name_value' | sort -uTwo caveats: a log entry proves a name was certified, not that it resolves today, so re-check each hit in DNS; and wildcard certificates hide individual hostnames. Searching for %yourbrand% surfaces look-alike domains carrying your brand. More on monitoring in Certificate Transparency logs explained. To read a live certificate (OpenSSL 1.1.1+):
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates -ext subjectAltNameIP address, hosting provider and ASN
The A record gives the IP; the regional internet registry (ARIN, RIPE NCC, APNIC and others) tells you who holds the block, and routing data gives the autonomous system number.
dig +short A example.com
whois 203.0.113.10 | grep -iE 'netname|orgname|org-name|country|abuse'
# IP to ASN via Team Cymru
whois -h whois.cymru.com " -v 203.0.113.10"
# Reverse DNS (PTR)
dig +short -x 203.0.113.10The main trap is CDNs and reverse proxies: behind Cloudflare, Akamai or Fastly you see the edge network, not the origin host. The ASN usually makes that obvious. Our guide on finding a website's host and IP covers the next steps.
Reverse IP: who shares the server
A reverse IP lookup lists other domains on the same address. On shared hosting that means hundreds of unrelated sites and no conclusion to draw. On a dedicated IP, neighbours often share an owner — a cluster of near-identical domains is a typical footprint of a fake-store network.
Technology stack
Response headers and markup reveal the platform: Server, X-Powered-By, cookie names such as PHPSESSID or wordpress_*, the generator meta tag, asset paths. An outdated CMS on your own site is today's to-do item.
curl -sI https://example.com | grep -iE 'server|x-powered-by|set-cookie'History: the Wayback Machine
The web archive answers what the site itself never will: what the domain hosted before, when the "company history" appeared, whether the name expired and was re-registered. Its CDX API lists captures:
curl -s "https://web.archive.org/cdx/search/cdx?url=example.com&output=json&limit=10"See how to use the Wayback Machine for deeper techniques.
Question, source, tool: a quick map
| Question | Open source | What to look at | Tool |
|---|---|---|---|
| Who is the registrar, how old is the domain | WHOIS / RDAP | Creation, expiry, name servers, status | WHOIS lookup |
| Where does mail go, which services are connected | DNS (MX, TXT, NS) | SPF includes, verification records | DNS lookup |
| Which subdomains exist | Certificate Transparency logs | SAN names, issue dates | crt.sh (external) |
| Who issued the certificate, when it expires | The certificate itself | Issuer, validity, hostnames | SSL checker |
| Where the site is hosted | DNS + IP registry | IP, country, provider, abuse contact | IP lookup |
| Whose network is it | BGP routing data | ASN, operator name | ASN lookup |
| What else lives on this IP | Passive DNS datasets | Neighbouring domains | Reverse IP |
| What the site is built with | HTTP headers and HTML | CMS, framework, analytics, CDN | Tech detector |
| What the domain hosted before | Web archive | Snapshots over the years | web.archive.org (external) |
How to check a website: a practical sequence
- Run a WHOIS lookup: creation date, registrar, expiry, name servers. Compare the domain's age with the story the site tells.
- Pull records with a DNS lookup: A, MX, TXT, NS. Does the domain have mail? Do the name servers match the registry record?
- Resolve the IP, identify the provider and ASN, and run a reverse IP lookup. Ignore neighbours on CDN addresses.
- Inspect the certificate and search the domain and brand name on crt.sh.
- Check the web archive for how long the current site has existed.
- Write down each conclusion with its source. Anything you cannot confirm twice is a hypothesis.
Reading the signals
| Signal | Possible meaning | How to confirm |
|---|---|---|
| Domain a few days old, certificate issued yesterday | Throwaway campaign site or phishing | Brand search on crt.sh, archive history |
Brand name in a subdomain of an unrelated domain (brand.secure-login.example) | Classic phishing pattern | WHOIS of the parent domain, IP reputation |
| Dozens of similar domains on one dedicated IP | Network of sites run by one operator | Registration dates, shared name servers |
| Archive shows a different site until last year | Expired domain bought for its history | Creation date versus last registration |
No single signal is a verdict; honest start-ups have young domains too. Weigh them together — our checklist on how to spot a fraudulent website helps.
OSINT Framework and other OSINT tools
OSINT Framework is not software but a clickable tree of resources grouped by data type — domain, IP, email, images, social networks. Markers next to entries describe the resource: (T) needs a local install, (D) is a search-engine dork, (R) requires registration, (M) is a URL you edit manually. Links age, so check that a service still works.
For domain work the command-line kit is small: whois, dig or nslookup, openssl, curl and jq. For bulk passive subdomain collection there are open tools such as subfinder (subfinder -d example.com -silent) and OWASP Amass; some of their data sources need API keys. Username hunters like Sherlock target people rather than infrastructure and are outside the scope of this guide.
Ethics and the law
- Personal data. A person's name, email or phone number is still personal data when it is public. In the EU and UK, building profiles of individuals requires a lawful basis under the GDPR. Investigate domains and servers, not people.
- Breached data. "Lookup" services built on leaked databases are not OSINT, and using them is unlawful in many jurisdictions.
- Active testing. Scanning, password guessing or exploiting someone else's systems without authorisation can breach computer misuse laws such as the US Computer Fraud and Abuse Act or the UK Computer Misuse Act 1990. On your own perimeter it is simply an audit.
FAQ
What is OSINT in simple terms?
Gathering and analysing information from lawful public sources — registries, DNS, certificate logs, archives, the press. For websites, it means checking a domain against independent records instead of trusting what the site claims.
Is OSINT legal?
Passive research on public infrastructure data is legal in most countries. Problems start with profiling individuals, using breached data or actively probing systems you are not authorised to test.
How do I find all subdomains of a website?
No open source gives a complete list, but Certificate Transparency logs via crt.sh reveal most names that ever had a certificate. Passive DNS datasets and tools like subfinder add more. Verify every hit in DNS, since logs keep long-deleted names.
What is OSINT Framework used for?
It is a free directory of OSINT resources organised as a tree by data type. It shows where to look but checks nothing itself; for a domain, WHOIS, DNS, IP and certificate data cover most questions.
Why does a site's IP point to Cloudflare instead of its host?
The site sits behind a CDN or reverse proxy, so only the edge network is visible. Use registration data, name servers, MX records and the domain's history instead.
