How to get website information using the Domain API

Recently we launched a new offering called the Domain API which allows you to easily get data about websites and their subdomains. With one simple query you can get information about the requested domain, including:

• Primary Categories
• Primary language used on that domain
• IP addresses of the hosting servers
• Known subdomains

Today, we are bringing you a quick and easy to use guide showing how to implement this new API in various use cases.

Scenario #1 – Find related subdomains

You are a security researcher who needs to discover all subdomains as part of hacking reconnaissance. You are attempting to discover if there are any unsecured subdomains that could be potentially used as breaching points so that you can fix them.

To get all of the known subdomains for a given domain name, you can use our API and you will get a JSON response containing the necessary information.

Scenario #2 – Discovering other sites on a shared hosting server

You are a technical SEO marketer and you have a client that’s running a website on a shared hosting server. For some reason, even if you do everything needed to optimize your website for search engines, you can’t seem to get good search rankings. One potential reason for this can be due to having other “spam” related websites hosted on the same hosting server. Since the sites on a shared host use the same IP addresses, if there is a “bad site” it can cause problems with your site due to that association. In order to investigate if that is the case you need to find out what other websites are being hosted on the shared IP address.

By sending the IP address to our Domain API, you can get the other domains that are hosted on that address (and therefore on the same shared hosting server.)

We have prepared four simple steps below that demonstrate how to make queries to the Domain API service.

Step 1 – Generate API access keys

Access keys allow you to connect to our services without sharing your username and password or other sensitive information in your application.

To generate API keys:

1. You need to sign-in to your account dashboard (https://dashboard.webshrinker.com).
2. Navigate to the “API Access Keys” area and select “Create API Key”
3. Add a description to your key and make sure that the “Hosts” service is selected

Step 2 – Send the Domain lookup request

Like in our previous articles, we are going to use HTTP basic authentication to make the API requests using the Python “requests” library.

Make sure you have your access key and secret key and substitute them into the sample code. Your next step is to send the request to our server.

import requests
import json
from base64 import urlsafe_b64encode

target_website = "<the domain name/IP address of the site to retrieve the information about>"

key = "<insert your API key>"
secret_key = "<insert your API secret key>"

api_url = "https://api.webshrinker.com/hosts/v2/%s" % urlsafe_b64encode(target_website)

response = requests.get(api_url, auth=(key, secret_key))

Depending on the provided domain name or IP address the response will contain different information. If you run a query for a domain name, you will get a list of all known subdomains and the primary categories for that domain. If you request an IP address, you will get a list of the domains known to be hosted on that address, along with the most relevant categories of the websites hosted on that IP address.

Step 3 – Parsing the response

Like previously mentioned, the response is returned in JSON format which makes it easy to extract the data that you need.

We are using json() function to parse the response and pass it to the application.

status_code = response.status_code
data = response.json()

Here is an example JSON response when querying for the domain “webshrinker.com”:

{
   "data": [
       {
           "addresses": {
               "ipv4": {
                   "104.25.182.29": [
                       "2016-08-31T00:00:00Z"
                   ],
                   "104.25.183.29": [
                       "2016-10-25T15:27:58Z"
                   ]
               }
           },
           "categories": ["informationtech", "business"],
           "end_date": "2016-11-01",
           "host": "webshrinker.com",
           "language": "en",
           "related": [
               "api.webshrinker.com",
               "docs.webshrinker.com"
           ],
           "start_date": "2016-08-01"
       }
   ]
}

Step 4 – Handling status codes

To ensure proper functioning of your code, you need to use HTTP status codes which return the status of your request.

if status_code == 200:
   # Do something with the JSON response
   print json.dumps(data, indent=4, sort_keys=True)
elif status_code == 400:
   # Bad or malformed HTTP request
   print "Bad or malformed HTTP request"
   print data
elif status_code == 401:
   # Unauthorized
   print "Unauthorized - check your access and secret key permissions"
   print data
elif status_code == 402:
   # Request limit reached
   print "Account request limit reached"
   print data
else:
   # General error occurred
   print "A general error occurred, try the request again"

• If your request was completed successfully, the request will return status code 200.
• Having a badly formed HTTP request will return status code 400. In this case, you need to check your code to make that you are sending a proper API request.
• In case you entered a bad secret or access API key, the HTTP status code will be 401. Please check your code and enter proper API keys.
• If you reached the limit of API requests on your account, the system will return status code 402.

Interested in trying it yourself?

Sign up now, we are offering a free plan that allows you to get started right away with any of our API services.

Create my free account