HttpStatus service

HttpStatus is a tool that allows users to check HTTP response status codes, headers, and other related information for any URL input. It is used for monitoring, debugging, and testing web services and applications by providing insights into how web servers respond to HTTP requests. The service performs requests on behalf of the user and does not follow links or act as a web crawler.

This documentation serves to inform web administrators, developers, and service users about the behaviour and structure of the HttpStatus User-Agent, ensuring transparency and responsible use of the service.

Standard User-Agent

The most reliable way to identify HttpStatus requests is by examining the User-Agent string. The User-Agent identifies the service making HTTP requests, providing transparency to the server about the origin of the request. For most API requests, HttpStatus sends the following default User-Agent string:

HttpStatus (+https://httpstatus.io/ua)

Components:

1. HttpStatus:
   Identifies the service making the request.
2. (+https://httpstatus.io/ua):
   Provides a link to the official service documentation (what you are reading now) allowing website owners to understand the purpose and behavior of the requests.

Predefined and custom User-Agents

In addition to the default User-Agent string, users of the HttpStatus service can override the User-Agent header to simulate requests from other clients or bots. This is useful for testing how a website responds to different agents, like browsers or search engine crawlers.

Example Custom User-Agents

Simulating Googlebot Smartphone:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.5304.110 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) HttpStatus

Simulating a Chrome Browser:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 HttpStatus

These strings combine the Googlebot or Google Chrome User-Agent with a reference to HttpStatus, indicating that the request originates from HttpStatus but simulates Googlebot or Google Chrome.

IP Addresses

HttpStatus requests originate from specific IP addresses, which are published here for transparency. While you can use these IPs to identify or block HttpStatus traffic, blocking by User-Agent is the recommended approach, as IP addresses may change.

Current HttpStatus API IP Addresses

Verifying HttpStatus Using Reverse DNS (rDNS)

In addition to using the User-Agent string or IP address, you can verify that a request is from HttpStatus by performing a reverse DNS (rDNS) lookup. This technique helps ensure the authenticity of the request.

Steps for Reverse DNS verification using the command line

1. Identify the IP address: when you receive a request, log the originating IP address.
2. Perform a reverse DNS lookup: use host to check the reverse DNS of the IP address using the command line (also known as the CLI, console, terminal or shell).
3. Verify the domain: Ensure that the result matches the domain httpstatus.io. The request is not sent by HttpStatus if the host name has a different ending.
4. Run a forward DNS lookup: Perform a forward DNS lookup on the domain name retrieved in step 2 using the host command on the retrieved domain name.
5. Verify the IP address: Check that the IP address is the same as the original accessing IP address from your logs (step 1).

$ host 49.12.76.189
189.76.12.49.in-addr.arpa domain name pointer ipv4-189-fsn1-dc14.httpstatus.io.

$ host ipv4-189-fsn1-dc14.httpstatus.io
ipv4-189-fsn1-dc14.httpstatus.io has address 49.12.76.189

Lookup the Hostname (PTR Record) using a Reverse DNS Tool

HttpStatus is not a crawler and does not follow links on a web page. It only sends requests to URLs provided by the user, which could be manually entered or automatically fed through user software integrations. Here's what you can expect:

• Single URL checks: HttpStatus only checks the URL you provide and does not crawl additional pages.
• No Robots.txt adherence: Since HttpStatus operates on user-provided URLs and does not crawl the web, it does not adhere to robots.txt rules (this may be subject to change in the future). However, users can still block HttpStatus through firewall rules if needed.
• Respect for server resources: The service is designed to perform lightweight requests that respect server resources and reduce unnecessary load. The service only requests and downloads the HTML source files that are rendered by the server.

Blocking by User-Agent or IP through a web application firewall (WAF) is an option if needed. It's important to note that this refers to blocking at the network or server level, not by using the User-Agent token in the robots.txt file, as HttpStatus does not follow robots.txt rules.

1. Blocking by User-Agent (Recommended)

If you wish to block requests from HttpStatus on your site, you can do so by filtering based on the User-Agent string. Most web servers allow administrators to create rules to block requests based on this string.

2. Blocking by IP

While blocking by IP address is possible, it's less reliable due to the potential for IP changes. Blocking by User-Agent is generally the preferred approach, but if you choose to block by IP, you can use the current list of IPs provided above.

A note on blocking HttpStatus

Blocking HttpStatus requests may be considered if they cause issues with a service. However, it is hoped that this won't be necessary. The service is intended to be useful for legitimate purposes such as monitoring, troubleshooting, and testing web application behavior.

If any issues arise with HttpStatus, support is available to help resolve them. Blocking by User-Agent or IP is an option, although it is preferable to maintain a positive, cooperative approach that benefits both the service and its users.

While this documentation is specific to the HttpStatus API, future updates may align the behavior of the web interface/web tool with the API, potentially unifying the User-Agent structure across the platform.

For any inquiries or issues regarding the HttpStatus service, or to learn more about its behaviour and how it impacts your site, please refer to this documentation.

For specific questions or support requests, contact support@httpstatus.io.

Last updated: October 5, 2024