In this blog post we look at how we can use Python to scan the HTTP response headers for some common security misconfigurations.
The OWASP Top 10 project ranks security misconfiguration as the 6th highest web application security risk.
When an internet browser interacts with a webpage, it uses the Hypertext Transfer Protocol (HTTP) to transmit data between server and client. This involves multiple request and response messages for different webpage elements before the full page is rendered in your browser.
Each request and response contains headers, which carry information about the message sender, for example browser information (client request) or server information (server response).
Headers consist of key-value pairs, transmitted in plain text and can easily be inspected by using the developer tools on your browser of choice. For a full list of HTTP headers, refer to this Wikipedia page.
Response headers in particular can be useful to learn more about certain configurations on a web server and may present potential insight into weaknesses which can be exploited by an attacker.
Creating a python ScanHeaders class
With the assumption that Python and the requests library are already installed, we are ready to begin.
A class called
ScanHeaders will be used to hold an instance of the target url and the headers and cookies returned from the web server. We begin with importing requests and creating the class with constructor method.Next, we create a response object to store the response from the webserver after issuing a get request. The response will include the response headers, as well as the cookies from the server, for which we add separate variables within the constructor. The created class and completed constructor should look as follows:
import requests class ScanHeaders: def __init__(self, url): self.url = url response = requests.get(self.url) self.headers = response.headers self.cookies = response.cookies
Separate class methods will be created for each header that we wish to scan, with a failure being printed to the terminal where the header either doesn’t exist (
KeyError) or is not configured correctly.We begin with the X-XSS-Protection header, which is used to enable the browser’s built in filter for detecting cross site scripting. If an appropriate Content-Security-Policy is present (see below), the X-XSS-Protection can be set to ‘0’ (disabled), otherwise the most appropriate value is ‘1; mode=block’, which will prevent the page from loading if an attack is detected. To scan for this header, we create the following class method:
def scan_xxss(self):
"""config failure if X-XSS-Protection header is not present"""
try:
if self.headers["X-XSS-Protection"]:
print("[+]", "X-XSS-Protection", ':', "pass")
except KeyError:
print("[-]", "X-XSS-Protection header not present", ':', "fail!")
The X-Content-Type-Options header is used to restrict MIME types to those as indicated in the Content-Type header and has only one valid value of ‘nosniff’. To test for this configuration, we create a method as follows:
def scan_nosniff(self):
"""X-Content-Type-Options should be set to 'nosniff' """
try:
if self.headers["X-Content-Type-Options"].lower() == "nosniff":
print("[+]", "X-Content-Type-Options", ':', "pass")
else:
print("[-]", "X-Content-Type-Options header not set correctly", ':', "fail!")
except KeyError:
print("[-]", "X-Content-Type-Options header not present", ':', "fail!")
Clickjacking attacks take place where a user is tricked into clicking something unexpected, either through misdirection or by using transparent overlays over an area where the user interacts with the web page. This is usually achieved through an iframe which tricks the user into clicking on a link from an external source. Such an attack can be prevented through using the X-Frame-Options header which can either limit the use of iframes to the same origin, or disable them entirely. The recommended values are thus SAMEORIGIN or DENY.
def scan_xframe(self):
"""X-Frame-Options should be set to DENY or SAMEORIGIN"""
try:
if "deny" in self.headers["X-Frame-Options"].lower():
print("[+]", "X-Frame-Options", ':', "pass")
elif "sameorigin" in self.headers["X-Frame-Options"].lower():
print("[+]", "X-Frame-Options", ':', "pass")
else:
print("[-]", "X-Frame-Options header not set correctly", ':', "fail!")
except KeyError:
print("[-]", "X-Frame-Options header not present", ':', "fail!")
One of the most important security features of any website is that communication between client and server should be encrypted and not submitted in plain text. The HTTP Strict-Transport-Security response header (HSTS) informs the browser that communication should be limited to HTTPS and should not take place over HTTP. Failure to correctly implement this header may allow an attacker to execute a Man in the Middle (MitM) attack and hijack traffic between the client and server. The ‘max-age’ parameter should be set to an appropriate value, but there are other parameters such as ‘includeSubDomains’ and ‘preload’, which should also be considered (beyond the scope of this exercise).
def scan_hsts(self):
"""config failure if HSTS header is not present"""
try:
if self.headers["Strict-Transport-Security"]:
print("[+]", "Strict-Transport-Security", ':', "pass")
except KeyError:
print("[-]", "Strict-Transport-Security header not present", ':', "fail!")
The Content-Security-Policy header has various parameters which can be set to assist with reducing the risk of attacks such as cross site scripting and injection. This header is supported by all modern browsers and is recommended to be implemented in all web applications.
def scan_policy(self):
"""config failure if Security Policy header is not present"""
try:
if self.headers["Content-Security-Policy"]:
print("[+]", "Content-Security-Policy", ':', "pass")
except KeyError:
print("[-]", "Content-Security-Policy header not present", ':', "fail!")
The Set-Cookie header is included in the response header for each cookie issued by the webserver. Two of the most important security considerations in the header are the ‘Secure’ and ‘HttpOnly’ attributes. The Secure attribute limits the cookie’s transmission to secure transport methods only, typically HTTPS. The HttpOnly attribute is used to mitigate the risk of cross site scripting attacks, by limiting client-side scripting access to the cookie.
def scan_secure(self, cookie):
"""Set-Cookie header should have the secure attribute set"""
if cookie.secure:
print("[+]", "Secure", ':', "pass")
else:
print("[-]", "Secure attribute not set", ':', "fail!")
def scan_httponly(self, cookie):
"""Set-Cookie header should have the HttpOnly attribute set"""
if cookie.has_nonstandard_attr('httponly') or cookie.has_nonstandard_attr('HttpOnly'):
print("[+]", "HttpOnly", ':', "pass")
else:
print("[-]", "HttpOnly attribute not set", ':', "fail!")
Scan target to assess HTTP header security
With the class methods now defined, we can use the
ScanHeaders class to assess the headers of a target web URL. For the purpose of testing, we will run a local version of the Damn Vulnerable Web Application, an intentionally vulnerable web app for practicing security assessments.First we create an instance of the class in the variable
target and print out the headers.if __name__ == "__main__":
target = ScanHeaders("http://localhost:8000/setup.php")
for key in target.headers:
print(key, ":", target.headers[key])
print()
The headers should look something like the following:
Host : localhost:8000 Date : Sun, 13 Sep 2020 11:29:04 GMT Connection : close X-Powered-By : PHP/7.3.11 Set-Cookie : PHPSESSID=06e4c7aef78f2044795ed6d32c7af129; path=/, PHPSESSID=06e4c7aef78f2044795ed6d32c7af129; path=/; HttpOnly, security=impossible; HttpOnly Pragma : no-cache Cache-Control : no-cache, must-revalidate Content-Type : text/html;charset=utf-8 Expires : Tue, 23 Jun 2009 12:00:00 GMT
Next, we run each of the scan methods against the target to evaluate the headers and also iterate through the list of cookies to scan for the attributes in which we are interested (note indentation continues from previous code block).
target.scan_xxss()
target.scan_nosniff()
target.scan_xframe()
target.scan_hsts()
target.scan_policy()
for cookie in target.cookies:
print("Set-Cookie:")
target.scan_secure(cookie)
target.scan_httponly(cookie)
My scan results were as follows, which is in line with expectations after inspecting the headers printed out (see above):
[-] X-XSS-Protection header not present : fail! [-] X-Content-Type-Options header not present : fail! [-] X-Frame-Options header not present : fail! [-] Strict-Transport-Security header not present : fail! [-] Content-Security-Policy header not present : fail! Set-Cookie: [-] Secure attribute not set : fail! [+] HttpOnly : pass Set-Cookie: [-] Secure attribute not set : fail! [+] HttpOnly : pass
The full code for this project can be found on my Github page.