http Archives - Networking HowTos

Change the User Agent in Wget

June 3, 2017 Linux

The Wget application is a non-interactive tool to download data from a server using either the HTTP, HTTPS or FTP protocols. This article will focus on the HTTP and HTTPS protocols, as the user agent value has no relevance with the FTP protocol.
There will be times when you need to download a web page, emulating a different browser. Some web sites may output different html code depending on the browser being used to request the page.
By modifying the User Agent header on the http/https request, you can tell the server what browser you are using.
Wget has the ability to modify the user agent value easily, using the “-a” parameter as shown in the example below.
Modifying the User Agent in Wget:

$ wget -U "User Agent Here" "http://website.url.here"

The example below will emulate Internet Explorer 6 requesting the page from www.google.com

$ wget -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://www.google.com"

$ wget --user-agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://www.google.com"

If the web page is designed to show different output depending on the browser, the above example would be show the output as though you were connecting to the web server from Internet Explorer 6.
For a list of some of the common user agent strings, please click here.

Change the User Agent in Curl

June 3, 2017 Linux

The Curl application is a tool to transfer data from a server using a number of different protocols. The main protocols that are generally used are HTTP and HTTPS.
There will be times when you need to download a web page, emulating a different browser. Some web sites may output different html code depending on the browser being used to request the page.
By modifying the User Agent header on the http/https request, you can tell the server what browser you are using.
Curl has the ability to modify the user agent value easily, using the “-a” parameter as shown in the example below.
Modifying the User Agent in Curl:

$ curl -A "User Agent Here" "http://website.url.here"

The example below will emulate Internet Explorer 6 requesting the page from www.google.com

$ curl -A "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://www.google.com"

Remove nginx version from HTTP response headers

July 11, 2012 Linux

Its usually a good idea to prevent users from being able to determine the version number of the web server software you are using. This helps prevent people from then going and looking for known vulnerabilities in that specific version, and attacking your site.
When a HTTP response comes back to the web browser, it also includes a number of headers. One of these headers is the “Server” header. This is used to identify the web server software being used.
eg:

curl -I http://yourwebserver

Sample output:

HTTP/1.1 200 OK
Server: nginx/1.1.19

To remove the version number from this response when using nginx, edit the /etc/nginx/nginx.conf file and add the following line into the “http” section:

server_tokens off;

If it already exists but is set to “on”, change it to “off” instead of adding the new line.
Restart nginx.
Now when you run the ‘curl’ command as per the example above, and you should get an output like this:

HTTP/1.1 200 OK
Server: nginx

Note: Make sure you keep up to date on software updates.