Show only specific columns in Linux log files

To troubleshoot specific application errors, a sysadmin needs to check his logs for warning, errors or other useful information.
However, most server logs are quite unreadable or contain too much information. This kind of situation is unacceptable and you need to extract only specific columns from that log.
Let’s say you are checking the Apache error log and you want to extract only a specific column. I’m trying to search the log for the word error, the display the IP’s that are getting that error and count how many times the error occurs per IP.
Here’s how you can do it.

First you need to find out where the Apache log file is. In my case it is /etc/httpd/logs/error_log (Redhat, CenOS). If you use Debian or Ubuntu the location will be /var/log/apache2/error.log.

Now that you know the log location, put the path in the command below:

cat /etc/httpd/logs/error_log | grep error | cut -d" " -f8 | uniq -c | sort

You should see something like this:

[root@nyx ~]# cat /etc/httpd/logs/error_log | grep error | cut -d" " -f8 | uniq -c | sort
      1 109.166.141.29]
      1 111.162.148.116]
      1 118.113.227.137]
      1 166.78.10.25]
      1 187.33.2.88]
      1 207.46.13.77]
      1 66.249.78.221]
      1 66.249.78.3]
      6 61.19.246.190]
      6 95.111.68.120]

So, the command syntax is pretty simple: I concatenate the log file, match the “error” word, then print the 8th column, remove duplicates, count the occurrence per IP and sort the list. Voila!

Now you know which IPs are receiving errors when they are visiting your site. You might want to do some checks and see what the problem is. Maybe those IPs are just running bots that are searching for vulnerabilities and it might be a good idea to block them in your firewall.