Apache Logs and Analyzing Suspicious Traffic
When managing a server, you often need to look into suspicious traffic on the web server. If it runs Apache, the following tips can help with the analysis
In this article
Every now and then, one of our Apache servers shows unusual spikes in the number of requests, with a sharp increase in load. That is when log analysis comes in, so we can find the cause and respond to any "scrapers" that have gone off the rails. We use several utilities for this and combine them as needed – e.g. goaccess, apachetop, iftop, server status, etc.
What has worked best for me, though, is chaining a few basic Linux commands that let me quickly parse and process a log so that it instantly shows suspicious IP addresses from one or more logs at once, for any time window and any URL.
IP addresses with the most requests
The following chain of commands parses all Apache logs in a directory and shows the 20 IP addresses that accessed the server most often on the current day (in this case August 22, 2015) between 13:00 and 13:59:59. If you need to go through a day other than today, you have to adjust the file pattern, because those files already have a number at the end indicating how old they are – e.g. access.log.2, etc.
cat *-access.log | grep "22/Aug/2015:13" | awk '{ print $1 }' | awk -F "." '{ print $1 "." $2 "." $3 "." $4}' | sort | uniq -c | sort -n | tail -n 20The command could certainly be written more simply, but this form makes it easy to adapt for filtering whole network segments. If you run it as it is, you may get output similar to this:
242 195.250.131.154
242 195.250.131.167
251 195.250.131.133
251 195.250.131.169
272 195.250.131.185
279 77.48.246.241
280 195.250.131.150
302 195.250.131.171
321 178.77.233.149
330 94.231.9.72
334 195.250.131.149
345 84.42.146.94
351 207.46.13.24
355 157.55.39.199
383 207.46.13.55
411 195.250.131.172
535 77.75.76.167
949 195.250.131.130
49823 127.0.0.1At first glance, no external IP address generates noticeably more traffic than the others. But if we modify the command to group /24 segments together (i.e. we ignore the last part of the IP address):
cat *-access.log | grep "22/Aug/2015:13" | awk '{ print $1 }' | awk -F "." '{ print $1 "." $2 "." $3}' | sort | uniq -c | sort -n | tail -n 20one significantly larger source of traffic immediately shows up in the new results:
235 109.81.210
236 212.79.106
242 37.48.34
247 109.81.209
262 78.108.103
279 77.48.246
282 109.81.211
316 66.249.78
321 178.77.233
330 94.231.9
334 109.81.208
342 194.228.20
345 84.42.146
347 194.228.13
483 157.55.39
535 77.75.76
816 207.46.13
9448 195.250.131
49823 127.0.0We got to this result within moments, and now we can check what exactly was being downloaded and how systematically. Our response then depends on the circumstances and on whatever else we find out.
Most frequently requested URLs
The command chain can also easily be adapted to show a list of the most frequently requested URLs. So if we want to see, for example, which URLs were requested most often from the suspicious segment we found in the previous step, we can do it simply like this:
cat host-access.log | grep "195.250.131" | grep "22/Aug/2015:13" | awk '{ print $7 }' | sort | uniq -c | sort -n | tail -n 20And the result may look something like this:
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_pronajem&page=4
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_pronajem&page=5
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_pronajem&page=6
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_spolubydleni&page=2
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_spolubydleni&page=3
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_spolubydleni&page=4
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_spolubydleni&page=5
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_offer_spolubydleni&page=6
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_request_spolubydleni&page=2
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_request_spolubydleni&page=3
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_request_spolubydleni&page=4
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_request_spolubydleni&page=5
42 /vypis/nabidka-prodej/nebytovy-prostor?aType=estate_request_spolubydleni&page=6
42 /vypis/nabidka-prodej/rekreacni-objekt?aType=estate_offer_prodej&page=6
46 /vypis?aType=estate_request_spolubydleni&estatetype=kancelar
47 /vypis?aType=estate_offer_prodej&estatetype=kancelar
47 /vypis?aType=estate_offer_pronajem&estatetype=kancelar
47 /vypis?aType=estate_offer_spolubydleni&estatetype=kancelar
47 /vypis?aType=estate_request_prodej&estatetype=kancelar
47 /vypis?aType=estate_request_pronajem&estatetype=kancelarOnce you understand the individual parts, you can easily modify the whole command and get a very powerful helper for working with logs. Within a second, you can filter out the information that matters most to you at that moment from gigabytes of logs containing millions of entries.