Splunk Project

By Jacob Taylor - Jan. 10, 2023

Environment Setup

To establish a SIEM (Security information and event management) environment, 3 virtual machines were created, via Linode, using variations of Linux OS.

The first machine is a Jumpbox which is accessed via SSH for a more secure, controlled access to then SSH into the Webserver, or Splunk Server.

Inside the Webserver, important services running are Apache2, MariaDB, and SSH. This allows for public access to a fresh install of a publicly accessible WordPress site and remote access to the server via Secure Socket Shell (SSH). Furthermore, the Webserver is configured with a Splunk Forwarder to send logs to the Splunk Server to then be indexed and monitored in real time.

The Splunk Server has a control panel for Splunk accessible via a web browser using the server's IP and port 8000 as well as a listener on port 9997 to gather the logs being sent by the Forwarder/Webserver.

Network Diagram

Below is the public facing WordPress site, followed by the customized Splunk dashboard:

WordPress

Splunk Dashboard

Monitoring Pings

Ping DDOS Visualization

To be able to see when the Webserver was pinged via a log file, the following command was ran from the Webserver:

iptables -I INPUT -p icmp --icmp-type 8 -m state --state NEW,ESTABLISHED,RELATED -j LOG --log-lev el=1 --log-prefix "Ping Request"

The pings are now stamped in the kern log, which is the source for the created Splunk search and alert:

Email Alert

To simulate a DDOS (Denial of Service) Attack, the following script was created and ran on local machines to attack the Webserver:

#!/bin/bash
#by @jacvbtaylor

RED="\e[31m"
GREEN="\e[32m"
ENDCOLOR="\e[0m"
 
printf "input the ip, then press enter\n\n"
read IP \
&& ping -c 1 ${IP}
printf "${GREEN}\n\n\n[x]Host is up[x]\n\n\n${ENDCOLOR}" && echo "_______________________________" &&\
ping -c 5 ${IP} && printf "${RED}\n\n~~~~STARTING ATTACK~~~~\n\n${ENDCOLOR}" \
&& ping -c 13 ${IP} && printf "${GREEN}\n\n\n~~~~RESTING BEFORE ANOTHER ATTACK~~~~\n\n\n${ENDCOLOR}" &&\
sleep 20 && ping -c 20 ${IP} && sleep 10 && ping -c 5 ${IP}\
&& sleep 10 && printf "${RED}\n\n\n[x]FINAL ATTACK[x]\n ${ENDCOLOR}"\
&& ping -c 30 ${IP}

Ping Attack

While this attack from 1-2 machines wasn't enough to exhaust the server's resources and availability, it could be used within a botnet (a much larger cluster of slave machines) to bring service down. Monitoring this kind of traffic can be beneficial to detect a possible DDOS attack with hopes of catching and stopping the malicious traffic before server takedown occurs.

WordPress Bruteforce

To create an alert and monitor control panel logins from threat actors, the following query was created using SPL (Splunk Processing Language):

Detect WP Login Locations

Detect URI Path

Analyzing this traffic will help with baselining, or determining normal activity thresholds. When URI requests are abnormally high such as underlined in the image below, it is easy to infer that the traffic is malicious. Although false positives and alert fatigue can occur, knowing the threshold for certain endpoints such as a login will make those less likely to affect the monitoring and incident response time.

Brute Force Detection

To simulate a brute force (a password cracking method) attack on the WordPress login hosted on the Webserver, a local machine was used along with a tool called Hydra. The Hydra syntax for WordPress that was used is as follows:

hydra -l <wordlist> -P <wordlist> <ip> http-post-form '/wp-login.php:log=^USER^&pwd=^PASS^&wp-submit=Log+In:incorrect' -vV -t 30

wp-login.php bruteforce

Depending on the wordlist used, the amount of POST requests sent to the webserver will vary. This is important to note as some brute forcing will be very obvious while others are much more discrete. In this case, the bruteforce is extremely noisy and the URI path count skyrocketed very quickly, making it easy to understand there was an attack taking place.

Stopping Attacks

The most straightforward way stop an active attack without turning the services off on the server is to block the malicious traffic that has been detected and analyzed. Below is the script called block.sh that was created and used in this instance:

block.sh

To use the block.sh script, run the executable then input the IP to block and it automates the block:

Although the script does output the iptables error, it still adds the IP(s) to the block last, which is shown in a later image. Removing service iptables save should get rid of the output message and finish with the grep output.

Stopping A Real Attack

After analyzing more logs on the Webserver, an SSH bruteforce was discovered. This occurred before any alerts and real-time monitoring were set up, causing the attack to last much longer than it should have. This Webserver had been running for less than 7 days and was already in danger, as the graph below demonstrates:

SSH BRUTEFORCE

An obvious attack transpired from an IP range located in China, lasting from Sunday, Jan. 8 12 AM - 10 PM and reoccurred Monday Jan. 9 10 PM - Tuesday Jan. 10 10 PM. The attack was persistent but no successful logins had been logged.

The biggest gap in the attack occurred when the attack was discovered and the firewall was enabled to block SSH traffic while this project was not being worked on. The firewall was later lifted and the striking took place again before being stopped for good.

China Attack

To conclude the attack from the Chinese IP range, the block.sh script was used:

Block range

Further investigation can now take place to ensure a breach or foothold did not take place in the backend of the Webserver.

Conclusion

This project demonstrated the reality of cyber attacks as many different IP's found their way to the Webserver, even though it had never been publicly shared after being created. Taking this experience of a brand new server, or website, reflects tremendously on the dangers and impact a well known server must face on a daily basis.

Had the Webserver truly been compromised, lateral movement from an adversary could not take place and the one IP logged to access the Webserver's backend would be the private IP of the Jumpbox. This keeps those accessing the servers through the Jumpbox safe as their true IP's are not disclosed and a new Jumpbox can always be established.

Composing the environments to communicate properly came with many challenges but were resolved in the end with patience and an eagerness to learn through navigating error logs and trying new steps as well as reusing skills from past experiences. The challenges resulting from this project provided experience and showcases capability to configure and manage security tools that come from a Security Operations Center role as an Analyst.

Splunk Setup

Using SSH via the webserver, install the Splunk forwarder with the following command:

wget -O splunkforwarder-9.0.3-dd0128b1f8cd-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.0.3/linux/splunkforwarder-9.0.3-dd0128b1f8cd-linux-2.6-amd64.deb"

Followed by apt install ./splunkforwarder-9.0.3-dd0128b1f8cd-linux-2.6-amd64.deb

In order to receive the logs, ownership change of the logs to root had to be done by running chown -R root:root /var/log/apache2

Install Command History

To setup Splunk manually on a different server, run the following script:

wget -O splunk-9.0.3-dd0128b1f8cd-linux-2.6-amd64.deb "https://download.splunk.com/products/splunk/releases/9.0.3/linux/splunk-9.0.3-dd0128b1f8cd-linux-2.6-amd64.deb" && dpkg -i splunk-9.0.3-dd0128b1f8cd-linux-2.6-amd64.deb && dpkg --status splunk && echo "/opt/splunk/bin/splunk start" > startspunk.sh && chmod +x start* && ./startspunk.sh