Linux Splunk

Authentication Logs

Authentication logs are one of the most important sources when analysing Linux systems in a SIEM. They help you understand how users interact with the system, especially regarding login attempts and privilege usage.

Unusual login activities Let’s move on to real scenarios and examine a situation in the SOC. As an SOC L1 analyst, you have received an alert about an unusual SSH login to the ubuntu username on the system. In this case, let’s write a query to search for successful and failed login attempts to the system for the ubuntu user.

index=linux source="auth.log" *ubuntu* process=sshd 
| search "Accepted password" OR "Failed password"

Or

index=task5 source="auth.log" process=sshd "jack-brown"
| search "Failed Password for"

Or

index="linux-alert" sourcetype="linux_secure" 10.10.242.248 
| search "Accepted password for" OR "Failed password for" OR "Invalid user"
| sort + _time

There were 97 events, and the last ones show successful login attempts to the ubuntu user. This can likely be classified as a successful brute-force attack - an activity that should be escalated to L2. From a learning perspective, let’s take a look at what happened .

• index="linux-alert": Splunk stores data in different databases called "indexes." This tells Splunk to only look in the index named linux-alert.

• sourcetype="linux_secure": This narrows the search down to a specific type of log file. In Linux, linux_secure usually refers to the /var/log/secure or /var/log/auth.log files, which record security and authentication events (like logins).

• 10.10.242.248: This is a simple keyword search. Out of all the Linux secure logs, it will only grab the events that contain this exact IP address. This could be the IP of the server being attacked, or the IP of the person/machine trying to log in.

Number of login attempts made for each user

index="linux-alert" sourcetype="linux_secure" 10.10.242.248
| rex field=_raw "^\d{4}-\d{2}-\d{2}T[^\s]+\s+(?<log_hostname>\S+)"
| rex field=_raw "sshd\[\d+\]:\s*(?<action>Failed|Accepted)\s+\S+\s+for(?: invalid user)? (?<username>\S+) from (?<src_ip>\d{1,3}(?:\.\d{1,3}){3})"
| eval process="sshd"
| stats count values(src_ip) as src_ip values(log_hostname) as hostname values(process) as process by username

Privilege Escalation behaviours Attackers often need root access on a system to gain access to certain files and more. Let’s see if, in our case, there are any signs of such activities.

index=linux source="auth.log" *su* | sort + _time

From the image below, it is clear that the attacker managed to gain access to the root account. How exactly this was achieved cannot be determined from auth.log alone; additional logs would be needed for that.

System Logs

System logs are another essential source of visibility. They capture events related to service activity, system restarts, and crons, all useful for identifying unusual system behaviour often linked to persistence or privilege escalation attempts. Persistence Mechanisms We hope you still remember our case in this task about the threat actor who successfully switched from ubuntu to root. In this scenario, syslog can also be useful, specifically for searching for activity related to persistence through cron jobs or services.

index=linux sourcetype=syslog ("CRON" OR "cron") | search ("python" OR "perl" OR "ruby" OR ".sh" OR "bash" OR "nc")

We detected three interesting events. First, we can see that a suspicious pnr5433sw.sh file from the /tmp folder is being executed via cron every 5 minutes. Next, there are clear signs of a Perl reverse shell attempting to establish a connection to 10.10.101.12 IP on port 9999.

As mentioned earlier, in addition to the core data sources, it's very common to see tools like auditd and osquery used in real-world environments. These tools fall outside the scope of this room, so if you're interested in learning more about them, feel free to explore other related rooms available on our platform.

Escalated their privileges to root

index=task5 source="auth.log" 
| search (sudo OR su OR "root") AND ("COMMAND" OR "session opened" OR "authentication")
| table _time, user, _raw
| sort _time

Account Creation

index=task5 | search "Account Created" OR "new user" OR "useradd" OR "account added"
| table _time, _raw
| sort _time

or

index=linux-alert sourcetype=linux_secure "adduser"