🧪 Log Analysis & System Debugging Lab

📌 Objective

  • Read system logs like real incidents
  • Identify root causes
  • Debug common system failures

⚙️ Environment

  • OS: Ubuntu
  • Tools: journalctl, df, du, dmesg

🚨 Scenario 1: Service Crash Investigation

Step 1 - Break the service

sudo nano /etc/nginx/nginx.conf

Add invalid line:

invalid_directive;

Restart:

sudo systemctl restart nginx

Step 2 - Investigate

Check status:

systemctl status nginx

Nginx

Failed

Check logs:

sudo journalctl -u nginx -n 50 --no-pager

Journal

Switch to application-level debugging:

sudo nginx -t

nginx_error

Step 3 - Fix conf

Remove bad line “invalid_directive”

Step 4 - Verify

sudo nginx -t
sudo systemctl restart nginx

nginx_fix

🚨 Scenario 2: Disk Full Incident

Step 1 - Fill disk

sudo fallocate -l 16G /bigfile

Step 2 - Trigger failiure

echo "hello" > file.txt

🔍 Expected Result: No space left on device

Step 3 - Investigate

df -h

Disk_Space

Check system messages:

dmesg | grep -i space

dmesg

Check logs (may be unreliable):

sudo systemctl -xe

journalctl

Find what filled the disk:

du -sh /* 2>/dev/null

Disk_Usage

Step 4 - Fix the issue:

sudo rm /bigfile
df -h

fix

Step 5 - Verify recovery:

echo "test" > file.txt

Recovery

🧠 Key Takeaways

  • Disk issues often look like unrelated failures
  • Logs may fail or be incomplete
  • Always check disk first
  • Must verify fixes, not assume