Linux System Administration Guide

Network Analysis¶

Connection Monitoring¶

ss -tanpur | awk '{print $6}' | cut -d':' -f1 | sort | uniq  #(1)!

1. Shows unique IP addresses of connected machines using modern ss command

SSH Operations¶

ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -o "GlobalKnownHostsFile=/dev/null" user@host  #(1)!

curl https://github.com/<username>.keys  #(1)!
curl https://gitlab.com/<username>.keys  #(2)!

ssh-keygen -l -E md5 -f .ssh/key.pub  #(1)!

Ghostty Terminfo on Remote Servers¶

When using Ghostty as your terminal, SSH sessions may fail with Error opening terminal: xterm-ghostty if the terminfo entry is missing on the remote server.

infocmp -x xterm-ghostty | ssh YOUR-SERVER -- sudo tic -x -o /usr/share/terminfo -  # (1)!

sudo ln -s /usr/share/terminfo/g/ghostty /usr/share/terminfo/x/xterm-ghostty

shell-integration-features = ssh-terminfo,ssh-env

System Updates & Configuration¶

Arch Linux Package Management¶

for i in $(find /etc -iname '*.pacnew'); do sudo meld $i ${i%%.pacnew} && sudo rm -i $i; done  #(1)!
for i in $(find /etc -iname '*.pacsave'); do sudo meld $i ${i%%.pacsave} && sudo rm -i $i; done  #(2)!

1. Compare and merge .pacnew files with current configs
2. Handle .pacsave backup files from package updates

Debian/Ubuntu Configuration Merging¶

for file in $(find /etc -iname '*.dpkg-dist'); do vimdiff ${file%%.dpkg-dist} $file; rm $file; done  #(1)!
for file in $(find /etc -iname '*.dpkg-old'); do vimdiff ${file%%.dpkg-old} $file; rm $file; done   #(2)!
for file in $(find /etc -iname '*.dpkg-new'); do vimdiff ${file%%.dpkg-new} $file; rm $file; done   #(3)!

1. Merge distribution config files with local versions
2. Handle old configuration backups
3. Process new configuration files from packages

SSL/TLS Management¶

Let's Encrypt Certificates¶

certbot certonly --non-interactive --webroot --webroot-path /var/www/html/ -d foo.bar -d www.foo.bar  #(1)!

1. Non-interactive certificate generation using webroot validation

Certificate Analysis¶

find /etc/ssl/certs/ -type f -print -exec openssl x509 -text -in {} \; | grep --color=auto -e etc -e CN= -e DNS: -e After  #(1)!

1. Display custom certificates with Common Names, SANs, and expiration dates

User Management¶

User Migration¶

for i in user1 user2 user3; do echo -n 'useradd -m -s /bin/bash -u '$(grep -E "^$i" /etc/passwd | cut -d':' -f3) && echo -en ' -p'\''$(grep -E "^$i" /etc/shadow | cut -d ':' -f2)'\'' $i ''\n'; done  #(1)!

1. Generates useradd commands with existing UIDs and password hashes for account migration

Group Management¶

for group in $(grep user1 /etc/group | cut -d':' -f1 | sed '/user1/d'); do adduser user2 $group; done  #(1)!

1. Adds user2 to all groups that user1 belongs to

Storage & File Systems¶

Loop Device Management¶

losetup -P -f --show my.img  #(1)!

1. Creates loop device with partition support for disk image files

File System Creation¶

mkfs.ext4 -O ^64bit,^metadata_csum  #(1)!

1. Creates ext4 without modern features for compatibility with older systems

File Operations¶

find . ! -newermt '2012-09-19 11:40:00' -exec cp {} /tmp/mails \;  #(1)!

find ./ -type f ! -perm /g=r -exec ls -l {} \;  #(1)!
find ./ -type d \( ! -perm /g=r -o ! -perm /g=x \) -exec ls -ld {} \;  #(2)!
find ./ -perm /g=w  #(3)!
find ./ -perm -007 -o -type f -perm -006  #(4)!

Database Operations¶

MySQL Management¶

mysql -e "set global innodb_max_dirty_pages_pct = 0;"  #(1)!
mysqladmin ext -i10 | grep dirty  #(2)!

#!/bin/bash
SEC=$1
IFS='|'
if [[ $SEC -lt 1 ]]; then
    echo "Usage: $0 SECONDS"
    exit 1
fi
mysqladmin proc -v|grep Query|grep -Evi "delete|update|insert|alter table" |while read dummy qid qusr qhost qdb qstat qsec qstat2 query; do
    if [ $qsec -gt $SEC ]; then
        echo "Killing query $qid..."
        mysqladmin kill $qid
    fi
done

# Source Server
mysql mysql -e "select * from user WHERE USER='user1' OR USER='user2' INTO OUTFILE '/tmp/mysql_user';"  #(1)!
mysql mysql -e "select * from db WHERE USER='user1' OR USER='user2' INTO OUTFILE '/tmp/mysql_db';"     #(2)!

# Destination Server
scp server:/tmp/mysql_{db,user} /tmp  #(3)!
chmod 664 /tmp/mysql_{db,user}        #(4)!
mysql mysql -e "LOAD DATA INFILE '/tmp/mysql_user' INTO TABLE user;"  #(5)!
mysql mysql -e "LOAD DATA INFILE '/tmp/mysql_db' INTO TABLE db;"      #(6)!

#!/bin/bash
tables=$(grep crashed /var/log/syslog | grep -Eo '\./.*' --color=auto | sed s#\'./## | sed s#\'## | uniq | tr -s '\n' ' ')  #(1)!
for tableC in $tables; do
    db=${tableC%/*}     #(2)!
    table=${tableC#*/}  #(3)!
    mysqlcheck --auto-repair --check $db $table  #(4)!
done

Mail System Management¶

Postfix Operations¶

postcat -q ID > mail          #(1)!
< mail sendmail -f FROM TO    #(2)!

for i in $(mailq | grep -Eo [A-F0-9]{10} | tr -s '\n' ' '); do postcat -q $i | grep userid | grep -Eo "[0-9]{4,}" >> tmp/userid; done  #(1)!
sort -n /tmp/userid | uniq  #(2)!

System Monitoring & Troubleshooting¶

Process Analysis¶

ps axo user:20,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm  #(1)!

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less  #(1)!

System Debugging¶

strace -p $(ps auwwwx | grep apache | tr -s '\t' ' ' | cut -d' ' -f2 | tr '\n' ' ' | sed 's/ / -p /g') 9999  #(1)!

tail -f $(lsof | grep -F .log | tr -s '\t' ' ' | cut -d' ' -f10 | sort | uniq | tr -s '\n' ' ')  #(1)!

grep -Eo '"POST .*.php' access.log | grep -ve cron -e login -e admin -e xmlrpc -e trackback -e comment -e 404 | sort -u  #(1)!

PHP Debugging¶

HTTP_HOST=www.site.com SCRIPT_FILENAME=index.php REDIRECT_STATUS=CGI SERVER_NAME=www.site.com strace -s 65535 -o /tmp/strace php-cgi -f index.php  #(1)!

1. Debug PHP execution with full environment and system call tracing

Package Management¶

Debian Package Analysis¶

dpkg-query -W -f='${Section}\\t${Package}\\n' | grep ^non-free  #(1)!

1. List all packages installed from non-free repository section

DNS Management¶

sed -ri 's/^\\s*[0-9]+\\s*; serial/\\t\\t\\t 2017041010\\t ; serial/' db.*  #(1)!

1. Update DNS zone serial numbers (requires customization for your format)

Service Management¶

Namespace Operations¶

nsenter -t $(cat /var/snap/lxd/common/lxd.pid) -m  #(1)!

1. Enter LXD namespace to access internal filesystem structures

GPG Agent Management¶

gpg-connect-agent
KEYINFO --ssh-list --ssh-fpr  #(1)!
DELETE_KEY $HASH              #(2)!

1. List SSH keys managed by GPG agent
2. Remove specific key by hash

Web Services¶

Python HTTP Server¶

cd /var/cache/munin/www
python -m SimpleHTTPServer 8080  #(1)!

1. Start simple HTTP server for testing (useful for Munin, etc.)

Squid Proxy Analysis¶

tail -n100 /var/log/squid3/access.log | grep -oE 'http.*' | cut -d ' ' -f1 | sort | uniq  #(1)!

1. Extract and list unique URLs from recent Squid access logs

System Recovery¶

Password Recovery¶

Boot-time password recovery procedure:

mount -o remount,rw /  #(1)!
passwd                 #(2)!

1. Remount root filesystem as read-write
2. Change root password

Log Management¶

savelog /var/log/application.log  #(1)!

1. Manually rotate log file without using logrotate

Contact Information¶

Technical Contact Templates¶

When contacting domains without published technical contacts:

abuse@<domain>
admin@<domain>
administrator@<domain>
contact@<domain>
info@<domain>
postmaster@<domain>
support@<domain>
webmaster@<domain>

Additional Resources¶

htop          # Interactive process viewer
iotop         # I/O monitoring
nethogs       # Network usage by process
tcpdump       # Network packet analysis
strace        # System call tracing
lsof          # List open files and network connections

Auditing Modified Config Files¶

After a major Ubuntu upgrade, some config files may have changed upstream but kept your old version. Use debsums to find which conffiles differ from their package originals, then diff them to see exactly what changed.

apt install -y debsums

debsums -ce

for f in $(debsums -ce 2>/dev/null); do
  pkg=$(dpkg -S "$f" 2>/dev/null | head -1 | cut -d: -f1)
  tmp=$(mktemp -d)
  cd "$tmp"
  apt download "$pkg" 2>/dev/null
  dpkg-deb -x *.deb extract
  echo "=== $f ==="
  diff -u "extract$f" "$f" || true
  echo
  cd /
  rm -rf "$tmp"
done

This is especially useful after jumping multiple Ubuntu releases (e.g. 24.04 → 25.10), where dpkg may have kept old config files that no longer match what the new package ships. Look for syntax changes, new defaults, or deprecated options that could cause silent failures.