Linux Snippets

Commands include Linux commands and commands also commonly found on Unix or Unix like operating systems or cross platform shells like Nushell

Apropos - Search commands for functions

Useful if you forgot a command and need to search command names and manual pages

apropos "search term"

# Example
apropos "list files"

awk / mawk

Pattern scanning and text processing language. Mawk interprets the AWK programming language
List of pattern action pairs and function definitions
Input can be list of files and standard input
Awk can be enclosed in ’ ’ or from a file

# Emulate cat - Print contents of a file
mawk '{print}' file.txt

# Get contents of a file where line contains name, split line on = and print name / value pair
mawk '/name/ {split($0,a,"="); print a[1], a[2]}' .gitconfig

cd - Change directory

# Using echo $$ to find the pid from another shell
# go to the current working directory of that shell using its PID
cd /proc/12467/cwd

# Change to previous directory
## bash
cd -
## fish
prevd

cpumonitor cpupower to set CPU Governance

# https://documentation.ubuntu.com/server/explanation/performance/perf-tune-cpupower/

# List all available cpupower monitors available on the system
sudo cpupower monitor -l

# Set the CPU governor to performance mode on all CPUs
cpupower frequency-set -g performance

dig - DNS lookup

dig google.com

fsck or smartctl - check disks for errors

Source: How to Run a Disk Check to Fix Bad Sectors - Baeldung

Alternatively, can use gnome-disks GUI app and click on the drive and use 3 dot menu to run SMART (Self-Monitoring, Analysis, and Reporting Technology) tests

Get the device identifiers for the disk(s) to scan

# List disks
df -h
# Full list including loops
lsblk -f

For later commands, /dev/sdb is the example disk to scan

fsck

# First unmount the disk to scan
sudo umount /dev/sdb

# Do a dry run
sudo fsck -N /dev/sdb

# Force filesystem check, perform all checks to search for corruptions even when it thinks there are no issues
sudo fsck -f /dev/sdb

# Fix Detected Error Automatically
sudo fsck -y /dev/sdb

# Force fsck to Do a Filesystem Check
# When you perform a fsck on a clean device, the tool skips the filesystem check. If you want to force the filesystem check, use the -f option.
sudo fsck -f /dev/sdb

# Remount the scanned disk
mount /dev/sdb

smartctl

# Can be installed for example with apt
sudo apt-get install smartmontools

# Run scan, note some drives do not support SMART
sudo smartctl -a /dev/sdb

grep - find patterns

print lines that match patterns

Using grep on cwd files should use -r or specify the file(s), otherwise, grep assumes grep on standard input

# ignore case, include lines that match pattern
grep -ir <pattern>
## -r search on files recursively

# exclude lines that match pattern
grep -vr <pattern> *

# Use grep with regex PATTERN on current directory
grep PATTERN *

# print lines that match pattern
grep <pattern> *
## * search in all files in current directory

grub

#####################################################
# List all grub entries
# per: https://askubuntu.com/questions/599208/how-to-list-grubs-menuentries-in-command-line
# Simple
cat /boot/grub/grub.cfg | grep menuentry
# Nicely printed
awk -F\' '/menuentry / {print $2}' /boot/grub/grub.cfg
# With numbers for use with grub-set-default, grub-reboot
awk -F\' '/^menuentry / {print $2}' /boot/grub/grub.cfg|cat -n|awk '{print $1-1,$1="",$0}'
# List all entries
awk -F\' '/(^|| )nuentry / {print $2}' /boot/grub/grub.cfg|cat -n|awk '{print $1-1,$1="",$0}'

# Boot into a specific grub entry with number or name
sudo grub-reboot "Windows Boot Manager (on /dev/nvme0n1p1)" # Reboot into Windows next time
# or
sudo grub-reboot 1

htop - Process and task information

# Start `htop` displaying processes owned by a specific user:
htop [-u|--user] username

# Display processes hierarchically in a tree view to show the parent-child relationships:
htop [-t|--tree]

# Sort processes by a specified `sort_item` (use `htop --sort help` for available options):
htop [-s|--sort] sort_item

ImageMagick, Image PDF Conversions, Combining, PDF Optical Character Recognition (OCR)

For image to PDF conversions and OCR, recommend use img2pdf and ocrmypdf

img2pdf "$file" | ocrmypdf - "$file.pdf" --rotate-pages

ocrmypdf path/to/pdf output.pdf

# Join / Merge / combine several PDFs to new file output.pdf
# https://www.omglinux.com/merge-pdf-files-on-linux/
pdfunite file1.pdf file2.pdf output.pdf

# per https://itsfoss.com/merge-pdf-linux/
# Deprecated:
convert -append file1.pdf file2.pdf file3.pdf output.pdf
# If error with above command convert-im6.q16: attempt to perform an operation not allowed by the security policy `PDF' @ error/constitute.c/IsCoderAuthorized/421
# do:
sudo nano /etc/ImageMagick-6/policy.xml
# Usually at bottom of file, you have to change the rights="none" to rights=read|write:
# <policy domain="coder" rights="read|write" pattern="PDF" />

# You can use the convert command in bash to convert multiple images to a single PDF file. Here’s an example command that you can use to convert 3 images to a single PDF file:
# This command will convert image1.jpg, image2.jpg, and image3.jpg to a single PDF file named output.pdf.
# Replace names of your images and the output file name with the desired name of your PDF file.
convert image1.jpg image2.jpg image3.jpg output.pdf

# Make images ready for web - Scale files in a directory to a width in pixels
#!/bin/bash
# for each file in current directory
for file in *; do
    # Use ImageMagick to scale the file's width as pixels in variable "resolution"
    # and rename the new image with the same name as the original file plus "resolution" variable
    resolution = "720"
    convert "$file" -resize "$resolution" "$file-$resolution.jpg"
done

lsblk - list block devices

List information on available devices, drives, disks

# List all block devices
lsblk

# List block devices with colourized output bat -l conf for config language and -p for plain
lsblk | bat -l conf -p

# List all block devices with size
lsblk -s

# List all block devices with size and mount point
lsblk -s -o NAME,SIZE,MOUNTPOINT

# List all block devices with size and mount point and exclude any loop devices
lsblk -s -o NAME,SIZE,MOUNTPOINT | grep -v loop

man - manual, whatis

# Manual for passwd command Section 1: Shell Commands and Applications
man passwd

# Manual for passwod file in Section 5: Standard file formats
man 5 passwd

# Search all man pages with keyword ftp
man -k ftp

# Check if a manual page exists for a certain concept and
# display search results and the relevant Linux manual page section like passwd (5)
man -f passwd
# equivalent command is also:
whatis passwd

# Show location of manual page for a concept
man -w passwd

# Extract arguments of ls command from man page (works on Unix from 2005)
function args() {
    man ls | col -b | grep '^[[:space:]]*ls \[' | awk -F '[][]' '{print $2}'
}

Manual Sections from RTFM! How to Read (and Understand) the Fantastic Man Pages in Linux

Section 1 : Shell commands and applications
Section 2 : Basic kernel services – system calls and error codes
Section 3 : Library information for programmers
Section 4 : Network services – if TCP/IP or NFS is installed Device drivers and network protocols
Section 5 : Standard file formats – for example: shows what a tar archive looks like.
Section 6 : Games
Section 7 : Miscellaneous files and documents
Section 8 : System administration and maintenance commands
Section 9 : Obscure kernel specs and interfaces

mount, umount - attach / mount file systems

#  Show all mounted filesystems:
mount

#  Mount a device to a directory:
mount path/to/device_file path/to/target_directory

#  Create a specific directory if it does not exist and mount a device to it:
mount [-m|--mkdir] path/to/device_file path/to/target_directory

#  Mount a device to a directory for a specific user:
mount [-o|--options] uid=user_id,gid=group_id path/to/device_file path/to/target_directory

#  Mount a CD-ROM device (with the filetype ISO9660) to `/cdrom` (readonly):
mount [-t|--types] iso9660 [-o|--options] ro /dev/cdrom /cdrom

#  Mount all the filesystems defined in `/etc/fstab`:
mount [-a|--all]

#  Mount a specific filesystem described in `/etc/fstab` (e.g. `/dev/sda1 /path/to/mount_point ext2 defaults 0 2`):
mount path/to/mount_point

#  Mount a directory to another directory:
mount [-B|--bind] path/to/old_directory path/to/new_directory

# Unmount a filesystem, by passing the path to the source it is mounted from:
sudo umount path/to/device_file

#  Unmount a filesystem, by passing the path to the target where it is mounted:
sudo umount path/to/mounted_directory

#  When an unmount fails, try to remount the filesystem read-only:
sudo umount [-r|--read-only] path/to/mounted_directory

#  Recursively unmount each specified directory:
sudo umount [-R|--recursive] path/to/mounted_directory

#  Unmount all mounted filesystems (except the `proc` filesystem):
sudo umount [-a|--all]

ncdu - ncurses disk usage, see disk usage by directories and navigate them

# See cwd
ncdu

# See root
ncdu -x /

# Scan with SSH with
ssh -C user@system ncdu -o- / | ./ncdu -f-

Networking

ifconfig - network information
netstat (deprecated, use ss) - active network connections
ps aux - check processes and network usage
nslookup - check IP address of domains
ping - check if a host is reachable

Other Linux/Unix Commands

The first command indicates the command name (searchable with a man page). The parens indicate useful extensions

Command	Usage/Function
egrep	Extract line containing word / pattern after the command
pwd	Print (output to console) current working directory
less, more	Console content control/reader
cp	copy
mv	move
rm (-rf)	Remove (with recursive and forced)

ampersand (&)Â - Executed after a UNIX command makes the command run while providing the command prompt back. Using & allows you to continue to type more UNIX commands.

Common use of the ampersand ( & ) is at the end of commands that open their own windows like a web browser or an editor..

The amperand (&) means something different when used immediately after a greater than (>) for output redirection or after the pipe symbol ( | ) for passing output to other commands.

Utility

Command	Usage/Function
cat
man	Manuals (help pages) for system commands.
ln –s	Create symbolic links between files
touch
ftp	File transfer protocol program
ping
finger
telnet
ssh	Secure Shell
bash	Bash shell (running activates bash shell on console if it is available).
whois	Query
traceroute	Trace network route
lynx	Text based HTML browser
mount, unmount	Mount or unmounts file systems
date	Date/time on system

runmqsc - (WebSphere Messaging Queue MQ)

ex. runmqsc

DISPLAY CHSTATUS(*)

display all channels info

DISPLAY CHSTATUS(”)

STOP CHANNEL(”)

RESET CHANNEL(”)

START CHANNEL(”)

DISPLAY CHANNEL(”)

System Administration

See DevOps Bootcamp - Operating Systems and Linux Basics - DevOps Bootcamp - Operating Systems and Linux Basics for common commands and file system structure.

Useful Concepts for Linux/Unix

Directory Structure

Logs, spools, and file resources (mail, logs, temp, etc.) :/var

Configuration Files :/etc, /etc/rc.init (startup scripts). /etc/sendmail.cf

Unix “blackhole” :/dev/null

Core Command locations if not in path :/usr/(s)bin, /usr/local/(s)bin, usr/sfw/bin | Solaris: /usr/openwin

Devices (I/O) :/dev

Mounted Systems :/mnt, /media

Code Libraries/Modules :/usr/lib, /usr/local/lib

Stored source :/usr/src

Services :/etc/services - example of entries in services file (ports and protocols associated with the ports.

netstat 15/tcp

ftp 21/tcp # File Transfer

ssh 22/tcp # Secure Shell

telnet 23/tcp
Platform Specific

User settings ~/.<user setting folder>

e.g.~/.kde/share/apps stores KDE desktop settings and configurations.

Checking Memory

# Check Free physical Memory
top
sar -r
vmstat

# For swap:
swap -s
swap --l

&> pipes

# pipe output of command to file
start_server &> filename.log

# >> redirection operator will append lines to the end of the specified file
start_server >> filename.log

# > empty and overwrite file
start_server > filename.log

pkill, kill, killall - stop process also known as kill

# Kill all processes with name firefox inside process name
pkill -f firefox

# Send signals to process
kill (-HUP) process_id
# kills a process with pid specified
kill -9 pid

# kill all processes by a name, for example firefox browser
killall firefox

ps - process status

# List all processes
ps -A

# List all processes with user name
ps -ef

# Search for a process that matches a string
ps aux | grep string

# in bash, show pid of current shell
echo $$

Redirects (>) and truncate to clear files

# Clear all contents of a specified file

# Non root method is
# https://superuser.com/questions/90008/how-to-clear-the-contents-of-a-file-from-the-command-line
> file-to-clear.txt

# Clear file that requires root permissions
# https://superuser.com/questions/90008/how-to-clear-the-contents-of-a-file-from-the-command-line/634217#634217
sudo truncate -s 0 /etc/environment

Rename

# Find all file names in current directory containing string searchString
# and replace in filename the string foo with string bar
# -n is test first, then run
find . -name "*foo*" -exec rename -n 's/foo/bar/' {} \;
find . -name "*foo*" -exec rename 's/foo/bar/' {} \;

rsync

Source: https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories, Everyday Rsync - Veronica Explains, Rsync on Archwiki

Synchronize directories and files across local and remote machines

# Test Synchronize a source and destination --dry-run
rsync --dry-run -anv source destination

# Synchronize with ssh destination
rsync -av ~/records username@destination.server:~/record-archives
## Best practice is destination is remote in case remote directory is not mounted properly

# Recurvisely copy first directory/file to destination file and exclude folder/files called Archive and include only 7z files
rsync -aPvh --delete "${LIBRARYDIR}/${array[$i]}/" "${LOCALDIR}/${array[$i]}/" --exclude "Archive" --include="*.7z"
# -a archive mode (recursively copy directories, copy symlinks without resolving, and preserve permissions, ownership and modification times)
# -P show progress, partial transfer of files
# -v verbose
# -h use human readable numbers
# --delete delete files in destination directories if not existing in source
#   Can also use --delete-before or --delete-after to have deletes occur at a certain time in job
# --exclude PATTERN do not copy files with PATTERN
# --include PATTERN only include files with PATTERN

# Copy exmaple.txt from local to remote server
rsync -avz -e ssh /path/to/example.txt user@remote-host:/path/to/destination/

# Copy from local to remote
# --progress : Show progress during a transfer using
# -a Archive for recursive and preserver file attributes
# -v verbose
# -z compress data to reduce network use, required processing power
rsync -avz --progress /local/destination/ -e ssh user@remote-host:/path/to/remote-file.txt

-a option is a combination flag. It stands for “archive” and syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions. It is more commonly used than -r and is usually what you want to use. n or –dry-run options (no changes made) -v verbose The -P flag is very helpful. It combines the flags –progress and –partial. The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers –exclude is used to ignore files and can have a recursive flag

sed (stream editor)

# [s]ubstitute all occurrences of "apple" with "mango" on all lines, print to stdout, use basic regex
command | sed 's/apple/mango/g'

# Use extended regular expressions with -E
sed -E 's/pattern/replacement/flags'
## Replace dog and cat with pet
"dog cat moose" | sed -E 's/(cat|dog)/(pet)/g'

# Replace "apple" with "mango" in-place in a file (overwriting original file), use basic regex
sed [-i|--in-place] 's/apple/mango/g' path/to/file
sed -i 's/apple/mange/g' fruits.txt

# Run multiple substitutions in one command
command | sed -e 's/apple/mango/g' -e 's/orange/lime/g'

# Use a custom delimiter (useful when the pattern contains slashes)
command | sed 's#////#____#g'

# [d]elete lines 1 to 5 of a file and back up the original file with a .orig extension
sed [-i|--in-place=].orig '1,5d' path/to/file

# [p]rint only the first line to stdout
command | sed [-n|--quiet] '1p'

# [i]nsert a new line at the beginning of a file, overwriting the original file
sed [-i|--in-place] '1i\your new line text\' path/to/file

# Delete blank lines (with or without spaces/tabs) from a file, overwriting the original file:
sed [-i|--in-place] '/^[[:space:]]*$/d' path/to/file

# Delete lines containing a string
sed -i '/lang.sql/d' lazyvim.json

source, environment variables

# Source temporary environment variables
# using an .env file
# from https://stackoverflow.com/questions/43267413/how-to-set-environment-variables-from-env-file

## Option 1
### This requires appropriate shell quoting. It's thus appropriate
### if you would have a line like foo='bar baz', but not if that same line would be written foo=bar baz
set -a # automatically export all variables
source .env
set +a

## Option 2
### if .env is a valid shell execution file like export foo=bar
### then simple source will work
source .env

## Option 3
### Assuming no whitespace in environment values
export $(xargs <.env)

ss (socket statistics)

Check open ports, connections

# Show all connections
ss

# Network statistics
ss -s
# Real time report, Ctrl + c to exit
watch ss -s

# List ports and processes
# -t TCP sessions
# -u UDP sessions
# -l Listening TCP connections
# -n numeric, human readable
# -p show process at socket
ss -lntup

# IPv4 connections
ss -4
# IPv6 connections
ss -6

tar - archiving utility

# Untar a file todo.txt_cli-2.12.0.tar.gz
tar -xvf todo.txt_cli-2.12.0.tar.gz

wget

# Mirror site - if you have a website, you can make a complete backup using this one simple command
wget -m www.everydaylinuxuser.com
# -m is a combination of all the -r, -l, -k commands below. Output will log to the screen.

# Download site, and do not go to links below the current URL
wget -m www.everydaylinuxuser.com/context/2 --no-parent
# --no-parent can be used to prevent it from navigating to links below the current URL

# Download site recursively
wget -r -k -l10 www.everydaylinuxuser.com
# -r Download a site recursively and create directories as needed (-x forces creation of directories)
# -l Up to 10 levels deep, by default the command only downloads 5 levels deep
# -k Turns links into relative local links

# Download links from file, infinite levels down
wget -r -k -l0 --no-parent -i artemislinks
# -i download using URLs from a file called artemislinks
# -l0 infinite levels

which, where as, also use newer command type

which <command> - find the location of a command

type <command>

Similar to:

whereis <command> - find the location of command and its manual page