Bash-Oneliner

Hi bash learners and bioinformaticans, welcome to Bash Oneliner. I started studying bioinformatics data four years ago (recently started working on cloud computing), and was amazed by those single-word bash commands which are much faster than my dull scripts, so much time can be saved through knowing command-line shortcuts and scripting. Not all the code here is oneliner (if the ';' counts..), but i put effort on making them brief and fast. I am mainly using Ubuntu and RedHat, sorry if the commands dont work on your system.

This blog will focus on simple bash commands for parsing data, most of which are for tsv files (tab-separated values); some of them are for Linux system maintenance. I apologize that there won't be any citation for the codes, but they are probably from dear Google and Stackoverflow.

English and bash are not my first language, so... correct me anytime, thank you!
And if you know any cool command that are not included here, Please Teach Me.

In case you would like to check and vote up my questions on Stackoverflow, here's my page: http://stackoverflow.com/users/4290753/once

Here's a more stylish version of Bash-Oneliner~ http://onceupon.github.io/Bash-Oneliner/

Handy Bash oneliner commands for tsv file editing

• Grep
• Sed
• Awk
• Xargs
• Find
• Loops
• Math
• Download
• Random
• Xwindow
• Others
• System

Grep

Extract text bewteen words (e.g. w1,w2)

grep -o -P '(?<=w1).*(?=w2)'

Grep only integer

grep -o '[0-9]*'

Grep lines without word (e.g. bbo)

grep -v bbo filename

Grep lines not begin with string (e.g. #)

grep -v '^#' file.txt

Grep variables with space within it (e.g. bbo="some strings")

grep "$boo" filename

#remember to quote the variable!

Grep only one/first match (e.g. bbo)

grep -m 1 bbo filename

Grep and count (e.g. bbo)

grep -c bbo filename

Insensitive grep (e.g. bbo/BBO/Bbo)

grep -i "bbo" filename 

Count occurrence (e.g. three times a line count three times)

grep -o bbo filename 

COLOR the match (e.g. bbo)!

grep --color bbo filename 

Grep search all files in a directory(e.g. bbo)

grep -R bbo /path/to/directory 

or

grep -r bbo /path/to/directory 

Search all files in directory, only output file names with matches(e.g. bbo)

grep -Rh bbo /path/to/directory 

or

grep -rh bbo /path/to/directory 

or only list filename with match

grep -rl bbo /path/to/directory

Grep OR (e.g. A or B or C or D)

grep 'A\|B\|C\|D' 

Grep AND (e.g. A and B)

grep 'A.*B' 

Grep all content of a fileA from fileB

grep -f fileA fileB 

Grep a tab

grep $'\t' 

Grep variable from variable

$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
#grep -q will output 0 if match found  
#remember to add space between []!

Grep strings between a bracket()

grep -oP '\(\K[^\)]+'

Grep number of characters with known strings in between(e.g. AAEL000001-RA)

grep -o -w "\w\{10\}\-R\w\{1\}"
# \w word character [0-9a-zA-Z_] \W not word character 

A lot examples here

http://www.cyberciti.biz/faq/grep-regular-expressions/

Sed

[back to top]

Remove the 1st line

sed 1d filename

Remove the 100 lines (remove line 1-100)

sed 1,100d filename

remove lines with string (e.g. bbo)

sed "/bbo/d" filename
- case insensitive:
sed "/bbo/Id" filename

Edit infile (edit and save)

sed -i "/bbo/d" filename

When using variable (e.g. $i), use double quotes " "

e.g. add >$i to the first line (to make a FASTA file)

sed "1i >$i"  
# notice the double quotes! in other examples, you can use a single quote, but here, no way!   
# '1i' means insert to first line

Delete/remove empty lines

sed '/^\s*$/d' 

or

sed 's/^$/d' 

Delete/remove last line

sed '$d' 

Delete/remove last character from end of file

sed -i '$ s/.$//' filename

Add string to end of file (e.g. "]")

sed '$s/$/]/' filename

Add newline to the end

sed '$a\'

Add string to beginning of every line (e.g. bbo)

sed -e 's/^/bbo/' file

Add string to end of each line (e.g. "}")

sed -e 's/$/\}\]/' filename

Add \n every nth character (e.g. every 4th character)

sed 's/.\{4\}/&\n/g' 

Concatenate/combine/join files with a seperator and next line (e.g seperate by ",")

sed -s '$a,' *.json > all.json

Substitution (e.g. replace A by B)

sed 's/A/B/g' filename 

Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)

sed "s/aaa=.*/aaa=\/my\/new\/path/g"

Select lines start with string (e.g. bbo)

sed -n '/^@S/p' 

Delete lines with string (e.g. bbo)

sed '/bbo/d' filename 

Print/get/trim a range of line (e.g. line 500-5000)

sed -n 500,5000p filename

Print every nth lines

sed -n '0~3p' filename

# catch 0: start; 3: step

Print every odd # lines

sed -n '1~2p' 

Print every third line including the first line

sed -n '1p;0~3p' 

Remove leading whitespace and tabs

sed -e 's/^[ \t]*//'

//notice a whitespace before '\t'!!

Remove only leading whitespace

sed 's/ *//'

# notice a whitespace before '*'!!

Remove ending commas

sed 's/,$//g' 

Add a column to the end

sed "s/$/\t$i/"

# $i is the valuable you want to add  
# e.g. add the filename to every last column of the file  

for i in $(ls);do sed -i "s/$/\t$i/" $i;done

Add extension of filename to last column

for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np

Remove newline\ nextline

sed ':a;N;$!ba;s/\n//g'

Print a particular line (e.g. 123th line)

sed -n -e '123p'

Print a number of lines (e.g. line 10th to line 33 rd)

sed -n '10,33p' <filename

Change delimiter

sed 's=/=\\/=g'

Replace with wildcard (e.g A-1-e or A-2-e or A-3-e....)

sed 's/A-.*-e//g' filename

Remove last character of file

sed '$ s/.$//'

Insert character at specified position of file (e.g. AAAAAA --> AAA#AAA)

sed -r -e 's/^.{3}/&#/' file

Awk

[back to top]

Set tab as field separator

awk -F $'\t'  

Output as tab separated (also as field separator)

awk -v OFS='\t' 

Pass variable

a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename 

Print line number and number of characters on each line

awk '{print NR,length($0);}' filename 

Find number of columns

awk '{print NF}' 

Reverse column order

awk '{print $2, $1}' 

Check if there is a comma in a column (e.g. column $1)

awk '$1~/,/ {print}'  

Split and do for loop

awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename 

Print all lines before nth occurence of a string (e.g stop print lines when bbo appears 7 times)

awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'

Print filename and last line of all files in directory

ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file

Add string to the beginning of a column (e.g add "chr" to column $3)

awk 'BEGIN{OFS="\t"}$3="chr"$3' 

Remove lines with string (e.g. bbo)

awk '!/bbo/' file 

Usage and meaning of NR and FNR

e.g.
fileA:
a
b
c
fileB:
d
e

awk 'print FILENAME, NR,FNR,$0}' fileA fileB 

fileA 1 1 a
fileA 2 2 b
fileA 3 3 c
fileB 4 1 d
fileB 5 2 e

AND gate

e.g.
fileA:
1 0
2 1
3 1
4 0

fileB:
1 0
2 1
3 0
4 1

awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB 

1 0
2 1
3 0
4 0

Round all numbers of file (e.g. 2 significant figure)

awk '{while (match($0, /[0-9]+\[0-9]+/)){
    \printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
    \$0=substr($0, RSTART+RLENGTH)
    \}
    \print
    \}'

Give number/index to every row

awk '{printf("%s\t%s\n",NR,$0)}'

Break combine column data into rows

e.g.
seperate

David cat,dog
into
David cat
David dog

detail here:　http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string

awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' file

Average a file (each line in file contains only one number)

awk '{s+=$1}END{print s/NR}'

Print field start with string (e.g Linux)

awk '$1 ~ /^Linux/'

Sort a row (e.g. 1 40 35 12 23 --> 1 12 23 35 40)

awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i <= length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'

Subtract previous row values (add column6 which equal to column4 minus last column5)

awk '{$6 = $4 - prev5; prev5 = $5; print;}'

Xargs

[back to top]

Set tab as delimiter (default:space)

xargs -d\t

Display 3 items per line

echo 1 2 3 4 5 6| xargs -n 3
# 1 2 3  
# 4 5 6

Prompt before execution

echo a b c |xargs -p -n 3

Print command along with output

xargs -t abcd
# bin/echo abcd  
# abcd

With find and rm

find . -name "*.html"|xargs rm -rf

Delete fiels with whitespace in filename (e.g. "hello 2001")

find . -name "*.c" -print0|xargs -0 rm -rf

Show limits

xargs --show-limits

Move files to folder

find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old

or

find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old

Move first 100th files to a directory (e.g. d1)

ls |head -100|xargs -I {} mv {} d1

Parallel

time echo {1..5} |xargs -n 1 -P 5 sleep

a lot faster than

time echo {1..5} |xargs -n1 sleep

Copy all files from A to B

find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B

# v: verbose|  
# p: keep detail (e.g. owner)

With sed

ls |xargs -n1 -I file sed -i '/^Pos/d' filename

Add the file name to the first line of file

ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt

Count all files

ls |xargs -n1 wc -l

Turn output into a single line

ls -l| xargs

Count files within directories

echo mso{1..8}|xargs -n1 bash -c 'echo -n "$1:"; ls -la "$1"| grep -w 74 |wc -l' --

# "--" signals the end of options and display further option processing

Download dependencies files and install (e.g. requirements.txt)

cat requirements.txt| xargs -n1 sudo pip install

Count lines in all file, also count total lines

ls|xargs wc -l

Xargs and grep

cat grep_list |xargs -I{} grep {} filename

Xargs and sed (replace all old ip address with new ip address under /etc directory)

grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'

Find

[back to top]

List all sub directory/file in the current directory

find .

List all files under the current directory

find . -type f

List all directories under the current directory

find . -type d

Edit all files under current directory (e.g. replace 'www' with 'ww')

find . name '*.php' -exec sed -i 's/www/w/g' {} \;

if no subdirectory

replace "www" "w" -- *
# a space before *

Find and output only filename (e.g. "mso")

find mso*/ -name M* -printf "%f\n"

Find and delete file with size less than (e.g. 74 byte)

find . -name "*.mso" -size -74c -delete

# M for MB, etc

Loops

[back to top]

While loop, column subtraction of a file (e.g. a 3 columns file)

while read a b c; do echo $(($c-$b));done < <(head filename)

#there is a space between the two '<'s

While loop, sum up column subtraction

i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename)

While loop, keep checking a running process (e.g. perl) and start another new process (e.g. python) immetiately after it. (BETTER use the wait command! Ctrl+F 'wait')

while [[ $(pidof perl) ]];do echo f;sleep 10;done && python timetorunpython.py

If loop

if (($j==$u+2))
#(( )) use for arithmetic operation

if [[$age >21]]
#[[ ]] use for comparison

Test if file exist

if [ -e 'filename' ]
then
  echo -e "file exists!"
fi

if else Test if file exist

 if [ -e $filename ]; then echo -e "file exists!"; else mkdir $filename; fi

For loop

for i in $(ls); do echo file $i;done

for loop, press any key to continue each loop

for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done

for loop, print a file line by line when a key is pressed

for line in $(cat myfile); do echo $line; read -n1; done

Math

[back to top]

Print out the prime factors of a number (e.g. 50)

factor 50

Sum up input list (e.g. seq 10)

seq 10|paste -sd+|bc

Sum up a file (each line in file contains only one number)

awk '{s+=$1} END {print s}' filename

Column subtraction

cat file| awk -F '\t' 'BEGIN {SUM=0}{SUM+=$3-$2}END{print SUM}'

Simple math with expr

expr 10+20 #30
expr 10\*20 #600
expr 30 \> 20 #1 (true)

More math with bc

• Number of decimal digit/ significant figure

echo "scale=2;2/3" | bc  
#.66

• Exponent operator

echo "10^2" | bc  
#100

• Using variables

echo "var=5;--var"| bc  
#4

Download

[back to top]

Download all from a page

wget -r -l1 -H -t1 -nd -N -np -A mp3 -e robots=off http://example.com

# -r: recursive and download all links on page  
# -l1: only one level link  
# -H: span host, visit other hosts  
# -t1: numbers of retries  
# -nd: don't make new directories, download to here  
# -N: turn on timestamp  
# -nd: no parent  
# -A: type (seperate by ,)  
# -e robots=off: ignore the robots.txt file which stop wget from crashing the site, sorry example.com

Upload a file to web and download (https://transfer.sh/)

--> upload:

curl --upload-file ./filename.txt https://transfer.sh/filename.txt

(the above command will return a URL, e.g: https://transfer.sh/tG8rM/filename.txt)
--> download:

curl https://transfer.sh/tG8rM/filename.txt -o filename.txt

Download file if necessary

data=file.txt
url=http://www.example.com/$data
if [! -s $data];then
    echo "downloading test data..."
    wget $url
fi

Wget to a filename (when a long name)

wget -O filename "http://example.com"

Wget files to a folder

wget -P /path/to/directory "http://example.com"

Random

[back to top]

Random pick 100 lines from a file

shuf -n 100 filename

Random order (lucky draw)

for i in a b c d e; do echo $i; done| shuf

Echo series of random numbers between a range (e.g. shuffle numbers from 0-100, then pick 15 of them randomly)

shuf -i 0-100 -n 15

Echo a random number

echo $RANDOM

Random from 0-9

echo $((RANDOM % 10))

Random from 1-10

echo $(((RANDOM %10)+1))

Xwindow

[back to top]

X11 GUI applications! Here are some GUI tools for you if you get bored by the text-only environment.

Enable X11 forwarding,in order to use graphical application on servers

ssh -X user_name@ip_address

or setting through xhost

--> Install the following for Centos:
xorg-x11-xauth
xorg-x11-fonts-*
xorg-x11-utils

Little xwindow tools

xclock
xeyes
xcowsay

Open pictures/images from ssh server

1. ssh -X user_name@ip_address
2. apt-get install eog
3. eog picture.png

Use gedit on server (GUI editor)

1. ssh -X user_name@ip_address
2. apt-get install gedit
3. gedit filename.txt

Open PDF file from ssh server

1. ssh -X user_name@ip_address
2. apt-get install evince
3. evince filename.pdf

Use google-chrome browser from ssh server

1. ssh -X user_name@ip_address
2. apt-get install libxss1 libappindicator1 libindicator7
3. wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
4. sudo apt-get install -f
5. dpkg -i google-chrome*.deb
6. google-chrome

[back to top]

System

[back to top]

Snapshot of the current processes

ps 

Check graphics card

lspci

Show IP address

$ip add show

or

ifconfig

Check system version

cat /etc/*-release

Linux Programmer's Manuel: hier- description of the filesystem hierarchy

man hier

List job

jobs -l

Export PATH

export PATH=$PATH:~/path/you/want

Make file execuable

chmod +x filename
# you can now ./filename to execute it

List screen

screen -d -r

Echo screen name

screen -ls

Check system (x86-64)

uname -i

Surf the net

links www.google.com

Add user, set passwd

useradd username
passwd username

Edit variable for bash, (e.g. displaying the whole path)

1. joe ~/.bash_profile 
2. export PS1='\u@\h:\w\$' 
# $PS1 is a variable that defines the makeup and style of the command prompt 

3. source ~/.bash_profile

Edit environment setting (e.g. alias)

1. joe ~/.bash_profile
2. alias pd="pwd" //no more need to type that 'w'!
3. source ~/.bash_profile

List environment variables (e.g. PATH)

$echo $PATH
# list of directories separated by a colon

List all environment variables for current user

$env

Show partition format

lsblk

Soft link program to bin

ln -s /path/to/program /home/usr/bin
# must be the whole path to the program

Show hexadecimal view of data

hexdump -C filename.class

Jump to different node

rsh node_name

Check port (active internet connection)

netstat -tulpn

Find whick link to a file

readlink filename

Check where a command link to (e.g. python)

which python

List total size of a directory

du -hs .

or

du -sb

Copy directory with permission setting

cp -rp /path/to/directory

Store current directory

pushd . $popd ;dirs -l 

Show disk usage

df -h 

or

du -h 

or

du -sk /var/log/* |sort -rn |head -10

Show current runlevel

runlevel

Switch runlevel

init 3 

or

telinit 3 

Permanently modify runlevel

1. edit /etc/init/rc-sysinit.conf 
2. env DEFAULT_RUNLEVEL=2 

Become root

su

Become somebody

su somebody

Report user quotes on device

requota -auvs

Get entries in a number of important databases

getent database_name

(e.g. the 'passwd' database)

getent passwd
# list all user account (all local and LDAP)  
# (e.g. fetch list of grop accounts)

getent group
# store in database 'group'

Change owner of file

chown user_name filename
chown -R user_name /path/to/directory/
# chown user:group filename

List current mount detail

df

List current usernames and user-numbers

cat /etc/passwd

Get all username

getent passwd| awk '{FS="[:]"; print $1}'

Show all users

compgen -u

Show all groups

compgen -g

Show group of user

group username

Show uid, gid, group of user

id username

Check if it's root

if [$(id -u) -ne 0];then
    echo "You are not root!"
    exit;
fi
# 'id -u' output 0 if it's not root

Find out CPU information

more /proc/cpuinfo

or

lscpu

Set quota for user (e.g. disk soft limit: 120586240; hard limit: 125829120)

setquota username 120586240 125829120 0 0 /home

Show quota for user

quota -v username

Fork bomb

dont try this at home

:(){:|:&};:

Check user login

lastlog

Edit path for all users

joe /etc/environment
# edit this file

Show running processes

ps aux

Find maximum number of processes

cat /proc/sys/kernal/pid_max

Show and set user limit

ulimit -u

Which ports are listening for TCP connections from the network

nmap -sT -O localhost

Print out number of cores/ processors

nproc --all

Check status of each core

1. top
2. press '1'

Show jobs and PID

jobs -l

List all running services

service --status-all

Schedule shutdown server

shutdown -r +5 "Server will restart in 5 minutes. Please save your work."

Cancel scheduled shutdown

shutdown -c

Boardcast to all users

wall -n hihi

Kill all process of a user

pkill -U user_name

Set gedit preference on server

-->you might have to install the following:

apt-get install libglib2.0-bin;

yum install dconf dconf-editor;
yum install dbus dbus-x11;

-->Check list

gsettings list-recursively

-->Change setting
e.g.

gsettings set org.gnome.gedit.preferences.editor highlight-current-line true
gsettings set org.gnome.gedit.preferences.editor scheme 'cobalt'
gsettings set org.gnome.gedit.preferences.editor use-default-font false
gsettings set org.gnome.gedit.preferences.editor editor-font 'Cantarell Regular 12'

Find out who has logged in on your system

--> [Quick] Printing out only the names:

users

--> [Detail] Printing out login time, load average, etc

w

Add user to a group (e.g add user 'nice' to the group 'docker', so that he can run docker without sudo)

sudo gpasswd -a nice docker

pip install python package without root

1. pip install --user package_name
2. You might need to export ~/.local/bin/ to PATH: export PATH=$PATH:~/.local/bin/

Removing old linux kernels (when /boot almost full...)

1. uname -a  #check current kernel, which should NOT be removed
2. sudo apt-get purge linux-image-X.X.X-X-generic  #replace old version

Change hostname

sudo hostname your-new-name

if not working, do also:

hostnamectl set-hostname your-new-hostname

then run:
hostnamectl

check /etc/hostname

if still not working..., edit:
/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-ensxxx
add HOSTNAME="your-new-hostname"

List installed packages

apt list --installed

or Red Hat:

yum list installed

Check which file make the device busy on umount

lsof /mnt/dir

When sound not working

killall pulseaudio

then press Alt-F2 and type in pulseaudio

When sound not working

killall pulseaudio

Finding Out Hardware Details Without Opening The Computer Case (e.g. memory device detail)

sudo dmidecode -t memory

List information about SCSI devices

lsscsi

List information about NIC

lsscsi|grep -i 'ethernet'

Tutorial for setting up your own DNS server

http://onceuponmine.blogspot.tw/2017/08/set-up-your-own-dns-server.html

Tutorial for creating a simple daemon

http://onceuponmine.blogspot.tw/2017/07/create-your-first-simple-daemon.html

Tutorial for using your gmail to send email

http://onceuponmine.blogspot.tw/2017/10/setting-up-msmtprc-and-use-your-gmail.html

Using telnet to test open ports, test if you can connect to a port (e.g 53) of a server (e.g 192.168.2.106)

telnet 192.168.2.106 53

change network maximum transmission unit (mtu) (e.g. change to 9000)

ifconfig eth0 mtu 9000

get pid of a running process (e.g python)

pidof python

or

ps aux|grep python

ntp

start ntp:

ntpd

check ntp:

ntpq -p

remove unnecessary files to clean your server

sudo apt-get autoremove
sudo apt-get clean
sudo rm -rf ~/.cache/thumbnails/*

Remove old kernal:

sudo dpkg --list 'linux-image*'
sudo apt-get remove linux-image-OLDER_VERSION

Increase/ resize root partition (root partition is an LVM logical volume)

pvscan
lvextend -L +130G /dev/rhel/root -r

#Adding -r will grow filesystem after resizing the volume.

Create a UEFI Bootable USB drive (e.g. /dev/sdc1)

sudo dd if=~/path/to/isofile.iso of=/dev/sdc1 oflag=direct bs=1048576

Locate and remove a package

sudo dpkg -l | grep <package_name>
sudo dpkg --purge <package_name>

Create a ssh tunnel

ssh -f -L 9000:targetservername:8088 root@192.168.14.72 -N
#-f: run in background; -L: Listen; -N: do nothing  
#the 9000 of your computer is now connected to the 8088 port of the targetservername through 192.168.14.72  
#so that you can see the content of targetservername:8088 by entering localhost:9000 from your browser.

Count the number of Segate hard disks

lsscsi|grep SEAGATE|wc -l
or
sg_map -i -x|grep SEAGATE|wc -l

Print detail of CPU hardware

dmidecode -t 4
#          Type   Information
#          0   BIOS
#          1   System
#          2   Base Board
#          3   Chassis
#          4   Processor
#          5   Memory Controller
#          6   Memory Module
#          7   Cache
#          8   Port Connector
#          9   System Slots
#         11   OEM Strings
#         13   BIOS Language
#         15   System Event Log
#         16   Physical Memory Array
#         17   Memory Device
#         18   32-bit Memory Error
#         19   Memory Array Mapped Address
#         20   Memory Device Mapped Address
#         21   Built-in Pointing Device
#         22   Portable Battery
#         23   System Reset
#         24   Hardware Security
#         25   System Power Controls
#         26   Voltage Probe
#         27   Cooling Device
#         28   Temperature Probe
#         29   Electrical Current Probe
#         30   Out-of-band Remote Access
#         31   Boot Integrity Services
#         32   System Boot
#         34   Management Device
#         35   Management Device Component
#         36   Management Device Threshold Data
#         37   Memory Channel
#         38   IPMI Device
#         39   Power Supply

Log out your account after a certain period of time (e.g 10 seconds)

TMOUT=10
#once you set this variable, logout timer start running!

Get process ID of a process (e.g. sublime_text)

#pidof
pidof sublime_text

#pgrep, you dont have to type the whole program name
pgrep sublim

#top, takes longer time
top|grep sublime_text

[back to top]

Others

[back to top]

Remove newline / nextline

tr --delete '\n' <input.txt >output.txt

Replace newline

tr '\n' ' ' <filename

To uppercase/lowercase

tr /a-z/ /A-Z/

Compare files (e.g. fileA, fileB)

diff fileA fileB
# a: added; d:delete; c:changed

or

sdiff fileA fileB
# side-to-side merge of file differences

Number a file (e.g. fileA)

nl fileA

or

nl -nrz fileA
# add leading zeros

or

nl -w1 -s ' '
# making it simple, blank seperated

Combine/ paste two files (e.g. fileA, fileB)

paste fileA fileB
# default tab seperated

Reverse string

echo 12345| rev

Read .gz file without extracting

zmore filename

or

zless filename

Run in background, output error file

some_commands  &>log &

or

some_commands 2>log &

or

some_commands 2>&1| tee logfile

or

some_commands 2>&1 >>outfile
#0: standard input; 1: standard output; 2: standard error

Send mail

echo 'heres the content'| mail -a /path/to/attach_file.txt -s 'mail.subject' me@gmail.com
# use -a flag to set send from (-a "From: some@mail.tld")

.xls to csv

xls2csv filename

Append to file (e.g. hihi)

echo 'hihi' >>filename

Make BEEP sound

speaker-test -t sine -f 1000 -l1

Set beep duration

(speaker-test -t sine -f 1000) & pid=$!;sleep 0.1s;kill -9 $pid

History edit/ delete

~/.bash_history

or

history -d [line_number]

Get last history/record filename

head !$

Clean screen

clear

or

Ctrl+l

Send data to last edited file

cat /directory/to/file
echo 100>!$

Run history number (e.g. 53)

!53

Run last command

!!

Run last command that began with (e.g. cat filename)

!cat

or

!c
# run cat filename again

Extract .xf

1.unxz filename.tar.xz  
2.tar -xf filename.tar

Install python package

pip install packagename

Delete current bash command

Ctrl+U

or

Ctrl+C

or

Alt+Shift+#
# to make it to history

Add something to history (e.g. "addmetohistory")

#addmetodistory
#just add a "#" before~~

Sleep awhile or wait for a moment or schedule a job

sleep 5;echo hi

Count the time for executing a command

time echo hi

Backup with rsync

rsync -av filename filename.bak
rsync -av directory directory.bak
rsync -av --ignore_existing directory/ directory.bak
rsync -av --update directory directory.bak
rsync -av directory user@ip_address:/path/to/directory.bak

//skip files that are newer on receiver (i prefer this one!)

Make all directories at one time!

mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat}
# -p: make parent directory  
# this will create project/doc/html/; project/doc/info; project/lib/ext ,etc

Run command only if another command returns zero exit status (well done)

cd tmp/ && tar xvf ~/a.tar

Run command only if another command returns non-zero exit status (not finish)

cd tmp/a/b/c ||mkdir -p tmp/a/b/c

Extract to a path

tar xvf -C /path/to/directory filename.gz

Use backslash "" to break long command

cd tmp/a/b/c \
> || \
>mkdir -p tmp/a/b/c

Get pwd

VAR=$PWD; cd ~; tar xvf -C $VAR file.tar
# PWD need to be capital letter

List file type of file (e.g. /tmp/)

file /tmp/
# tmp/: directory

Bash script

#!/bin/bash
file=${1#*.}
# remove string before a "."

file=${1%.*}
# remove string after a "."

Search from history

Ctrl+r

Python simple HTTP Server

python -m SimpleHTTPServer

Variables

{i/a/,}

e.g. replace all

{i//a/,}

//for variable i, replace all 'a' with a comma e.g. with grep

 test="god the father"
 grep ${test// /\\\|} file.txt
 # turning the space into 'or' (\|) in grep

Read user input

read input
echo $input

Generate sequence 1-10

seq 10

Find average of input list/file

i=`wc -l filename|cut -d ' ' -f1`; cat filename| echo "scale=2;(`paste -sd+`)/"$i|bc

Generate all combination (e.g. 1,2)

echo {1,2}{1,2}
# 1 1, 1 2, 2 1, 2 2

Generate all combination (e.g. A,T,C,G)

set = {A,T,C,G}
group= 5
for ((i=0; i<$group; i++));do
    repetition=$set$repetition;done
    bash -c "echo "$repetition""

Read file content to variable

foo=$(<test1)

Echo size of variable

echo ${#foo}

Echo tab

echo -e ' \t '

Array

declare -A array=()

Send a directory

scp -r directoryname user@ip:/path/to/send

Split file into lines (e.g. 1000 lines/smallfile)

$ split -d -l 1000 bigfilename

Create a large amount of dummy files (e.g 100000 files, 10 bytes each):

#1. Create a big file
dd if=/dev/zero of=bigfile bs=1 count=1000000

#2. Split the big file to 100000 10-bytes files
 split -b 10 -a 10 bigfile

Rename all files (e.g. remove ABC from all .gz files)

rename 's/ABC//' *.gz

Remove extention (e.g remove .gz from filename.gz)

basename filename.gz .gz

zcat filename.gz> $(basename filename.gz .gz).unpacked

Use the squeeze repeat option (e.g. /t/t --> /t)

tr -s "/t" < filename

Do not print nextline with echo

echo -e 'text here \c'

Use the last argument

!$

Check last exit code

echo $?

View first 50 characters of file

head -c 50 file

Group/combine rows into one row

e.g.
AAAA
BBBB
CCCC
DDDD

cat filename|paste - -
-->
AAAABBBB
CCCCDDDD
cat filename|paste - - - -
-->
AAAABBBBCCCCDDDD

Fastq to fasta

cat file.fastq | paste - - - - | sed 's/^@/>/g'| cut -f1-2 | tr '\t' '\n' >file.fa

Cut and get last column

cat file|rev | cut -d/ -f1 | rev

Add one to variable/increment/ i++ a numeric variable (e.g. $var)

((var++))
or
var=$((var+1))

Some handy environment variables

$0 :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$# :number of positional parameters.
$? :most recent foreground pipeline exit status.
$- :current options set for the shell.
:pid of the current shell (not subshell).
$! :is the PID of the most recent background command.

Clear the contents of a file (e.g. filename)

>filename

Unzip tar.bz2 file (e.g. file.tar.bz2)

tar xvfj file.tar.bz2

Unzip tar.xz file (e.g. file.tar.xz)

unxz file.tar.xz
tar xopf file.tar

Output a y/n repeatedly until killed

'y':

yes

or 'n':

yes n

or 'anything':

yes anything

For example:

yes | rm -r large_directory

Create dummy file of certain size instantly (e.g. 200mb)

dd if=/dev/zero of=//dev/shm/200m bs=1024k count=200
or
dd if=/dev/zero of=//dev/shm/200m bs=1M count=200

Standard output:
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.0955679 s, 2.2 GB/s

Cat to a file

cat >myfile
let me add sth here
exit by control + c
^C

Keep /repeatedly executing the same command (e.g Repeat 'wc -l filename' every 1 second)

watch -n 1 wc -l filename

Print commands and their arguments when execute (e.g. echo expr 10 + 20 )

set -x; echo `expr 10 + 20 `

Print some meaningful sentences to you (install fortune first)

fortune

Colorful (and useful) version of top (install htop first)

htop

Press any key to continue

read -rsp $'Press any key to continue...\n' -n1 key

Run sql-like command on files from terminal

download:
https://github.com/harelba/q
example:

q -d "," "select c3,c4,c5 from /path/to/file.txt where c3='foo' and c5='boo'"

Sreen and tmux

create session and attach:

screen
or  
tmux

create detached session foo:

screen -S foo -d -m	
or  
tmux new -s foo -d

detached session foo:

screen: ^a^d
or  
tmux: ^bd

list sessions:

screen -ls	
or  
tmux ls

attach:

screen -r	
or  
tmux attach

attach to session foo:

screen -r foo	
or 
tmux attach -t foo

kill session foo:

screen -r foo -X quit
or  
tmux kill-session -t foo

scroll: Hit your screen prefix combination (C-a / control+A), then hit Escape. Move up/down with the arrow keys (↑ and ↓).

Send command to all panes in tmux:

Ctrl-B :
setw synchronize-panes

Cut the last column

cat filename|rev|cut -f1|rev

pass password to ssh

sshpass -p mypassword ssh root@10.102.14.88 "df -h"

wait for a pid (job) to complete

wait %1 
or 
wait $PID
wait ${!}
#wait ${!} to wait till the last background process ($! is the PID of the last background process)

pdf to txt

sudo apt-get install poppler-utils
pdftotext example.pdf example.txt

list only directory

ls -ld -- */

Capture/record/save terminal output (capture everything you type and output)

script output.txt
#start using terminal 
#to logout the screen session (stop saving the contents), type exit.

list contents of directories in a tree-like format.

tree
#go to the directory you want to list, and type tree (sudo apt-get install tree)
#output:
#one/
#└── two
#    ├── 1
#    ├── 2
#    ├── 3
#    ├── 4
#    └── 5
#

=-=-=-=-=-A lot more coming!! =-=-=-=-=-=-=-=-=-=waitwait-=-=-=-=-=-=-=-=-=-