2016-07-01 10:59:40 +00:00
# Bash-Oneliner
2018-02-08 02:06:43 +00:00
Hi bash learners and bioinformaticans, welcome to Bash Oneliner. I started studying bioinformatics data four years ago (recently started working on cloud computing), and was amazed by those single-word bash commands which are much faster than my dull scripts, so much time can be saved through knowing command-line shortcuts and scripting. Not all the code here is oneliner (if the ';' counts..), but i put effort on making them brief and fast. I am mainly using Ubuntu, RedHat and Linux Mint, sorry if the commands dont work on your system.
2016-06-14 06:53:59 +00:00
2017-09-15 02:40:51 +00:00
This blog will focus on simple bash commands for parsing data, most of which are for tsv files (tab-separated values); some of them are for Linux system maintenance. I apologize that there won't be any citation for the codes, but they are probably from dear Google and Stackoverflow.
2016-06-14 06:53:59 +00:00
2017-11-23 02:43:12 +00:00
English and bash are not my first language, so... correct me anytime, thank you!
2017-11-23 02:43:28 +00:00
And if you know any cool command that are not included here, Please Teach Me.
2016-06-14 06:53:59 +00:00
2016-09-05 07:50:06 +00:00
In case you would like to check and vote up my questions on Stackoverflow, here's my page:
2016-06-14 06:53:59 +00:00
http://stackoverflow.com/users/4290753/once
2016-06-16 06:55:27 +00:00
2017-09-15 02:41:34 +00:00
Here's a more stylish version of Bash-Oneliner~
2016-07-01 11:28:09 +00:00
http://onceupon.github.io/Bash-Oneliner/
2016-06-16 06:55:27 +00:00
2017-03-29 08:52:59 +00:00
## Handy Bash oneliner commands for tsv file editing
2016-06-14 07:11:00 +00:00
2016-06-14 07:22:55 +00:00
- [Grep ](#grep )
- [Sed ](#sed )
- [Awk ](#awk )
- [Xargs ](#xargs )
- [Find ](#find )
2016-06-16 06:44:57 +00:00
- [Loops ](#loops )
2017-09-12 09:38:43 +00:00
- [Math ](#math )
2016-06-16 06:44:57 +00:00
- [Download ](#download )
2016-06-27 05:54:48 +00:00
- [Random ](#random )
2017-06-09 04:26:34 +00:00
- [Xwindow ](#xwindow )
2016-06-16 06:44:57 +00:00
- [System ](#system )
2018-02-12 06:53:34 +00:00
- [Hardware ](#hardware )
2018-02-08 02:05:09 +00:00
- [Others ](#others )
2016-06-14 07:20:16 +00:00
2017-03-29 08:52:59 +00:00
## Grep
2017-06-09 08:53:36 +00:00
##### Extract text bewteen words (e.g. w1,w2)
2016-06-14 07:20:16 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -o -P '(?< =w1).*(?=w2)'
```
2016-06-14 07:20:16 +00:00
2017-11-01 09:59:03 +00:00
##### Grep only integer
```bash
grep -o '[0-9]*'
```
2017-06-09 08:21:06 +00:00
##### Grep lines without word (e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2016-07-20 04:08:30 +00:00
grep -v bbo filename
```
2017-11-29 08:31:10 +00:00
##### Grep lines not begin with string (e.g. #)
```bash
grep -v '^#' file.txt
```
2017-10-26 08:20:59 +00:00
##### Grep variables with space within it (e.g. bbo="some strings")
```bash
grep "$boo" filename
```
#remember to quote the variable!
2017-06-09 08:21:06 +00:00
##### Grep only one/first match (e.g. bbo)
2016-07-20 04:08:30 +00:00
```bash
grep -m 1 bbo filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep and count (e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -c bbo filename
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Insensitive grep (e.g. bbo/BBO/Bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -i "bbo" filename
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Count occurrence (e.g. three times a line count three times)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -o bbo filename
```
2016-06-14 06:57:38 +00:00
2017-03-29 08:52:59 +00:00
##### COLOR the match (e.g. bbo)!
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep --color bbo filename
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep search all files in a directory(e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -R bbo /path/to/directory
```
2016-06-14 07:55:08 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
grep -r bbo /path/to/directory
```
2017-06-09 08:53:36 +00:00
##### Search all files in directory, only output file names with matches(e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -Rh bbo /path/to/directory
```
or
```bash
grep -rh bbo /path/to/directory
2017-10-11 07:49:28 +00:00
```
or only list filename with match
```bash
grep -rl bbo /path/to/directory
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-10-11 07:49:28 +00:00
2017-06-09 08:53:36 +00:00
##### Grep OR (e.g. A or B or C or D)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```
grep 'A\|B\|C\|D'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep AND (e.g. A and B)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep 'A.*B'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep all content of a fileA from fileB
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep -f fileA fileB
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep a tab
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
grep $'\t'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep variable from variable
2016-08-25 11:40:18 +00:00
```bash
$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
2018-01-10 01:47:26 +00:00
#grep -q will output 0 if match found
#remember to add space between []!
2016-08-25 11:40:18 +00:00
```
2017-06-09 08:53:36 +00:00
##### Grep strings between a bracket()
2016-09-05 07:26:20 +00:00
```bash
grep -oP '\(\K[^\)]+'
```
2017-06-09 08:53:36 +00:00
##### Grep number of characters with known strings in between(e.g. AAEL000001-RA)
2016-09-05 07:26:20 +00:00
```bash
grep -o -w "\w\{10\}\-R\w\{1\}"
2018-01-10 01:47:26 +00:00
# \w word character [0-9a-zA-Z_] \W not word character
2017-03-29 09:54:35 +00:00
```
2018-01-10 01:47:26 +00:00
2016-08-25 11:40:18 +00:00
2017-06-09 08:53:36 +00:00
##### A lot examples here
2016-09-05 07:37:30 +00:00
http://www.cyberciti.biz/faq/grep-regular-expressions/
2017-03-29 08:52:59 +00:00
## Sed
2016-06-14 07:32:20 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2016-06-14 07:25:46 +00:00
2017-10-02 02:38:44 +00:00
##### Remove the 1st line
```bash
sed 1d filename
```
##### Remove the 100 lines (remove line 1-100)
```bash
sed 1,100d filename
```
2017-10-26 08:14:20 +00:00
#####
remove lines with string (e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed "/bbo/d" filename
2017-10-26 08:14:20 +00:00
- case insensitive:
sed "/bbo/Id" filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Edit infile (edit and save)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -i "/bbo/d" filename
```
2017-06-09 08:53:36 +00:00
##### When using variable (e.g. $i), use double quotes " "
2017-03-29 09:54:35 +00:00
e.g. add >$i to the first line (to make a FASTA file)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed "1i >$i"
2018-01-10 01:47:26 +00:00
# notice the double quotes! in other examples, you can use a single quote, but here, no way!
# '1i' means insert to first line
2016-07-01 10:52:19 +00:00
```
2017-10-02 02:38:44 +00:00
##### Delete/remove empty lines
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed '/^\s*$/d'
```
2016-06-14 07:55:08 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
sed 's/^$/d'
```
2017-10-02 02:38:44 +00:00
##### Delete/remove last line
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed '$d'
```
2016-06-14 06:57:38 +00:00
2017-10-02 02:38:44 +00:00
##### Delete/remove last character from end of file
2016-07-13 08:37:37 +00:00
```bash
sed -i '$ s/.$//' filename
```
2017-06-09 08:53:36 +00:00
##### Add string to end of file (e.g. "]")
2016-07-27 02:34:04 +00:00
2016-07-13 08:37:37 +00:00
```bash
sed '$s/$/]/' filename
```
2017-06-09 08:53:36 +00:00
##### Add newline to the end
2017-05-16 07:52:33 +00:00
```bash
2017-05-10 08:18:56 +00:00
sed '$a\'
```
2016-07-13 08:37:37 +00:00
2017-06-09 08:53:36 +00:00
##### Add string to beginning of every line (e.g. bbo)
2016-09-09 10:14:09 +00:00
```bash
sed -e 's/^/bbo/' file
```
2017-06-09 08:53:36 +00:00
##### Add string to end of each line (e.g. "}")
2016-07-13 08:29:31 +00:00
```bash
2016-07-13 08:37:37 +00:00
sed -e 's/$/\}\]/' filename
2016-07-13 08:29:31 +00:00
```
2017-06-09 08:53:36 +00:00
##### Add \n every nth character (e.g. every 4th character)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed 's/.\{4\}/& \n/g'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Concatenate/combine/join files with a seperator and next line (e.g seperate by ",")
2016-07-27 02:34:04 +00:00
```bash
sed -s '$a,' *.json > all.json
```
2017-06-09 08:53:36 +00:00
##### Substitution (e.g. replace A by B)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed 's/A/B/g' filename
```
2017-12-29 09:30:49 +00:00
##### Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)
```bash
sed "s/aaa=.*/aaa=\/my\/new\/path/g"
```
2017-06-09 08:53:36 +00:00
##### Select lines start with string (e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -n '/^@S/p'
```
2017-06-09 08:53:36 +00:00
##### Delete lines with string (e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed '/bbo/d' filename
```
2017-11-24 08:50:22 +00:00
##### Print/get/trim a range of line (e.g. line 500-5000)
```bash
sed -n 500,5000p filename
```
2017-06-09 08:53:36 +00:00
##### Print every nth lines
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -n '0~3p' filename
2018-01-10 01:47:26 +00:00
# catch 0: start; 3: step
2016-07-01 10:52:19 +00:00
```
2018-01-10 01:47:26 +00:00
2016-06-14 06:57:38 +00:00
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### Print every odd # lines
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -n '1~2p'
```
2017-06-09 08:53:36 +00:00
##### Print every third line including the first line
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -n '1p;0~3p'
```
2017-06-09 08:53:36 +00:00
##### Remove leading whitespace and tabs
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -e 's/^[ \t]*//'
```
2016-06-14 06:57:38 +00:00
//notice a whitespace before '\t'!!
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### Remove only leading whitespace
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed 's/ *//'
2016-06-14 08:03:32 +00:00
2018-01-10 01:47:26 +00:00
# notice a whitespace before '*'!!
```
2017-06-09 08:53:36 +00:00
##### Remove ending commas
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed 's/,$//g'
```
2017-06-09 08:53:36 +00:00
##### Add a column to the end
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed "s/$/\t$i/"
2018-01-10 01:47:26 +00:00
# $i is the valuable you want to add
# e.g. add the filename to every last column of the file
2016-07-01 10:52:19 +00:00
```
2018-01-10 01:47:26 +00:00
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
for i in $(ls);do sed -i "s/$/\t$i/" $i;done
```
2017-06-09 08:53:36 +00:00
##### Add extension of filename to last column
2016-08-17 10:05:54 +00:00
```bash
for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Remove newline\ nextline
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed ':a;N;$!ba;s/\n//g'
```
2016-06-14 06:57:38 +00:00
2017-09-28 10:10:55 +00:00
##### Print a particular line (e.g. 123th line)
```bash
sed -n -e '123p'
```
2017-06-09 08:53:36 +00:00
##### Print a number of lines (e.g. line 10th to line 33 rd)
2016-07-01 06:47:42 +00:00
2016-07-01 10:52:19 +00:00
```bash
sed -n '10,33p' < filename
```
2016-07-01 06:47:42 +00:00
2017-06-09 08:53:36 +00:00
##### Change delimiter
2016-09-01 09:24:27 +00:00
```bash
sed 's=/=\\/=g'
```
2017-06-09 08:53:36 +00:00
##### Replace with wildcard (e.g A-1-e or A-2-e or A-3-e....)
2016-10-13 09:57:34 +00:00
```bash
sed 's/A-.*-e//g' filename
```
2017-06-09 08:53:36 +00:00
##### Remove last character of file
2018-01-03 04:21:42 +00:00
```bash
2017-02-10 07:00:54 +00:00
sed '$ s/.$//'
```
2016-10-13 09:57:34 +00:00
2017-06-09 08:53:36 +00:00
##### Insert character at specified position of file (e.g. AAAAAA --> AAA#AAA)
2018-01-03 04:21:42 +00:00
```bash
2017-02-13 08:15:38 +00:00
sed -r -e 's/^.{3}/& #/' file
```
2016-06-14 06:57:38 +00:00
2017-05-10 08:18:56 +00:00
2017-03-29 08:52:59 +00:00
## Awk
2016-06-14 07:55:08 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2017-06-09 08:53:36 +00:00
##### Set tab as field separator
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk -F $'\t'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Output as tab separated (also as field separator)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk -v OFS='\t'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Pass variable
2016-06-14 06:57:38 +00:00
2016-07-01 10:52:19 +00:00
```bash
a=bbo;b=obb;
2017-09-15 02:55:22 +00:00
awk -v a="$a" -v b="$b" "$1==a & & $10=b" filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2017-06-09 08:53:36 +00:00
##### Print line number and number of characters on each line
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2017-05-10 11:22:41 +00:00
awk '{print NR,length($0);}' filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Find number of columns
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk '{print NF}'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Reverse column order
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk '{print $2, $1}'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Check if there is a comma in a column (e.g. column $1)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk '$1~/,/ {print}'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Split and do for loop
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2017-09-15 03:07:16 +00:00
awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Print all lines before nth occurence of a string (e.g stop print lines when bbo appears 7 times)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk -v N=7 '{print}/bbo/& & --N< =0 {exit}'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Print filename and last line of all files in directory
2016-07-15 09:12:52 +00:00
```bash
2017-09-15 03:07:16 +00:00
ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file
2016-07-15 09:12:52 +00:00
```
2017-06-09 08:53:36 +00:00
##### Add string to the beginning of a column (e.g add "chr" to column $3)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk 'BEGIN{OFS="\t"}$3="chr"$3'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Remove lines with string (e.g. bbo)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk '!/bbo/' file
```
2016-06-14 06:57:38 +00:00
2018-02-09 02:36:52 +00:00
##### Remove last column
```bash
awk 'NF{NF-=1};1' file
```
2017-06-09 08:53:36 +00:00
##### Usage and meaning of NR and FNR
2017-03-29 09:54:35 +00:00
e.g.
fileA:
a
b
c
fileB:
d
e
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk 'print FILENAME, NR,FNR,$0}' fileA fileB
```
2017-03-29 09:54:35 +00:00
fileA 1 1 a
fileA 2 2 b
fileA 3 3 c
fileB 4 1 d
fileB 5 2 e
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### AND gate
2016-06-14 08:03:32 +00:00
2017-03-29 09:54:35 +00:00
e.g.
fileA:
1 0
2 1
3 1
4 0
2016-06-14 08:03:32 +00:00
2017-03-29 09:54:35 +00:00
fileB:
1 0
2 1
3 0
4 1
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB
```
2016-06-14 08:03:32 +00:00
2017-03-29 09:54:35 +00:00
1 0
2 1
3 0
4 0
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Round all numbers of file (e.g. 2 significant figure)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk '{while (match($0, /[0-9]+\[0-9]+/)){
2016-06-14 07:55:08 +00:00
\printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
\$0=substr($0, RSTART+RLENGTH)
\}
\print
\}'
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2017-06-09 08:53:36 +00:00
##### Give number/index to every row
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
awk '{printf("%s\t%s\n",NR,$0)}'
```
2017-06-09 08:53:36 +00:00
##### Break combine column data into rows
2016-08-18 03:03:45 +00:00
2017-03-29 09:54:35 +00:00
e.g.
seperate
2016-08-18 03:04:58 +00:00
2017-03-29 09:54:35 +00:00
David cat,dog
into
David cat
David dog
2016-08-18 03:04:58 +00:00
2016-08-18 03:03:45 +00:00
detail here: http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string
```bash
awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' file
```
2017-06-09 08:53:36 +00:00
##### Average a file (each line in file contains only one number)
2016-09-05 07:37:30 +00:00
```bash
awk '{s+=$1}END{print s/NR}'
```
2017-06-09 08:53:36 +00:00
##### Print field start with string (e.g Linux)
2016-09-02 03:29:05 +00:00
2016-09-01 09:24:27 +00:00
```bash
awk '$1 ~ /^Linux/'
```
2017-06-09 08:53:36 +00:00
##### Sort a row (e.g. 1 40 35 12 23 --> 1 12 23 35 40)
2016-09-02 03:29:05 +00:00
```bash
awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i < = length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'
```
2016-09-01 09:24:27 +00:00
2017-06-16 09:53:05 +00:00
##### Subtract previous row values (add column6 which equal to column4 minus last column5)
```bash
awk '{$6 = $4 - prev5; prev5 = $5; print;}'
```
2016-06-14 06:57:38 +00:00
2017-03-29 08:52:59 +00:00
## Xargs
2016-06-14 07:55:08 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2016-06-28 02:59:23 +00:00
2017-06-09 08:53:36 +00:00
##### Set tab as delimiter (default:space)
2016-06-14 06:57:38 +00:00
2016-07-01 10:52:19 +00:00
```bash
xargs -d\t
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Display 3 items per line
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
echo 1 2 3 4 5 6| xargs -n 3
2018-01-10 01:47:26 +00:00
# 1 2 3
# 4 5 6
2016-06-14 08:03:32 +00:00
2018-01-10 01:47:26 +00:00
```
2017-06-09 08:53:36 +00:00
##### Prompt before execution
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
echo a b c |xargs -p -n 3
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Print command along with output
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
xargs -t abcd
2018-01-10 01:47:26 +00:00
# bin/echo abcd
# abcd
2016-06-14 08:03:32 +00:00
2018-01-10 01:47:26 +00:00
```
2017-06-09 08:53:36 +00:00
##### With find and rm
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.html"|xargs rm -rf
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Delete fiels with whitespace in filename (e.g. "hello 2001")
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.c" -print0|xargs -0 rm -rf
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Show limits
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
xargs --show-limits
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Move files to folder
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old
```
2016-06-14 07:55:08 +00:00
2016-06-14 06:57:38 +00:00
or
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Move first 100th files to a directory (e.g. d1)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
ls |head -100|xargs -I {} mv {} d1
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Parallel
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
time echo {1..5} |xargs -n 1 -P 5 sleep
```
a lot faster than
```bash
time echo {1..5} |xargs -n1 sleep
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Copy all files from A to B
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B
2018-01-10 01:47:26 +00:00
# v: verbose|
# p: keep detail (e.g. owner)
2016-06-14 06:57:38 +00:00
2018-01-10 01:47:26 +00:00
```
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### With sed
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
ls |xargs -n1 -I file sed -i '/^Pos/d' filename
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Add the file name to the first line of file
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Count all files
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
ls |xargs -n1 wc -l
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Turn output into a single line
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
ls -l| xargs
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Count files within directories
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
echo mso{1..8}|xargs -n1 bash -c 'echo -n "$1:"; ls -la "$1"| grep -w 74 |wc -l' --
2016-06-14 06:57:38 +00:00
2018-01-10 01:47:26 +00:00
# "--" signals the end of options and display further option processing
```
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### Download dependencies files and install (e.g. requirements.txt)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
cat requirements.txt| xargs -n1 sudo pip install
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Count lines in all file, also count total lines
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
ls|xargs wc -l
```
2017-06-09 08:53:36 +00:00
##### Xargs and grep
2016-06-14 06:57:38 +00:00
2016-09-29 09:17:45 +00:00
```bash
cat grep_list |xargs -I{} grep {} filename
```
2016-06-14 06:57:38 +00:00
2017-10-11 07:49:28 +00:00
##### Xargs and sed (replace all old ip address with new ip address under /etc directory)
```bash
grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'
```
2017-03-29 08:52:59 +00:00
## Find
2016-06-14 07:55:08 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2017-06-09 08:53:36 +00:00
##### List all sub directory/file in the current directory
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find .
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### List all files under the current directory
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -type f
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### List all directories under the current directory
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -type d
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Edit all files under current directory (e.g. replace 'www' with 'ww')
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . name '*.php' -exec sed -i 's/www/w/g' {} \;
```
2016-06-14 07:55:08 +00:00
if no subdirectory
2016-07-01 10:52:19 +00:00
```bash
replace "www" "w" -- *
2018-01-10 01:47:26 +00:00
# a space before *
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### Find and output only filename (e.g. "mso")
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find mso*/ -name M* -printf "%f\n"
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Find and delete file with size less than (e.g. 74 byte)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.mso" -size -74c -delete
2018-01-10 01:47:26 +00:00
# M for MB, etc
2016-07-01 10:52:19 +00:00
```
2018-01-10 01:47:26 +00:00
2016-06-14 06:57:38 +00:00
2017-03-29 08:52:59 +00:00
## Loops
2016-06-16 06:44:57 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2017-06-09 08:53:36 +00:00
##### While loop, column subtraction of a file (e.g. a 3 columns file)
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
while read a b c; do echo $(($c-$b));done < < (head filename)
2016-06-16 06:44:57 +00:00
2018-01-10 01:47:26 +00:00
#there is a space between the two '<'s
```
2017-06-09 08:53:36 +00:00
##### While loop, sum up column subtraction
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < < (head filename)
```
2016-06-16 06:44:57 +00:00
2017-11-23 02:40:30 +00:00
##### While loop, keep checking a running process (e.g. perl) and start another new process (e.g. python) immetiately after it. (BETTER use the wait command! Ctrl+F 'wait')
2017-10-26 10:29:14 +00:00
```bash
while [[ $(pidof perl) ]];do echo f;sleep 10;done & & python timetorunpython.py
```
2017-06-09 08:53:36 +00:00
##### If loop
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
if (($j==$u+2))
2018-01-10 01:47:26 +00:00
#(( )) use for arithmetic operation
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
if [[$age >21]]
2018-01-10 01:47:26 +00:00
#[[ ]] use for comparison
2016-07-01 10:52:19 +00:00
```
2017-01-05 07:44:41 +00:00
2017-06-09 08:53:36 +00:00
##### Test if file exist
2017-01-05 07:44:41 +00:00
```bash
2017-06-09 08:53:36 +00:00
if [ -e 'filename' ]
2017-01-05 07:44:41 +00:00
then
2017-01-05 07:45:36 +00:00
echo -e "file exists!"
2017-01-05 07:44:41 +00:00
fi
```
2017-10-25 09:59:22 +00:00
##### if else Test if file exist
```bash
if [ -e $filename ]; then echo -e "file exists!"; else mkdir $filename; fi
```
2017-01-05 07:44:41 +00:00
2017-06-09 08:53:36 +00:00
##### For loop
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
for i in $(ls); do echo file $i;done
```
2016-06-16 06:44:57 +00:00
2017-09-26 01:12:22 +00:00
##### for loop, press any key to continue each loop
```bash
2017-09-26 01:16:16 +00:00
for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done
2017-09-26 01:12:22 +00:00
```
2017-10-25 06:44:16 +00:00
##### for loop, print a file line by line when a key is pressed
2017-10-25 06:43:51 +00:00
```bash
for line in $(cat myfile); do echo $line; read -n1; done
```
2017-09-12 09:38:43 +00:00
## Math
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
##### Print out the prime factors of a number (e.g. 50)
```bash
factor 50
```
2017-12-12 02:58:22 +00:00
##### Sum up input list (e.g. seq 10)
```bash
seq 10|paste -sd+|bc
```
2017-12-12 02:59:35 +00:00
##### Sum up a file (each line in file contains only one number)
```bash
awk '{s+=$1} END {print s}' filename
```
2017-12-12 02:58:22 +00:00
##### Column subtraction
```bash
cat file| awk -F '\t' 'BEGIN {SUM=0}{SUM+=$3-$2}END{print SUM}'
```
2017-09-12 09:38:43 +00:00
##### Simple math with expr
```bash
expr 10+20 #30
expr 10\*20 #600
expr 30 \> 20 #1 (true)
```
##### More math with bc
- Number of decimal digit/ significant figure
```bash
echo "scale=2;2/3" | bc
#.66
```
- Exponent operator
```bash
echo "10^2" | bc
#100
```
- Using variables
```bash
echo "var=5;--var"| bc
#4
```
2017-03-29 08:52:59 +00:00
## Download
2016-06-16 06:44:57 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2017-06-09 08:53:36 +00:00
##### Download all from a page
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
wget -r -l1 -H -t1 -nd -N -np -A mp3 -e robots=off http://example.com
2018-01-10 01:47:26 +00:00
# -r: recursive and download all links on page
# -l1: only one level link
# -H: span host, visit other hosts
# -t1: numbers of retries
# -nd: don't make new directories, download to here
# -N: turn on timestamp
# -nd: no parent
# -A: type (seperate by ,)
# -e robots=off: ignore the robots.txt file which stop wget from crashing the site, sorry example.com
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2017-06-09 08:53:36 +00:00
##### Upload a file to web and download (https://transfer.sh/)
--> upload:
```bash
curl --upload-file ./filename.txt https://transfer.sh/filename.txt
```
(the above command will return a URL, e.g: https://transfer.sh/tG8rM/filename.txt)
--> download:
```bash
curl https://transfer.sh/tG8rM/filename.txt -o filename.txt
```
##### Download file if necessary
```bash
data=file.txt
url=http://www.example.com/$data
if [! -s $data];then
echo "downloading test data..."
wget $url
fi
```
##### Wget to a filename (when a long name)
```bash
wget -O filename "http://example.com"
```
##### Wget files to a folder
```bash
wget -P /path/to/directory "http://example.com"
```
2016-06-16 06:44:57 +00:00
2016-06-27 05:54:48 +00:00
2017-03-29 08:52:59 +00:00
## Random
2016-06-27 05:54:48 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2017-06-09 08:53:36 +00:00
##### Random pick 100 lines from a file
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
shuf -n 100 filename
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Random order (lucky draw)
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
for i in a b c d e; do echo $i; done| shuf
```
2016-06-27 05:54:48 +00:00
2017-06-28 04:31:30 +00:00
##### Echo series of random numbers between a range (e.g. shuffle numbers from 0-100, then pick 15 of them randomly)
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
2017-06-28 04:31:30 +00:00
shuf -i 0-100 -n 15
2016-07-01 10:52:19 +00:00
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Echo a random number
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
echo $RANDOM
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Random from 0-9
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
echo $((RANDOM % 10))
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Random from 1-10
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
echo $(((RANDOM %10)+1))
```
2016-06-27 05:54:48 +00:00
2017-06-09 04:26:34 +00:00
## Xwindow
2017-09-28 05:40:02 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2017-06-09 04:26:34 +00:00
X11 GUI applications! Here are some GUI tools for you if you get bored by the text-only environment.
2017-06-09 08:53:36 +00:00
##### Enable X11 forwarding,in order to use graphical application on servers
2017-06-09 04:26:34 +00:00
```bash
ssh -X user_name@ip_address
```
or setting through xhost
--> Install the following for Centos:
xorg-x11-xauth
xorg-x11-fonts-*
xorg-x11-utils
2017-06-09 08:53:36 +00:00
##### Little xwindow tools
2017-06-09 04:26:34 +00:00
```bash
xclock
xeyes
2017-09-14 04:20:58 +00:00
xcowsay
2017-06-09 04:26:34 +00:00
```
2017-06-09 08:53:36 +00:00
##### Open pictures/images from ssh server
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install eog
3. eog picture.png
```
##### Use gedit on server (GUI editor)
```bash
1. ssh -X user_name@ip_address
2. apt-get install gedit
3. gedit filename.txt
```
2017-06-09 08:53:36 +00:00
##### Open PDF file from ssh server
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install evince
3. evince filename.pdf
```
2017-06-09 08:53:36 +00:00
##### Use google-chrome browser from ssh server
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install libxss1 libappindicator1 libindicator7
3. wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
4. sudo apt-get install -f
5. dpkg -i google-chrome*.deb
6. google-chrome
```
2018-02-08 02:00:06 +00:00
## System
2017-06-09 04:26:34 +00:00
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
2018-02-08 02:00:06 +00:00
##### Snapshot of the current processes
2016-06-27 05:54:48 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
ps
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Check graphics card
2017-04-14 09:19:59 +00:00
```bash
2018-02-08 02:00:06 +00:00
lspci
2017-04-14 09:19:59 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Show IP address
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
$ip add show
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
or
2018-02-08 02:00:06 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
ifconfig
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### Check system version
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat /etc/*-release
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### Linux Programmer's Manuel: hier- description of the filesystem hierarchy
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
man hier
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### List job
2017-12-29 02:14:00 +00:00
```bash
2018-02-08 02:00:06 +00:00
jobs -l
2017-12-29 02:14:00 +00:00
```
2018-02-08 02:00:06 +00:00
##### Export PATH
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
export PATH=$PATH:~/path/you/want
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Make file execuable
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
chmod +x filename
# you can now ./filename to execute it
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### List screen
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen -d -r
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Echo screen name
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen -ls
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Check system (x86-64)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
uname -i
2016-09-08 09:31:55 +00:00
```
2018-02-08 02:00:06 +00:00
##### Surf the net
2016-09-08 09:31:55 +00:00
```bash
2018-02-08 02:00:06 +00:00
links www.google.com
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2018-02-08 02:00:06 +00:00
##### Add user, set passwd
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
useradd username
passwd username
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2018-02-08 02:00:06 +00:00
##### Edit variable for bash, (e.g. displaying the whole path)
```bash
1. joe ~/.bash_profile
2. export PS1='\u@\h:\w\$'
# $PS1 is a variable that defines the makeup and style of the command prompt
```
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
3. source ~/.bash_profile
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### Edit environment setting (e.g. alias)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
1. joe ~/.bash_profile
2. alias pd="pwd" //no more need to type that 'w'!
3. source ~/.bash_profile
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### List environment variables (e.g. PATH)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
$echo $PATH
# list of directories separated by a colon
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### List all environment variables for current user
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
$env
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Show partition format
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
lsblk
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Soft link program to bin
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
ln -s /path/to/program /home/usr/bin
# must be the whole path to the program
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Show hexadecimal view of data
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
hexdump -C filename.class
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2018-02-08 02:00:06 +00:00
##### Jump to different node
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
rsh node_name
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Check port (active internet connection)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
netstat -tulpn
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Find whick link to a file
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
readlink filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2018-02-08 02:00:06 +00:00
##### Check where a command link to (e.g. python)
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
which python
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### List total size of a directory
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
du -hs .
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
du -sb
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Copy directory with permission setting
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cp -rp /path/to/directory
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Store current directory
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
pushd . $popd ;dirs -l
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2018-02-08 02:00:06 +00:00
##### Show disk usage
```bash
df -h
```
2016-06-14 06:57:38 +00:00
or
2018-02-08 02:00:06 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
du -h
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
du -sk /var/log/* |sort -rn |head -10
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Show current runlevel
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
runlevel
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2018-02-08 02:00:06 +00:00
##### Switch runlevel
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
init 3
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
telinit 3
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Permanently modify runlevel
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
1. edit /etc/init/rc-sysinit.conf
2. env DEFAULT_RUNLEVEL=2
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### Become root
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
su
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Become somebody
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
su somebody
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Report user quotes on device
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
requota -auvs
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### Get entries in a number of important databases
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
getent database_name
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
(e.g. the 'passwd' database)
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
getent passwd
# list all user account (all local and LDAP)
# (e.g. fetch list of grop accounts)
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
getent group
# store in database 'group'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Change owner of file
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
chown user_name filename
chown -R user_name /path/to/directory/
# chown user:group filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### List current mount detail
2016-06-14 07:55:08 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
df
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### List current usernames and user-numbers
2016-06-14 08:03:32 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat /etc/passwd
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Get all username
2016-06-14 08:03:32 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
getent passwd| awk '{FS="[:]"; print $1}'
2016-07-01 10:52:19 +00:00
```
2018-01-10 01:47:26 +00:00
2018-02-08 02:00:06 +00:00
##### Show all users
2016-06-14 08:03:32 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
compgen -u
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2018-02-08 02:00:06 +00:00
##### Show all groups
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
compgen -g
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Show group of user
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
group username
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Show uid, gid, group of user
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
id username
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Check if it's root
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
if [$(id -u) -ne 0];then
echo "You are not root!"
exit;
fi
# 'id -u' output 0 if it's not root
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Find out CPU information
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
more /proc/cpuinfo
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2017-10-27 03:13:54 +00:00
```bash
2018-02-08 02:00:06 +00:00
lscpu
2017-10-27 03:13:54 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Set quota for user (e.g. disk soft limit: 120586240; hard limit: 125829120)
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
setquota username 120586240 125829120 0 0 /home
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Show quota for user
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
quota -v username
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Fork bomb
# dont try this at home
```bash
:(){:|:&}; :
```
##### Check user login
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
lastlog
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Edit path for all users
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
joe /etc/environment
# edit this file
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Show running processes
2016-06-16 06:44:57 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
ps aux
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Find maximum number of processes
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat /proc/sys/kernal/pid_max
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Show and set user limit
```bash
ulimit -u
```
##### Which ports are listening for TCP connections from the network
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
nmap -sT -O localhost
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Print out number of cores/ processors
2016-08-25 11:57:18 +00:00
```bash
2018-02-08 02:00:06 +00:00
nproc --all
2016-08-25 11:57:18 +00:00
```
2018-02-08 02:00:06 +00:00
##### Check status of each core
1. top
2. press '1'
##### Show jobs and PID
2016-08-25 11:57:18 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
jobs -l
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### List all running services
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
service --status-all
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2018-02-08 02:00:06 +00:00
##### Schedule shutdown server
2016-09-02 03:56:37 +00:00
```bash
2018-02-08 02:00:06 +00:00
shutdown -r +5 "Server will restart in 5 minutes. Please save your work."
2016-09-02 03:56:37 +00:00
```
2018-02-08 02:00:06 +00:00
##### Cancel scheduled shutdown
2018-01-03 05:56:20 +00:00
```bash
2018-02-08 02:00:06 +00:00
shutdown -c
2018-01-03 05:56:20 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Boardcast to all users
2016-09-23 03:15:49 +00:00
```bash
2018-02-08 02:00:06 +00:00
wall -n hihi
2016-09-23 03:15:49 +00:00
```
2018-02-08 02:00:06 +00:00
##### Kill all process of a user
2016-10-06 08:39:20 +00:00
```bash
2018-02-08 02:00:06 +00:00
pkill -U user_name
2016-10-06 08:39:20 +00:00
```
2018-02-08 02:00:06 +00:00
##### Set gedit preference on server
2016-10-06 08:39:20 +00:00
2018-02-08 02:00:06 +00:00
-->you might have to install the following:
apt-get install libglib2.0-bin;
yum install dconf dconf-editor;
yum install dbus dbus-x11;
-->Check list
2016-11-07 07:21:55 +00:00
```bash
2018-02-08 02:00:06 +00:00
gsettings list-recursively
2016-11-07 07:21:55 +00:00
```
2018-02-08 02:00:06 +00:00
-->Change setting
e.g.
2017-01-05 07:41:58 +00:00
```bash
2018-02-08 02:00:06 +00:00
gsettings set org.gnome.gedit.preferences.editor highlight-current-line true
gsettings set org.gnome.gedit.preferences.editor scheme 'cobalt'
gsettings set org.gnome.gedit.preferences.editor use-default-font false
gsettings set org.gnome.gedit.preferences.editor editor-font 'Cantarell Regular 12'
2017-01-05 07:41:58 +00:00
```
2018-02-08 02:00:06 +00:00
##### Find out who has logged in on your system
--> [Quick] Printing out only the names:
2017-01-05 07:41:58 +00:00
```bash
2018-02-08 02:00:06 +00:00
users
2017-01-05 07:41:58 +00:00
```
2018-02-08 02:00:06 +00:00
--> [Detail] Printing out login time, load average, etc
2017-01-05 07:41:58 +00:00
```bash
2018-02-08 02:00:06 +00:00
w
2017-01-05 07:41:58 +00:00
```
2018-02-08 02:00:06 +00:00
##### Add user to a group (e.g add user 'nice' to the group 'docker', so that he can run docker without sudo)
2017-01-18 09:52:44 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo gpasswd -a nice docker
2017-01-18 09:52:44 +00:00
```
2017-01-23 06:27:30 +00:00
2018-02-08 02:00:06 +00:00
##### pip install python package without root
2017-01-23 06:26:35 +00:00
```bash
2018-02-08 02:00:06 +00:00
1. pip install --user package_name
2. You might need to export ~/.local/bin/ to PATH: export PATH=$PATH:~/.local/bin/
2017-01-23 06:26:35 +00:00
```
2018-02-08 02:00:06 +00:00
##### Removing old linux kernels (when /boot almost full...)
2017-01-23 06:26:35 +00:00
```bash
2018-02-08 02:00:06 +00:00
1. uname -a #check current kernel, which should NOT be removed
2. sudo apt-get purge linux-image-X.X.X-X-generic #replace old version
2017-01-23 06:26:35 +00:00
```
2018-02-08 02:00:06 +00:00
##### Change hostname
2017-02-16 12:25:13 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo hostname your-new-name
2017-02-16 12:25:13 +00:00
```
2018-02-08 02:00:06 +00:00
if not working, do also:
2017-02-17 03:25:34 +00:00
```bash
2018-02-08 02:00:06 +00:00
hostnamectl set-hostname your-new-hostname
2017-02-17 03:25:34 +00:00
```
2018-02-08 02:00:06 +00:00
then run:
hostnamectl
2017-01-23 06:26:35 +00:00
2018-02-08 02:00:06 +00:00
check /etc/hostname
2017-03-29 09:48:14 +00:00
2018-02-08 02:00:06 +00:00
if still not working..., edit:
/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-ensxxx
add HOSTNAME="your-new-hostname"
##### List installed packages
2017-04-17 04:02:43 +00:00
```bash
2018-02-08 02:00:06 +00:00
apt list --installed
2017-04-17 04:02:43 +00:00
```
2018-02-08 02:00:06 +00:00
or Red Hat:
2017-06-16 09:01:53 +00:00
```bash
2018-02-08 02:00:06 +00:00
yum list installed
2017-06-16 09:01:53 +00:00
```
2018-02-08 02:00:06 +00:00
##### Check which file make the device busy on umount
2018-01-23 08:41:00 +00:00
```bash
2018-02-08 02:00:06 +00:00
lsof /mnt/dir
2018-01-23 08:41:00 +00:00
```
2018-02-08 02:00:06 +00:00
##### When sound not working
2017-06-28 12:27:43 +00:00
```bash
2018-02-08 02:00:06 +00:00
killall pulseaudio
2017-06-28 12:27:43 +00:00
```
2018-02-08 02:00:06 +00:00
then press Alt-F2 and type in pulseaudio
##### When sound not working
2017-06-28 12:27:43 +00:00
```bash
2018-02-08 02:00:06 +00:00
killall pulseaudio
2017-08-16 06:28:50 +00:00
```
2018-02-08 02:00:06 +00:00
##### List information about SCSI devices
2017-08-16 06:20:57 +00:00
```bash
2018-02-08 02:00:06 +00:00
lsscsi
```
##### Tutorial for setting up your own DNS server
http://onceuponmine.blogspot.tw/2017/08/set-up-your-own-dns-server.html
2017-09-08 01:22:48 +00:00
2018-02-08 02:00:06 +00:00
##### Tutorial for creating a simple daemon
http://onceuponmine.blogspot.tw/2017/07/create-your-first-simple-daemon.html
2017-09-08 01:22:48 +00:00
2018-02-08 02:00:06 +00:00
##### Tutorial for using your gmail to send email
http://onceuponmine.blogspot.tw/2017/10/setting-up-msmtprc-and-use-your-gmail.html
2017-08-23 00:50:24 +00:00
2018-02-08 02:00:06 +00:00
##### Using telnet to test open ports, test if you can connect to a port (e.g 53) of a server (e.g 192.168.2.106)
2017-09-12 09:38:43 +00:00
```bash
2018-02-08 02:00:06 +00:00
telnet 192.168.2.106 53
```
##### change network maximum transmission unit (mtu) (e.g. change to 9000)
2017-09-12 09:38:43 +00:00
```bash
2018-02-08 02:00:06 +00:00
ifconfig eth0 mtu 9000
```
##### get pid of a running process (e.g python)
2017-09-26 01:12:22 +00:00
```bash
2018-02-08 02:00:06 +00:00
pidof python
```
or
2017-09-28 05:28:23 +00:00
```bash
2018-02-08 02:00:06 +00:00
ps aux|grep python
```
##### ntp
start ntp:
2017-09-28 05:34:16 +00:00
```bash
2018-02-08 02:00:06 +00:00
ntpd
```
check ntp:
2017-09-28 05:28:23 +00:00
```bash
2018-02-08 02:00:06 +00:00
ntpq -p
```
##### remove unnecessary files to clean your server
2017-10-24 05:46:22 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo apt-get autoremove
sudo apt-get clean
sudo rm -rf ~/.cache/thumbnails/*
```
Remove old kernal:
2017-09-28 05:28:23 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo dpkg --list 'linux-image*'
sudo apt-get remove linux-image-OLDER_VERSION
```
2017-09-28 05:28:23 +00:00
2018-02-08 02:00:06 +00:00
##### Increase/ resize root partition (root partition is an LVM logical volume)
2017-09-28 05:28:23 +00:00
```bash
2018-02-08 02:00:06 +00:00
pvscan
lvextend -L +130G /dev/rhel/root -r
```
#Adding -r will grow filesystem after resizing the volume.
2017-09-28 05:28:23 +00:00
2017-09-28 10:10:55 +00:00
2018-02-08 02:00:06 +00:00
##### Create a UEFI Bootable USB drive (e.g. /dev/sdc1)
2017-10-19 06:48:57 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo dd if=~/path/to/isofile.iso of=/dev/sdc1 oflag=direct bs=1048576
2017-11-23 02:40:30 +00:00
```
2018-02-08 02:00:06 +00:00
##### Locate and remove a package
2017-11-23 02:40:30 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo dpkg -l | grep < package_name >
sudo dpkg --purge < package_name >
2017-10-19 06:48:57 +00:00
```
2018-02-08 02:00:06 +00:00
##### Create a ssh tunnel
2017-11-29 06:00:46 +00:00
```bash
2018-02-08 02:00:06 +00:00
ssh -f -L 9000:targetservername:8088 root@192.168.14.72 -N
#-f: run in background; -L: Listen; -N: do nothing
#the 9000 of your computer is now connected to the 8088 port of the targetservername through 192.168.14.72
#so that you can see the content of targetservername:8088 by entering localhost:9000 from your browser.
2017-11-29 06:00:46 +00:00
```
2018-02-12 06:53:34 +00:00
##### Log out your account after a certain period of time (e.g 10 seconds)
2018-01-02 02:17:50 +00:00
```bash
2018-02-12 06:53:34 +00:00
TMOUT=10
#once you set this variable, logout timer start running!
```
##### Get process ID of a process (e.g. sublime_text)
```bash
#pidof
pidof sublime_text
#pgrep, you dont have to type the whole program name
pgrep sublim
#top, takes longer time
top|grep sublime_text
```
## Hardware
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
##### Finding Out memory device detail
```bash
sudo dmidecode -t memory
2018-01-02 02:17:50 +00:00
```
2017-09-28 05:28:23 +00:00
2018-02-08 02:00:06 +00:00
##### Print detail of CPU hardware
2018-01-03 04:20:55 +00:00
```bash
2018-02-08 02:00:06 +00:00
dmidecode -t 4
# Type Information
# 0 BIOS
# 1 System
# 2 Base Board
# 3 Chassis
# 4 Processor
# 5 Memory Controller
# 6 Memory Module
# 7 Cache
# 8 Port Connector
# 9 System Slots
# 11 OEM Strings
# 13 BIOS Language
# 15 System Event Log
# 16 Physical Memory Array
# 17 Memory Device
# 18 32-bit Memory Error
# 19 Memory Array Mapped Address
# 20 Memory Device Mapped Address
# 21 Built-in Pointing Device
# 22 Portable Battery
# 23 System Reset
# 24 Hardware Security
# 25 System Power Controls
# 26 Voltage Probe
# 27 Cooling Device
# 28 Temperature Probe
# 29 Electrical Current Probe
# 30 Out-of-band Remote Access
# 31 Boot Integrity Services
# 32 System Boot
# 34 Management Device
# 35 Management Device Component
# 36 Management Device Threshold Data
# 37 Memory Channel
# 38 IPMI Device
# 39 Power Supply
2018-01-03 04:20:55 +00:00
```
2018-02-12 06:53:34 +00:00
##### Count the number of Segate hard disks
2018-01-17 05:36:56 +00:00
```bash
2018-02-12 06:53:34 +00:00
lsscsi|grep SEAGATE|wc -l
or
sg_map -i -x|grep SEAGATE|wc -l
2018-01-17 05:36:56 +00:00
```
2018-02-12 06:53:34 +00:00
##### Print detail of all hard disks
2016-07-01 10:52:19 +00:00
```bash
2018-02-12 06:53:34 +00:00
lsblk -io KNAME,TYPE,MODEL,VENDOR,SIZE,ROTA
#where ROTA means rotational device / spinning hard disks (1 if true, 0 if false)
```
##### List information about NIC
```bash
lsscsi|grep -i 'ethernet'
```
##### Found out power status of the server
```bash
ipmitool -U your_bmc_username -P your_bmc_userpassword -I lanplus -H your_bmc_ip_address power status
```
##### Found out server sensor temperature
```bash
ipmitool sensors |grep -i Temp
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
## Others
[[back to top ](#handy-bash-oneliner-commands-for-tsv-file-editing )]
##### Remove newline / nextline
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
tr --delete '\n' < input.txt > output.txt
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Replace newline
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
tr '\n' ' ' < filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### To uppercase/lowercase
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
tr /a-z/ /A-Z/
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Compare files (e.g. fileA, fileB)
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
diff fileA fileB
# a: added; d:delete; c:changed
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
sdiff fileA fileB
# side-to-side merge of file differences
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Number a file (e.g. fileA)
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
nl fileA
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
nl -nrz fileA
# add leading zeros
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
```bash
nl -w1 -s ' '
# making it simple, blank seperated
```
##### Combine/ paste two files (e.g. fileA, fileB)
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
paste fileA fileB
# default tab seperated
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Reverse string
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo 12345| rev
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Read .gz file without extracting
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
zmore filename
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
zless filename
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Run in background, output error file
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
some_commands & >log &
2016-07-01 10:52:19 +00:00
```
2018-01-10 01:47:26 +00:00
2018-02-08 02:00:06 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
some_commands 2>log &
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
some_commands 2>& 1| tee logfile
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
some_commands 2>& 1 >>outfile
#0: standard input; 1: standard output; 2: standard error
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Send mail
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo 'heres the content'| mail -a /path/to/attach_file.txt -s 'mail.subject' me@gmail.com
# use -a flag to set send from (-a "From: some@mail.tld")
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### .xls to csv
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
xls2csv filename
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Append to file (e.g. hihi)
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo 'hihi' >>filename
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Make BEEP sound
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
speaker-test -t sine -f 1000 -l1
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Set beep duration
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
(speaker-test -t sine -f 1000) & pid=$!;sleep 0.1s;kill -9 $pid
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### History edit/ delete
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
~/.bash_history
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
history -d [line_number]
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Get last history/record filename
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
head !$
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Clean screen
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
clear
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
2016-06-16 06:52:03 +00:00
or
2018-02-08 02:00:06 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
Ctrl+l
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Send data to last edited file
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat /directory/to/file
echo 100>!$
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Run history number (e.g. 53)
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
!53
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Run last command
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
!!
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Run last command that began with (e.g. cat filename)
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
!cat
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
2016-06-16 06:52:03 +00:00
or
2018-02-08 02:00:06 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
!c
# run cat filename again
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Extract .xf
1.unxz filename.tar.xz
2.tar -xf filename.tar
##### Install python package
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
pip install packagename
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Delete current bash command
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
Ctrl+U
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
2016-07-01 10:52:19 +00:00
or
2018-02-08 02:00:06 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
Ctrl+C
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
or
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
Alt+Shift+#
# to make it to history
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Add something to history (e.g. "addmetohistory")
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
#addmetodistory
#just add a "#" before~~
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2018-02-08 02:00:06 +00:00
##### Sleep awhile or wait for a moment or schedule a job
2016-06-16 06:52:03 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
sleep 5;echo hi
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Count the time for executing a command
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
time echo hi
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Backup with rsync
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
rsync -av filename filename.bak
rsync -av directory directory.bak
rsync -av --ignore_existing directory/ directory.bak
rsync -av --update directory directory.bak
rsync -av directory user@ip_address:/path/to/directory.bak
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
//skip files that are newer on receiver (i prefer this one!)
##### Make all directories at one time!
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat}
# -p: make parent directory
# this will create project/doc/html/; project/doc/info; project/lib/ext ,etc
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Run command only if another command returns zero exit status (well done)
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cd tmp/ & & tar xvf ~/a.tar
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Run command only if another command returns non-zero exit status (not finish)
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cd tmp/a/b/c ||mkdir -p tmp/a/b/c
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Extract to a path
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
tar xvf -C /path/to/directory filename.gz
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Use backslash "\" to break long command
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
cd tmp/a/b/c \
> || \
>mkdir -p tmp/a/b/c
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Get pwd
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
VAR=$PWD; cd ~; tar xvf -C $VAR file.tar
# PWD need to be capital letter
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### List file type of file (e.g. /tmp/)
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
file /tmp/
# tmp/: directory
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Bash script
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
#!/bin/bash
file=${1#*.}
# remove string before a "."
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
file=${1%.*}
# remove string after a "."
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Search from history
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
Ctrl+r
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Python simple HTTP Server
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
python -m SimpleHTTPServer
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Variables
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
{i/a/,}
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
e.g. replace all
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
{i//a/,}
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
//for variable i, replace all 'a' with a comma
e.g. with grep
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
test="god the father"
grep ${test// /\\\|} file.txt
# turning the space into 'or' (\|) in grep
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Read user input
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
read input
echo $input
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Generate sequence 1-10
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
seq 10
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Find average of input list/file
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
i=`wc -l filename|cut -d ' ' -f1`; cat filename| echo "scale=2;(`paste -sd+`)/"$i|bc
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
2018-02-08 02:00:06 +00:00
##### Generate all combination (e.g. 1,2)
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo {1,2}{1,2}
# 1 1, 1 2, 2 1, 2 2
2016-07-01 10:52:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Generate all combination (e.g. A,T,C,G)
2016-06-17 10:55:54 +00:00
2016-07-01 10:52:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
set = {A,T,C,G}
group= 5
for ((i=0; i< $group; i++));do
repetition=$set$repetition;done
bash -c "echo "$repetition""
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Read file content to variable
2016-09-01 09:24:27 +00:00
```bash
2018-02-08 02:00:06 +00:00
foo=$(< test1 )
2016-09-01 09:24:27 +00:00
```
2016-06-14 06:57:38 +00:00
2018-02-08 02:00:06 +00:00
##### Echo size of variable
2016-09-02 07:40:41 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo ${#foo}
2016-09-02 07:40:41 +00:00
```
2018-02-08 02:00:06 +00:00
##### Echo tab
2016-09-02 07:40:41 +00:00
2016-09-10 09:39:01 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo -e ' \t '
2016-09-10 09:39:01 +00:00
```
2016-09-02 07:40:41 +00:00
2018-02-08 02:00:06 +00:00
##### Array
2016-09-10 09:46:30 +00:00
```bash
2018-02-08 02:00:06 +00:00
declare -A array=()
2016-09-10 09:46:30 +00:00
```
2018-02-08 02:00:06 +00:00
##### Send a directory
2017-01-06 04:32:31 +00:00
```bash
2018-02-08 02:00:06 +00:00
scp -r directoryname user@ip:/path/to/send
2017-01-06 04:32:31 +00:00
```
2016-09-10 09:46:30 +00:00
2018-02-08 02:00:06 +00:00
##### Split file into lines (e.g. 1000 lines/smallfile)
2017-01-06 04:37:00 +00:00
```bash
2018-02-08 02:00:06 +00:00
$ split -d -l 1000 bigfilename
2017-01-06 04:37:00 +00:00
```
2018-02-08 02:00:06 +00:00
##### Create a large amount of dummy files (e.g 100000 files, 10 bytes each):
2017-01-06 08:16:43 +00:00
```bash
2018-02-08 02:00:06 +00:00
#1. Create a big file
dd if=/dev/zero of=bigfile bs=1 count=1000000
#2. Split the big file to 100000 10-bytes files
split -b 10 -a 10 bigfile
2017-01-06 08:16:43 +00:00
```
2017-01-23 10:17:35 +00:00
2018-02-08 02:00:06 +00:00
##### Rename all files (e.g. remove ABC from all .gz files)
```bash
rename 's/ABC//' *.gz
```
2017-01-23 10:17:35 +00:00
2018-02-08 02:00:06 +00:00
##### Remove extention (e.g remove .gz from filename.gz)
```bash
basename filename.gz .gz
2017-01-23 10:18:38 +00:00
2018-02-08 02:00:06 +00:00
zcat filename.gz> $(basename filename.gz .gz).unpacked
```
2017-01-23 10:17:35 +00:00
2018-02-08 02:00:06 +00:00
##### Use the squeeze repeat option (e.g. /t/t --> /t)
2017-01-23 10:19:41 +00:00
```bash
2018-02-08 02:00:06 +00:00
tr -s "/t" < filename
2017-01-23 10:19:41 +00:00
```
2018-02-08 02:00:06 +00:00
##### Do not print nextline with echo
2017-01-23 10:17:35 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo -e 'text here \c'
2017-01-23 10:17:35 +00:00
```
2018-02-08 02:00:06 +00:00
##### Use the last argument
2017-06-08 09:42:18 +00:00
```bash
2018-02-08 02:00:06 +00:00
!$
2017-06-08 09:42:18 +00:00
```
2017-01-06 08:16:43 +00:00
2018-02-08 02:00:06 +00:00
##### Check last exit code
2017-06-08 09:42:18 +00:00
```bash
2018-02-08 02:00:06 +00:00
echo $?
2017-06-08 09:42:18 +00:00
```
2018-02-08 02:00:06 +00:00
##### View first 50 characters of file
2017-06-16 09:29:19 +00:00
```bash
2018-02-08 02:00:06 +00:00
head -c 50 file
2017-06-16 09:29:19 +00:00
```
2018-02-08 02:00:06 +00:00
##### Group/combine rows into one row
2017-06-16 09:29:19 +00:00
2018-02-08 02:00:06 +00:00
e.g.
AAAA
BBBB
CCCC
DDDD
2017-06-26 09:29:35 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat filename|paste - -
-->
AAAABBBB
CCCCDDDD
cat filename|paste - - - -
-->
AAAABBBBCCCCDDDD
2017-06-26 09:29:35 +00:00
```
2018-02-08 02:00:06 +00:00
##### Fastq to fasta
2017-06-30 09:26:26 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat file.fastq | paste - - - - | sed 's/^@/>/g'| cut -f1-2 | tr '\t' '\n' >file.fa
2017-06-30 09:26:26 +00:00
```
2018-02-08 02:00:06 +00:00
##### Cut and get last column
2017-08-23 00:53:20 +00:00
```bash
2018-02-08 02:00:06 +00:00
cat file|rev | cut -d/ -f1 | rev
2017-08-23 00:53:20 +00:00
```
2018-02-08 02:00:06 +00:00
##### Add one to variable/increment/ i++ a numeric variable (e.g. $var)
2017-08-28 01:51:29 +00:00
```bash
2018-02-08 02:00:06 +00:00
((var++))
or
var=$((var+1))
2017-08-28 01:51:29 +00:00
2018-02-08 02:00:06 +00:00
```
2017-08-23 00:53:20 +00:00
2018-02-08 02:00:06 +00:00
##### Some handy environment variables
$0 :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$# :number of positional parameters.
$? :most recent foreground pipeline exit status.
$- :current options set for the shell.
$$ :pid of the current shell (not subshell).
$! :is the PID of the most recent background command.
2017-08-28 02:58:22 +00:00
2018-02-08 02:00:06 +00:00
##### Clear the contents of a file (e.g. filename)
2017-08-23 09:17:47 +00:00
```bash
2018-02-08 02:00:06 +00:00
>filename
2017-08-23 09:17:47 +00:00
```
2018-02-08 02:00:06 +00:00
##### Unzip tar.bz2 file (e.g. file.tar.bz2)
2017-08-23 09:17:47 +00:00
```bash
2018-02-08 02:00:06 +00:00
tar xvfj file.tar.bz2
2017-08-23 09:17:47 +00:00
```
2018-02-08 02:00:06 +00:00
##### Unzip tar.xz file (e.g. file.tar.xz)
2017-09-15 03:20:26 +00:00
```bash
2018-02-08 02:00:06 +00:00
unxz file.tar.xz
tar xopf file.tar
2017-09-15 03:20:26 +00:00
```
2018-02-08 02:00:06 +00:00
##### Output a y/n repeatedly until killed
'y':
2017-09-20 07:33:16 +00:00
```bash
2018-02-08 02:00:06 +00:00
yes
2017-09-20 07:33:16 +00:00
```
2018-02-08 02:00:06 +00:00
or 'n':
2017-10-06 07:48:47 +00:00
```bash
2018-02-08 02:00:06 +00:00
yes n
2017-10-06 07:48:47 +00:00
```
2018-02-08 02:00:06 +00:00
or 'anything':
2017-10-06 07:48:47 +00:00
```bash
2018-02-08 02:00:06 +00:00
yes anything
2017-10-06 07:48:47 +00:00
```
2018-02-08 02:00:06 +00:00
For example:
2017-10-06 07:48:47 +00:00
```bash
2018-02-08 02:00:06 +00:00
yes | rm -r large_directory
```
##### Create dummy file of certain size instantly (e.g. 200mb)
2018-02-07 10:23:31 +00:00
```bash
2018-02-08 02:00:06 +00:00
dd if=/dev/zero of=//dev/shm/200m bs=1024k count=200
or
dd if=/dev/zero of=//dev/shm/200m bs=1M count=200
```
2017-10-06 07:48:47 +00:00
2018-02-08 02:00:06 +00:00
Standard output:
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.0955679 s, 2.2 GB/s
2017-09-01 02:46:34 +00:00
2018-02-08 02:00:06 +00:00
##### Cat to a file
```bash
cat >myfile
let me add sth here
exit by control + c
^C
```
2017-09-01 02:48:00 +00:00
2018-02-08 02:00:06 +00:00
##### Keep /repeatedly executing the same command (e.g Repeat 'wc -l filename' every 1 second)
```bash
watch -n 1 wc -l filename
```
2017-09-20 07:33:16 +00:00
2018-02-08 02:00:06 +00:00
##### Print commands and their arguments when execute (e.g. echo `expr 10 + 20 `)
2017-10-19 05:38:40 +00:00
```bash
2018-02-08 02:00:06 +00:00
set -x; echo `expr 10 + 20 `
```
##### Print some meaningful sentences to you (install fortune first)
2017-10-25 03:21:15 +00:00
```bash
2018-02-08 02:00:06 +00:00
fortune
```
##### Colorful (and useful) version of top (install htop first)
2017-10-26 10:29:14 +00:00
```bash
2018-02-08 02:00:06 +00:00
htop
```
##### Press any key to continue
2017-10-26 10:29:14 +00:00
```bash
2018-02-08 02:00:06 +00:00
read -rsp $'Press any key to continue...\n' -n1 key
```
##### Run sql-like command on files from terminal
download:
https://github.com/harelba/q
example:
2017-10-31 02:15:07 +00:00
```bash
2018-02-08 02:00:06 +00:00
q -d "," "select c3,c4,c5 from /path/to/file.txt where c3='foo' and c5='boo'"
```
##### Sreen and tmux
create session and attach:
2017-10-31 02:15:07 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen
or
tmux
```
create detached session foo:
2017-11-14 07:44:15 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen -S foo -d -m
or
tmux new -s foo -d
```
detached session foo:
2017-11-14 07:44:15 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen: ^a^d
or
tmux: ^bd
```
2017-09-20 07:33:16 +00:00
2018-02-08 02:00:06 +00:00
list sessions:
2017-12-07 07:32:32 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen -ls
or
tmux ls
```
2017-12-07 07:32:32 +00:00
2018-02-08 02:00:06 +00:00
attach:
```bash
screen -r
or
tmux attach
```
2017-10-19 06:48:57 +00:00
2018-02-08 02:00:06 +00:00
attach to session foo:
2017-12-07 11:34:09 +00:00
```bash
2018-02-08 02:00:06 +00:00
screen -r foo
or
tmux attach -t foo
```
kill session foo:
```bash
screen -r foo -X quit
or
tmux kill-session -t foo
```
scroll:
Hit your screen prefix combination (C-a / control+A), then hit Escape.
Move up/down with the arrow keys (↑ and ↓).
Send command to all panes in tmux:
```bash
Ctrl-B :
setw synchronize-panes
```
##### Cut the last column
```bash
cat filename|rev|cut -f1|rev
```
##### pass password to ssh
```bash
sshpass -p mypassword ssh root@10.102.14.88 "df -h"
2017-12-07 11:34:09 +00:00
```
2018-02-08 02:00:06 +00:00
##### wait for a pid (job) to complete
2017-12-12 02:55:08 +00:00
```bash
2018-02-08 02:00:06 +00:00
wait %1
or
wait $PID
wait ${!}
#wait ${!} to wait till the last background process ($! is the PID of the last background process)
2017-12-12 02:55:08 +00:00
```
2018-02-08 02:00:06 +00:00
##### pdf to txt
2017-12-12 02:55:08 +00:00
```bash
2018-02-08 02:00:06 +00:00
sudo apt-get install poppler-utils
pdftotext example.pdf example.txt
2018-01-10 01:48:50 +00:00
```
2018-02-08 02:00:06 +00:00
##### list only directory
2017-12-12 04:05:18 +00:00
```bash
2018-02-08 02:00:06 +00:00
ls -ld -- */
2017-12-12 04:05:18 +00:00
```
2018-02-08 02:00:06 +00:00
##### Capture/record/save terminal output (capture everything you type and output)
2018-01-25 03:27:35 +00:00
```bash
2018-02-08 02:00:06 +00:00
script output.txt
#start using terminal
#to logout the screen session (stop saving the contents), type exit.
2017-12-12 04:05:18 +00:00
```
2018-02-08 02:00:06 +00:00
##### list contents of directories in a tree-like format.
2017-12-14 01:43:55 +00:00
```bash
2018-02-08 02:00:06 +00:00
tree
#go to the directory you want to list, and type tree (sudo apt-get install tree)
#output:
#one/
#└── two
# ├── 1
# ├── 2
# ├── 3
# ├── 4
# └── 5
#
2017-12-14 01:43:55 +00:00
```
2016-06-14 06:57:38 +00:00
=-=-=-=-=-A lot more coming!! =-=-=-=-=-=-=-=-=-=waitwait-=-=-=-=-=-=-=-=-=-