linux Featured

So you think you know linux

This is my notes going over all the coreutils package and picking out the ones i Didn't know, I didn't include every command because some I already knew. Enjoy....

Oran C

Mar 8, 2021 • 7 min read

a deep dive into the weird and wonderful Linux coreutils package

Snow penguin in Hokkaido, Japan — Photo by Kanjanapa Srisuwan / Unsplash

This is my notes going over all the coreutils package and picking out the ones Ididnt know, I didnt include every command becuse some I already knew. Enjoy....

Source comes from this greate blog post by Dave Gauerratfactor.com

Deep dive into linux core commands

https://www.ratfactor.com/slackware/pkgblog/coreutils

DD is actually a cargocult

The Cult of DD

use cp instead of dd

dd if=/slackware14.2.iso of=/dev/sdb

all hail the cp way

cp downloads/os.3.9.iso /dev/sdb

how to show progress bar with pip commands

sudo apt install pv

demo using creation of 100mb file
cd ~
head -c 100MB /dev/zero | pv>image.iso
OUTPUT
wisemonkey@crapple:/mnt/c/Users/oranm/Downloads$ head -c 100MB /dev/zero | pv>image.iso
95.4MiB 0:00:06 [14.9MiB/s] [            <=>

`DU` Disk usage

du -sh
448M    .


$ du -sh *
11M    bin
110M   docs
7.1M   dotfiles
53M    img
96M    proj
173M   wiki

cksum This does a cyclic redundancy check (CRC) of a given file (or STDIN).

$ echo "hello" | cksum
3015617425 6

`comm` Compares sorted lines of files and displays the results in three columns. The output is intuitively understood using an example:

> comm test.csv progs.csv

        Name                             Id
        -----------------------------------------------------------------
        Robo3T                           3T.Robo3T
        Robo3T - Beta                    3T.Robo3TBeta
        4K Slideshow Maker               4KDownload.4KSlideshowMaker

Echo cool color option

echo -e "\e[31mRed Text\e[0m"
Red Text

`env` gives you control over the environment variables, working directory, signal handling (such as blocking signals from being delivered), and CLI arguments.

/usr/bin/env --chdir=/home/wisemonkey bash

runs bash in a new directory
> pwd
/mnt/c/Users/oranm/Downloads
> /usr/bin/env --chdir=/home/wisemonkey bash
> pwd
/home/wisemonkey

`factor` Returns the prime factors of an integer:

This comes to us from 1974. Is it the weirdest thing that ships with coreutils? We’ll see…

factor 96
96: 2 2 2 2 2 3

`false` Always returns a failure (false) exit status.


> false
> echo $?
1
> ls
> echo $?
0
>

`head` lets you specify the number of lines to display from the "head" of a text file:

$ head -n 2 meow
 The Cats Will Know
By Cesare Pavese

Preview beginings of files in the current directory

 head -n 2 * | less

`logname` Nobody knows who wrote this and it doesn’t work?

its a standard command and ist broaken!

> logname
logname: no login name

md5sum

>md5sum meow
2731deb26ce04aa850042dcc40cccdb3  meow

`mkfifo` creates a queue mechanism we’re all very familiar with, a pipe. Specifically a named pipe.

>mkdir -p /tmp/
>cd /tmp/foo
/tmp/foo> mkfifo foo
/tmp/foo> ls -l
prw-r--r--  1 dave users    0 Aug 30 14:43 foo

`mknod`

> sudo mknod rando c 1 11
[sudo] password for wisemonkey:
> ls
foo  rando  test
> ls -l rando
crw-r--r-- 1 root root 1, 11 Mar  8 11:17 rando
> echo "test" >rando
-bash: rando: Permission denied
> sudo echo "test" >rando
-bash: rando: Permission denied
> ls -l
crw-r--r-- 1 root       root       1, 11 Mar  8 11:17 rando
> sudo chmod 777 rando
> ls -l
total 0
prw-r--r-- 1 wisemonkey wisemonkey     0 Mar  8 11:13 foo
crwxrwxrwx 1 root       root       1, 11 Mar  8 11:17 rando
prw-r--r-- 1 wisemonkey wisemonkey     0 Mar  8 11:12 test
> sudo echo "test" >rando
> head --bytes=16 rando  | hexdump

`mktemp` is an incredibly handy tool for a lot of file-handling tasks, especially in scripts

I think the most important option to remember is -d to make a new directory rather than a file (which is usually what I need):

$ mktemp -d
/tmp/tmp.ecmk4s

$ ls -ld /tmp/tmp*
drwx------ 2 dave users 4096 Sep  4 00:49 /tmp/tmp.ecmk4s/
-rw------- 1 dave users    0 Sep  4 00:48 /tmp/tmp.tVn7z3

`nl` stands for "number lines" and here’s what it does:

$ cat foo
Wolves enjoy waterballoons.
Don't eat the wrong fruit.
My skull is made of cheese.
Welcome to the pain castle.

$ nl foo
     1  Wolves enjoy waterballoons.
     2  Don't eat the wrong fruit.
     3  My skull is made of cheese.
     4  Welcome to the pain castle.

`nohup` if you want to launch a process from the current terminal and not have it die when you exit that terminal, use nohup.

 ping www.orancollins.com nohup &

`pr` It’s for converting text files "for printing". output for page printing could be good for ascii printing

pr -m test.sh

`realpath` gets the full path of a file/dir `get the canonical path`

> ls test.sh
test.sh
> realpath test.sh
/mnt/c/Users/oranm/Downloads/test.sh
>

`rm` -d option were on by default (remove empty directories)

rm -frd ./test
removes all files recursivly, and removes empty files

`rmdir` Remove a directory. The nice feature I didn’t know about was -p, which removes empty ancestor directories (exactly the opposite of mkdir -p)!


$ tree foo
foo
`-- bar
    `-- biz
        `-- baz

3 directories, 0 files
$ rmdir foo/bar/biz/baz

`seq` t simply prints a sequence of numbers

 >seq 4
1
2
3
4
# ADVANCED
>seq -s, 10 -2 0
10,8,6,4,2,0

`sha1sum` compute SHA-1 hashes for files

ha1sum foo.txt
fa7dd7e51436401f0555f0cb6030393a0f18cfd5  foo.txt
$ sha1sum foo.txt > foo.sha1
$ sha1sum -c foo.sha1
foo.txt: OK

# also see 
sha1sum foo.txt
fa7dd7e51436401f0555f0cb6030393a0f18cfd5  foo.txt

$ sha224sum foo.txt
0dbfebfe2057dd9b63ebbbeb8d21925323bc4ea293e4b23e1eb4a66b  foo.txt

$ sha256sum foo.txt
959a0da619f2594a481ee6e78c7c11f3686abdbbbab91f5b7d429ef8a0b46383  foo.txt

$ sha384sum foo.txt
ce87107ae3baa9f2217266770d37ddc8350609f856fd4441b6a80dd7a1fb0c362bdc427f5505a56e70aed08315
4fce2f  foo.txt

$ sha512sum foo.txt
0e83f638730bec5d0531382a4e40ea4fe9b1da05e444833282af16af03020697faf0baaa8db23b05a650b210477b7e50618a903584d140529cb2203198906b92  foo.txt

`shred` This is a super-cool spy command. It overwrites a file multiple times with random data, which makes it very hard to recover the file


> touch foo.txt
> nano foo.txt
> shred foo.txt
> cat foo.txt

����6J)���@7��Ӌ�'R�h�*�n-G�I/��@�[|��}>Y�Ǝ��1{]5�\�U

$ cat > foo.txt
Super cool spy stuff.
I am a secret agent from Mars.
$ file foo.txt
foo.txt: ASCII text
$ shred foo.txt
$ file foo.txt
foo.txt: data
$ hexdump foo.txt
0000000 5b90 3445 6e50 da24 69f4 5f77 4ee9 3f9e
0000010 6d1b ddfe 47d8 ba69 bd10 72cc a59f ee52
0000020 2184 3f03 3d29 8de9 fb32 3bc2 f758 242e

`shuf` is short for shuffle (say that three times fast) and it randomly shuffles elements from a file:

cat > words.txt
Apple
Bat
Cat
Donkey
Elephant
Fruit
Goat
Horse

$ shuf words.txt
Elephant
Donkey
Bat
Goat
Fruit
Cat
Apple
Horse

It’s a really nice tool with excellent, useful options. For example, -e shuffles the input arguments:


$ shuf -e A B C D E F G
E
F
B
G
A
C
D

Or you can give it a range of numbers:



$ shuf -i 1-10
10
5
2
1
4
8
3
9
7

fun example rool a 6 sided dice

$ alias rolld6='shuf -n 1 -i 1-6'
$ rolld6
6
$ rolld6
4

`tee` is like a "T" junction in a water pipe. It lets you send output to multiple places at once! Ever want to redirect output to a file but also see it?

> uname | tee outpoot
Linux
> cat outpoot
Linux

`tr` Translate (or remove) characters from input stream and write out the result.

note only works for single character translations

$ echo "foo" | tr 'o' 'z'
fzz
$ echo "foo" | sed 's/o/z/g'
fzz

`uniq` Given a sorted list of items, returns only uniq items:

TL:DR use `sort animals | uniq`

cat > animals
cow
cow
chicken
chicken
chicken
pig
pig
pig
pig
chicken
cow

$ uniq animals
cow
chicken
pig
chicken
cow

Also supports counting number of times shows up

uniq -c sorted_animals
      4 chicken
      3 cow
      4 pig

Or printing only the duplicated entries (with -d).

uniq -d sorted_animals

Or showing all lines, but grouping them:

 uniq --group sorted_animals
chicken
chicken
chicken
chicken

cow
cow
cow

pig
pig
pig
pig

most useful when paired with -w, which lets you specify how many characters to check:

Returns groupings of uneqiue words that match the first letter

>uniq -w 1 --group sorted_animals
chicken
chicken
chicken
chicken
cow
cow
cow

pig
pig
pig
pig

`timeout` Oooh, this is cool! I had no idea this existed. It runs a command and then kills it after a specified timeout:

>timeout 5 ping phobos
PING phobos.localdomain (10.0.0.37) 56(84) bytes of data.
64 bytes from phobos.localdomain (10.0.0.37): icmp_seq=1 ttl=64 time=0.161 ms
64 bytes from phobos.localdomain (10.0.0.37): icmp_seq=2 ttl=64 time=0.914 ms
64 bytes from phobos.localdomain (10.0.0.37): icmp_seq=3 ttl=64 time=0.900 ms
64 bytes from phobos.localdomain (10.0.0.37): icmp_seq=4 ttl=64 time=0.919 ms
64 bytes from phobos.localdomain (10.0.0.37): icmp_seq=5 ttl=64 time=0.904 ms

I got lazy and put all the somewhat intrestging commands here

`sleep`: `sleep 5`,

`sort`: `sort foo` , numeric sort :`sort -n foo.txt`

`split`: `split --lines=1 foo.txt` , Here I separate the Bilbo Baggins quote on the word "I":`split -t 'I' -l 1 baggins.txt`

`stat`: `stat foo.txt`, point stat at a file and you can see detail about it

`stdbuf`: `tail -f access.log | stdbuf -oL cut -d aq aq -f1 | uniq`, Unusefull but intresetging This lets you set the modes of the STDIN, STDOUT, and STDERR of a process.

`stty`: `stty`, This utility can either print the current settings or change them.. one of the magic commands within linux

`sum`: `sum foo.txt`, The first number is a checksum (using one of two algorithms available) and the second is the number of disk blocks it’s using. strictly historic.

`sync`: `sync`, a good write-up of this command here

`tac`: `UUUUUUUU`, Reverse lines! (It’s cat backwards, get it?) `tac --separator=,` : But it can also reverse using any separator:

`truncate`: `truncate --size=16 foo2.txt`, Hacks the ends off of files, making them the size you specify. It also grows files to make them longer (where the rest of the file becomes "sparse": `truncate --size=128 foo2.txt`

`tsort`: does sorting for graphs givin a set of tuple rules a before e kind of stuff, intreseted but not usefull

`tty`: `tty` ouput `/dev/pts/1`, prints the device filename of your current terminal: Nice and simple: good for scripting to see if your on a commandline vs typing

`unexpand`: `unexpand -t 2 tab_poem`, Converts spaces to tabs

`users`: `users`: output `wisehackermonkey`, | Who’s logged into this system?

`vdir`: `vdir -h` | another historical way to display files. See dir above. This one does a "long" output, much like ls -l: similare to ls -hal

`wc`: `wc foo.txt`,`wc -w foo.txt`, `wc -L foo.txt` | Word count. One of my favorites! I use this all the time.

`who`: `who -a -H` | Who is currently logged in?

`yes` : `$ yes | annoying_script.sh` | Typically used to "answer" programs which expect you to answer "y" to confirm things interactively.

Fin

by oran collins
github.com/wisehackermonkey
oranbusiness gmail com
20210308

a deep dive into the weird and wonderful Linux coreutils package

This is my notes going over all the coreutils package and picking out the ones Ididnt know, I didnt include every command becuse some I already knew. Enjoy....

Source comes from this greate blog post by Dave Gauerratfactor.com

Deep dive into linux core commands

DD is actually a cargocult

use cp instead of dd

all hail the cp way

how to show progress bar with pip commands

DU Disk usage

cksum This does a cyclic redundancy check (CRC) of a given file (or STDIN).

comm Compares sorted lines of files and displays the results in three columns. The output is intuitively understood using an example:

Echo cool color option

env gives you control over the environment variables, working directory, signal handling (such as blocking signals from being delivered), and CLI arguments.

factor Returns the prime factors of an integer:

This comes to us from 1974. Is it the weirdest thing that ships with coreutils? We’ll see…​

false Always returns a failure (false) exit status.

head lets you specify the number of lines to display from the "head" of a text file:

Preview beginings of files in the current directory

logname Nobody knows who wrote this and it doesn’t work?

its a standard command and ist broaken!

md5sum

mkfifo creates a queue mechanism we’re all very familiar with, a pipe. Specifically a named pipe.

mknod

mktemp is an incredibly handy tool for a lot of file-handling tasks, especially in scripts

I think the most important option to remember is -d to make a new directory rather than a file (which is usually what I need):

nl stands for "number lines" and here’s what it does:

nohup if you want to launch a process from the current terminal and not have it die when you exit that terminal, use nohup.

pr It’s for converting text files "for printing". output for page printing could be good for ascii printing

realpath gets the full path of a file/dir get the canonical path

rm -d option were on by default (remove empty directories)

rmdir Remove a directory. The nice feature I didn’t know about was -p, which removes empty ancestor directories (exactly the opposite of mkdir -p)!

seq t simply prints a sequence of numbers

sha1sum compute SHA-1 hashes for files

shred This is a super-cool spy command. It overwrites a file multiple times with random data, which makes it very hard to recover the file

shuf is short for shuffle (say that three times fast) and it randomly shuffles elements from a file:

It’s a really nice tool with excellent, useful options. For example, -e shuffles the input arguments:

Or you can give it a range of numbers:

fun example rool a 6 sided dice

tee is like a "T" junction in a water pipe. It lets you send output to multiple places at once! Ever want to redirect output to a file but also see it?

tr Translate (or remove) characters from input stream and write out the result.

note only works for single character translations

uniq Given a sorted list of items, returns only uniq items:

TL:DR use sort animals | uniq