Article summary
I’ve been using *nix systems for quite a while. But there are a few commands that I somehow overlooked and I wish I’d discovered years earlier.
1. man ascii
This prints out the ascii tables in octal, hexadeciamal and decimal. I can’t believe I didn’t know about this one until a month ago. I’d always resorted to googling for the tables. This is much more convenient.
ASCII(7) BSD Miscellaneous Information Manual ASCII(7)
NAME
ascii -- octal, hexadecimal and decimal ASCII character sets
DESCRIPTION
The octal set:
000 nul 001 soh 002 stx 003 etx 004 eot 005 enq 006 ack 007 bel
010 bs 011 ht 012 nl 013 vt 014 np 015 cr 016 so 017 si
020 dle 021 dc1 022 dc2 023 dc3 024 dc4 025 nak 026 syn 027 etb
030 can 031 em 032 sub 033 esc 034 fs 035 gs 036 rs 037 us
2. cal
Pulling up a calendar on most systems is almost always a multi-step process by default. Or you can just use the cal command.
> cal
August 2013
Su Mo Tu We Th Fr Sa
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31
3. xxd
> xxd somefile.bin
0000000: 83ff 0010 8d01 0408 d301 0408 a540 0408 .............@..
0000010: d701 0408 d901 0408 db01 0408 0000 0000 ................
0000020: 0000 0000 0000 0000 0000 0000 1199 0508 ................
0000030: df01 0408 0000 0000 e199 0508 1d9a 0508 ................
0000040: e501 0408 2912 0508 e901 0408 eb01 0408 ....)...........
0000050: ed01 0408 ef01 0408 39e0 0408 55e0 0408 ........9...U...
0000060: 71e0 0408 8de0 0408 a9e0 0408 39f7 0408 q...........9...
0000070: 6df7 0408 a5f7 0408 ddf7 0408 15f8 0408 m...............
0000080: 49f8 0408 81f8 0408 7de5 0408 0b02 0408 I.......}.......
0000090: 4ded 0408 a9ed 0408 1102 0408 c5e0 0408 M...............
00000a0: 1502 0408 1702 0408 1902 0408 1b02 0408 ................
00000b0: e50a 0508 1d0b 0508 590b 0508 2302 0408 ........Y...#...
00000c0: 2502 0408 253d 0508 2941 0508 7106 0508 %...%=..)A..q...
00000d0: 8106 0508 690e 0508 990e 0508 c90e 0508 ....i...........
00000e0: 19e1 0408 3702 0408 3902 0408 3b02 0408 ....7...9...;...
This is another command I can’t believe I didn’t know about until recently. xxd can generate a hex dump of a given file, and also convert an edited hex dump back into its original binary form. It can also output the hex dump as a C array which is also super handy:
> xxd -i data.bin
unsigned char data_bin[] = {
0x6d, 0x61, 0x64, 0x65, 0x20, 0x79, 0x6f, 0x75, 0x20, 0x6c, 0x6f, 0x6f,
0x6b, 0x0a
};
unsigned int data_bin_len = 14;
I’ve also used it to compare binary files by generating a hex dump of two files and then diff’ing them.
4. ssh
ssh was one of the first non-trivial unix utilites that I got familiar with, but it was a while before I realized that it can be used for a lot more than just logging into remote machines.
ssh and its accompanying tools can be used for:
- Copying files between computers (using scp)
- X-forwarding – connect to a remote machine and have any gui applications started, displayed as if they were started locally, even if the remote machine doesn’t have an X server.
- Port forwarding – forward a connection to a local port to a port on a remote machine OR forward connections to a port on a remote machine to a local port
- SOCKS proxy – allows you forward any connections of an application that supports SOCKS proxies through the remote host. Useful for more secure browsing over public wifi and for getting around overly restrictive firewalls.
- typing a password on your local machine once, then using a secure identity to login to several remote machines without having to retype your password by using an ssh key agent. This is awesome.
For more information, see the ssh man page.
5. mdfind
This one is specific to mac, as there are other *nix equivalents. It has similar functionality to find but uses the Spotlight index. It allows you to search your entire filesystem in seconds. You can also use it to give you live updates when new files that match your query appear. I use it most often when I’m trying to find the obscure location that an application stores some critical file.
> mdfind -name homebrew
/usr/local/Library/Homebrew
/Users/job/Library/Logs/Homebrew
Fore more information, see the mdfind man page
Don’t forget that you can use xxd (well, or just about any command) from within Vim as well. When you have opened a binary file use this Vim command to run xxd over the contents of the opened file:
:%!xxdMake your edits, then revert the file like so before saving:
:%!xxd -rIf you’re on linux, cal -3 is particularly handy when you’re looking near the end or beginning of a month.
Cool. Thanks for the list!
“comm” is one that i wish i had found 10 years ago
it compares two sorted files and prints the differences (not unlike diff, but better suited for lists rather than code)
mdfind is called “locate” in the rest of the unixy-world
mdfindcan do way more thanlocate, because the latter searches by file name only, whereas the former searches by a host of other criteria.That said, I think
locatecovers the most common use case.sdiff
There’s also a Linux equivalent (predecessor ?) of mdfind, which is locate.
Never knew about
mdfind! Thanks for the list.I also use “bc” (calculator) to do quick calculation while I’m in terminal.
After years of using hexdump (or something similar, I cannot recall the exact name of the command, I’m very tab-completion dependent) and not remembering the right arguments for getting a “proper” dump, xdd is a godsend.
Thanks! Reading these articles makes me feel ashamed that one can be so many years using Linux and still not know about those hidden gems.
mdfind sounds a lot the locate utility on Linux systems.
mdfind sounds a lot like the locate utility on Linux systems.
we have a phrase for that
rtfm
rtfm doesn’t help. I think a switch
-rshould be added toman, and it should open a random man page.Wow, I never new about xxd, I use “od -c” but the former is way more flexible — thanks!
Note that “ssh” can be set to only make new connections to a server when it needs to. If you’re doing a lot of things, commands mixed with copies, it’s *very* fast. Search “ssh controlmaster auto”. (I use it for deploys with Fabric)
My favorite hidden command is “xargs”. You give it a command, pipe it a list of arguments, and it’ll process it all for you. Great for directories of >10,000 images, as the shell “*” will start to fail. Hacky example: find . -name ‘*.jpg’ | xargs -i convert {} -resize 100×100 thumb_{}
Oh did I mention it’ll do things in parallel with the “-P” flag? Yes.
another example from my web site: http://johntellsall.blogspot.com/2013/08/converting-video-for-media-player.html
I like “dc” for quick calculations in the terminal… probably because I’m partial to reverse polish notation.
I ve been using Shell for almost a year, now and I didnt know the first 4 yet!
Thanks for sharing :-)
ack-grep is also really useful, for searching for text in all files in a directory tree
grep -ri
‘grep -ri’ is just part of what ack does. The best part of ack is that it ignores the files that you’d want to ignore anyway (most of the time), like temp files, backup files, version control information files etc. You can easily ask it to search for only a certain type of files and ignore everything else. If you’re a C/C++ programmer, for example, just add –cpp option and ack searches only the relevant files with the right extensions. It knows several types and you can easily define new types.
see http://beyondgrep.com for more info.
Please note that ack isn’t aimed for “text in all files”. It is specifically aimed at trees of source code, not trees of arbitrary text. There are many cases where ack will ignore files that you might not want it to.
apropos might be useful :)
Unlike locate, mdfind has a database that updates in realtime, and can parse file contents. You can write plugins that parse out whatever metadata you’re interested in, for example http://lemonodor.com/index.php/2005/08/lisp_for_spotli/
My ignorance was blissful. Thanks for the information.
Locate has a cron task that updates its db daily. Also you can run updatedb as root.
Have you tried ‘calendar’, may be not so useful, but very cool
Check out ‘remind’ (in the link) for a beefy command-line alternative to ‘cal’. In fact, remind is a fairly sophisticated command-line task manager.
Another one I discovered years later was “apropos”, which does a keyword search on all man pages. Quite handy.
Why would anyone need to call a shell command to see a calendar when they’re on an OS that’s perfectly capable of running Emacs? Just map M-x calendar to something convenient like C-M-f10, and you can bring up a quick calendar. And with diary mode, you can save appointments to the calendar, and get reminders. Why do things the hard way?
(Kidding, of course. Well, almost entirely.)
Ubuntu (and maybe other Distros too) has a desktop search tool called tracker that updates real time, searches within the file contents and has a command line interface called tracker-search.
http://manpages.ubuntu.com/manpages/lucid/man1/tracker-search.1.html
Hi.
Interesting list, on a mac, bc isn’t like something you’ll have to use polish reverse notation on, as a matter of fact I think it is a lot like “hoc”, in that it is a full programming language, with loops and everything for doing math! Now, that is a good freebie, if you ever have to do some calculations, and prefer to do them from the commandline, or some cron job.
The biggest ephiphany I had on a mac, is that I can just go to a folder and enter “make”, to make a project! -Awesome! It works!
Other commands I find very useful, is man -aw utility-name, since I have some custom installations, and then I can see which manual pages that pertain to the utility.
Maybe I also should mention od -cb, to see what is in a stream, you cat something |od -cb, and you’ll see the special characters. cat something |sed l does almost the same.
My favorite was always – aafire, :)
mdfind is much more than Unix locate. It is an interface to Mac OS X’s spotlight utility.
Here’s an article from 2006 that should still be relevant today as an overview, even if some options are a little different.
http://www.macdevcenter.com/pub/a/mac/2006/01/04/mdfind.html
shopt -s globstar
The one I’m always stunned that most people don’t know is ‘less’. I’m forever seeing people running ‘tail -f -somenumber’ and then trying to page up or down to find what they are looking for. less provides a super useful pager, backwards and forwards, with a follow mode that makes ‘tail -f’ completely unnecessary, IMHO.
zipgrep, combined with find -print -exec
Gangster. Thanks!
You *have* to add `pbcopy` and `pbpaste` to the list!
Thank you for the article! Really insightful, I didn’t know about these commands, as well.