We're hiring!

We're actively seeking developers for our new Detroit location. Learn more

5 Unix Commands I Wish I’d Discovered Years Earlier

I’ve been using *nix systems for quite a while. But there are a few commands that I somehow overlooked and I wish I’d discovered years earlier.

1. man ascii

This prints out the ascii tables in octal, hexadeciamal and decimal. I can’t believe I didn’t know about this one until a month ago. I’d always resorted to googling for the tables. This is much more convenient.

ASCII(7)           BSD Miscellaneous Information Manual           ASCII(7)
 
NAME
    ascii -- octal, hexadecimal and decimal ASCII character sets
 
DESCRIPTION
    The octal set:
 
    000 nul  001 soh  002 stx  003 etx  004 eot  005 enq  006 ack  007 bel
    010 bs   011 ht   012 nl   013 vt   014 np   015 cr   016 so   017 si
    020 dle  021 dc1  022 dc2  023 dc3  024 dc4  025 nak  026 syn  027 etb
    030 can  031 em   032 sub  033 esc  034 fs   035 gs   036 rs   037 us

For more information, see the ascii man page.

2. cal

Pulling up a calendar on most systems is almost always a multi-step process by default. Or you can just use the cal command.

> cal
 
    August 2013
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

For more information, see the cal man page.

3. xxd

> xxd somefile.bin
 
0000000: 83ff 0010 8d01 0408 d301 0408 a540 0408  .............@..
0000010: d701 0408 d901 0408 db01 0408 0000 0000  ................
0000020: 0000 0000 0000 0000 0000 0000 1199 0508  ................
0000030: df01 0408 0000 0000 e199 0508 1d9a 0508  ................
0000040: e501 0408 2912 0508 e901 0408 eb01 0408  ....)...........
0000050: ed01 0408 ef01 0408 39e0 0408 55e0 0408  ........9...U...
0000060: 71e0 0408 8de0 0408 a9e0 0408 39f7 0408  q...........9...
0000070: 6df7 0408 a5f7 0408 ddf7 0408 15f8 0408  m...............
0000080: 49f8 0408 81f8 0408 7de5 0408 0b02 0408  I.......}.......
0000090: 4ded 0408 a9ed 0408 1102 0408 c5e0 0408  M...............
00000a0: 1502 0408 1702 0408 1902 0408 1b02 0408  ................
00000b0: e50a 0508 1d0b 0508 590b 0508 2302 0408  ........Y...#...
00000c0: 2502 0408 253d 0508 2941 0508 7106 0508  %...%=..)A..q...
00000d0: 8106 0508 690e 0508 990e 0508 c90e 0508  ....i...........
00000e0: 19e1 0408 3702 0408 3902 0408 3b02 0408  ....7...9...;...

This is another command I can’t believe I didn’t know about until recently. xxd can generate a hex dump of a given file, and also convert an edited hex dump back into its original binary form. It can also output the hex dump as a C array which is also super handy:

> xxd -i data.bin
 
unsigned char data_bin[] = {
  0x6d, 0x61, 0x64, 0x65, 0x20, 0x79, 0x6f, 0x75, 0x20, 0x6c, 0x6f, 0x6f,
  0x6b, 0x0a
};
unsigned int data_bin_len = 14;

I’ve also used it to compare binary files by generating a hex dump of two files and then diff’ing them. For more information, see the xxd man page.

4. ssh

ssh was one of the first non-trivial unix utilites that I got familiar with, but it was a while before I realized that it can be used for a lot more than just logging into remote machines.

ssh and its accompanying tools can be used for:

  • Copying files between computers (using scp)
  • X-forwarding – connect to a remote machine and have any gui applications started, displayed as if they were started locally, even if the remote machine doesn’t have an X server.
  • Port forwarding – forward a connection to a local port to a port on a remote machine OR forward connections to a port on a remote machine to a local port
  • SOCKS proxy – allows you forward any connections of an application that supports SOCKS proxies through the remote host. Useful for more secure browsing over public wifi and for getting around overly restrictive firewalls.
  • typing a password on your local machine once, then using a secure identity to login to several remote machines without having to retype your password by using an ssh key agent. This is awesome.

For more information, see the ssh man page.

5. mdfind

This one is specific to mac, as there are other *nix equivalents. It has similar functionality to find but uses the Spotlight index. It allows you to search your entire filesystem in seconds. You can also use it to give you live updates when new files that match your query appear. I use it most often when I’m trying to find the obscure location that an application stores some critical file.

> mdfind -name homebrew
 
/usr/local/Library/Homebrew
/Users/job/Library/Logs/Homebrew

Fore more information, see the mdfind man page
 

Job Vranish (26 Posts)

This entry was posted in Developer Tools and tagged . Bookmark the permalink. Both comments and trackbacks are currently closed.

38 Comments

  1. Aaron Cornelius
    Posted September 9, 2013 at 10:04 am

    Don’t forget that you can use xxd (well, or just about any command) from within Vim as well. When you have opened a binary file use this Vim command to run xxd over the contents of the opened file:
    :%!xxd
    Make your edits, then revert the file like so before saving:
    :%!xxd -r

  2. Posted September 9, 2013 at 11:03 am

    If you’re on linux, cal -3 is particularly handy when you’re looking near the end or beginning of a month.

  3. m
    Posted September 10, 2013 at 11:59 am

    Cool. Thanks for the list!

  4. DH
    Posted September 10, 2013 at 12:00 pm

    “comm” is one that i wish i had found 10 years ago

    it compares two sorted files and prints the differences (not unlike diff, but better suited for lists rather than code)

    mdfind is called “locate” in the rest of the unixy-world

    • hibbelig
      Posted September 10, 2013 at 4:50 pm

      mdfind can do way more than locate, because the latter searches by file name only, whereas the former searches by a host of other criteria.

      That said, I think locate covers the most common use case.

    • Stelios
      Posted September 11, 2013 at 12:03 am

      sdiff

  5. Victor
    Posted September 10, 2013 at 12:14 pm

    There’s also a Linux equivalent (predecessor ?) of mdfind, which is locate.

  6. Posted September 10, 2013 at 12:18 pm

    Never knew about mdfind! Thanks for the list.

  7. Posted September 10, 2013 at 12:21 pm

    I also use “bc” (calculator) to do quick calculation while I’m in terminal.

  8. Posted September 10, 2013 at 12:29 pm

    After years of using hexdump (or something similar, I cannot recall the exact name of the command, I’m very tab-completion dependent) and not remembering the right arguments for getting a “proper” dump, xdd is a godsend.

    Thanks! Reading these articles makes me feel ashamed that one can be so many years using Linux and still not know about those hidden gems.

  9. Michael R
    Posted September 10, 2013 at 12:45 pm

    mdfind sounds a lot the locate utility on Linux systems.

  10. Michael R
    Posted September 10, 2013 at 12:45 pm

    mdfind sounds a lot like the locate utility on Linux systems.

  11. Posted September 10, 2013 at 12:48 pm

    we have a phrase for that

    rtfm

    • hibbelig
      Posted September 10, 2013 at 4:53 pm

      rtfm doesn’t help. I think a switch -r should be added to man, and it should open a random man page.

  12. John M
    Posted September 10, 2013 at 12:49 pm

    Wow, I never new about xxd, I use “od -c” but the former is way more flexible — thanks!

    Note that “ssh” can be set to only make new connections to a server when it needs to. If you’re doing a lot of things, commands mixed with copies, it’s *very* fast. Search “ssh controlmaster auto”. (I use it for deploys with Fabric)

    My favorite hidden command is “xargs”. You give it a command, pipe it a list of arguments, and it’ll process it all for you. Great for directories of >10,000 images, as the shell “*” will start to fail. Hacky example: find . -name ‘*.jpg’ | xargs -i convert {} -resize 100×100 thumb_{}
    Oh did I mention it’ll do things in parallel with the “-P” flag? Yes.

    another example from my web site: http://johntellsall.blogspot.com/2013/08/converting-video-for-media-player.html

  13. calc
    Posted September 10, 2013 at 1:05 pm

    I like “dc” for quick calculations in the terminal… probably because I’m partial to reverse polish notation.

  14. Posted September 10, 2013 at 1:17 pm

    I ve been using Shell for almost a year, now and I didnt know the first 4 yet!

    Thanks for sharing :-)

  15. Nasrullah
    Posted September 10, 2013 at 2:40 pm

    ack-grep is also really useful, for searching for text in all files in a directory tree

    • Greg Swallow
      Posted September 10, 2013 at 8:20 pm

      grep -ri

      • prakash
        Posted September 12, 2013 at 3:07 pm

        ‘grep -ri’ is just part of what ack does. The best part of ack is that it ignores the files that you’d want to ignore anyway (most of the time), like temp files, backup files, version control information files etc. You can easily ask it to search for only a certain type of files and ignore everything else. If you’re a C/C++ programmer, for example, just add –cpp option and ack searches only the relevant files with the right extensions. It knows several types and you can easily define new types.

        see http://beyondgrep.com for more info.

    • Posted September 12, 2013 at 3:13 pm

      Please note that ack isn’t aimed for “text in all files”. It is specifically aimed at trees of source code, not trees of arbitrary text. There are many cases where ack will ignore files that you might not want it to.

  16. wmu
    Posted September 10, 2013 at 3:35 pm

    apropos might be useful :)

  17. Posted September 10, 2013 at 4:20 pm

    Unlike locate, mdfind has a database that updates in realtime, and can parse file contents. You can write plugins that parse out whatever metadata you’re interested in, for example http://lemonodor.com/index.php/2005/08/lisp_for_spotli/

    • Michael R
      Posted September 10, 2013 at 4:58 pm

      My ignorance was blissful. Thanks for the information.

    • Posted September 10, 2013 at 5:18 pm

      Locate has a cron task that updates its db daily. Also you can run updatedb as root.

  18. Juan
    Posted September 10, 2013 at 5:41 pm

    Have you tried ‘calendar’, may be not so useful, but very cool

  19. Posted September 10, 2013 at 8:53 pm

    Check out ‘remind’ (in the link) for a beefy command-line alternative to ‘cal’. In fact, remind is a fairly sophisticated command-line task manager.

  20. cellofellow
    Posted September 11, 2013 at 2:00 am

    Another one I discovered years later was “apropos”, which does a keyword search on all man pages. Quite handy.

  21. Tess
    Posted September 11, 2013 at 3:51 am

    Why would anyone need to call a shell command to see a calendar when they’re on an OS that’s perfectly capable of running Emacs? Just map M-x calendar to something convenient like C-M-f10, and you can bring up a quick calendar. And with diary mode, you can save appointments to the calendar, and get reminders. Why do things the hard way?

    (Kidding, of course. Well, almost entirely.)

  22. Hassan Dibani
    Posted September 11, 2013 at 5:04 am

    Ubuntu (and maybe other Distros too) has a desktop search tool called tracker that updates real time, searches within the file contents and has a command line interface called tracker-search.

    http://manpages.ubuntu.com/manpages/lucid/man1/tracker-search.1.html

  23. McUsr
    Posted September 11, 2013 at 6:39 am

    Hi.

    Interesting list, on a mac, bc isn’t like something you’ll have to use polish reverse notation on, as a matter of fact I think it is a lot like “hoc”, in that it is a full programming language, with loops and everything for doing math! Now, that is a good freebie, if you ever have to do some calculations, and prefer to do them from the commandline, or some cron job.

    The biggest ephiphany I had on a mac, is that I can just go to a folder and enter “make”, to make a project! -Awesome! It works!

    Other commands I find very useful, is man -aw utility-name, since I have some custom installations, and then I can see which manual pages that pertain to the utility.

    Maybe I also should mention od -cb, to see what is in a stream, you cat something |od -cb, and you’ll see the special characters. cat something |sed l does almost the same.

  24. Erez
    Posted September 11, 2013 at 7:12 am

    My favorite was always – aafire, :)

  25. Posted September 11, 2013 at 1:08 pm

    mdfind is much more than Unix locate. It is an interface to Mac OS X’s spotlight utility.

    Here’s an article from 2006 that should still be relevant today as an overview, even if some options are a little different.

    http://www.macdevcenter.com/pub/a/mac/2006/01/04/mdfind.html

  26. mnorbi
    Posted September 13, 2013 at 1:40 pm

    shopt -s globstar

  27. Howard
    Posted September 13, 2013 at 4:56 pm

    The one I’m always stunned that most people don’t know is ‘less’. I’m forever seeing people running ‘tail -f -somenumber’ and then trying to page up or down to find what they are looking for. less provides a super useful pager, backwards and forwards, with a follow mode that makes ‘tail -f’ completely unnecessary, IMHO.

  28. mnorbi
    Posted September 13, 2013 at 5:01 pm

    zipgrep, combined with find -print -exec

  29. Posted September 26, 2013 at 1:50 pm

    Gangster. Thanks!

  30. Erik Allik
    Posted February 16, 2014 at 10:00 am

    You *have* to add `pbcopy` and `pbpaste` to the list!