How (and Why) to Log Your Entire Bash History

For the last three and a half years, every single command I’ve run from the command line on my MacBook Pro has been logged to a set of log files.

Uncompressed, these files take up 16 MB of disk space on my laptop. But the return I’ve gotten on that small investment is immense. Being able to go back and find any command you’ve run in the past is so valuable, and it’s so easy to configure, you should definitely set it up today. I’m going to share how to do this so you can take advantage of it as well.

Bash Configuration File

You’ll need to configure an environment variable so that it’s loaded in every command line session. On my MacBook Pro, I use the .bash_profile file. On other operating systems, the .bashrc file is an option. See this blog post on .bash_profile vs .bashrc for more on the differences.

PROMPT_COMMAND

The Bash Prompt HOWTO describes the PROMPT_COMMAND environment variable as follows:

Bash provides an environment variable called PROMPT_COMMAND. The contents of this variable are executed as a regular Bash command just before Bash displays a prompt.

We’re going to set the PROMPT_COMMAND variable to be something that logs the most recent line of history to a file. To do this, add the following to your chosen Bash configuration file (.bash_profile for me):


export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'

First, this checks to make sure we’re not root.

If that checks out, it appends a line that includes the current timestamp, the current working directory, and the last command executed to a log file that includes the current date in the filename.

Having the commands stored in separate files like this really helps when you’re trying to find a command you ran sometime last month, for example.


> grep -h logcat ~/.logs/bash-history-2016-04*
2016-04-01.10:18:03 /Users/me 66555  adb logcat
2016-04-01.10:19:56 /Users/me 66555  adb logcat
2016-04-01.11:01:36 /Users/me 66555  adb logcat
2016-04-05.09:50:25 /Users/me/git/android-project 66368  adb logcat
2016-04-05.13:42:54 /Users/me/git/android-project 66349  adb -s emulator-5554 logcat
2016-04-06.10:40:08 /Users/me/git/android-project 66390  adb logcat
2016-04-06.10:48:54 /Users/me/git/android-project 66342  adb logcat

Conclusion

It will only take a few seconds to update your PROMPT_COMMAND so that it logs every command to a file.

And the next time you’re trying to remember the command line options you used with find that one time (but can’t find in your current session’s history), you’ll be able to look it up in the log files.

Oh, and if you want to know how many times you’ve done a git push in the last three and a half years, you can look that up, too (5,585 git pushes for me)!

Conversation
  • Arthur says:

    Why not use your normal bash history support? Adding a export HISTSIZE=”” to your bashrc should give you infinite history AND you get to search it with crtl+r.

    To get the timestamps, you can add a export HISTTIMEFORMAT=”%d/%m/%y %T ” orso.

    • Michael Aquilina says:

      I was wondering the same thing actually. Is there any benefit to this technique?

      • Michael Aquilina says:

        set
        “`
        HISTSIZE=-1
        HISTFILESIZE=-1
        “`

  • Patrick Bacon Patrick Bacon says:

    I don’t think normal bash history works across multiple terminals. You can easily test this by opening two terminals side by side, trying a few commands, and running the history command. On OS X using iTerm the history diverges in each terminal.

    Searching reveals some possible ways to force history to be saved and reloaded after every command (http://unix.stackexchange.com/a/48113) but that doesn’t strike me as any cleaner than what I’ve described above. Maybe just a personal preference to have it stored in files at that point.

    • gcb says:

      yep. you have to call `history -a; history -r` on bash_command, because bash tries to optimize things by never reading the file, while it also tries to make it resilient to power failures by always writing the file from memory to the disk.

      the elegant solution is to have a central service, akin to syslog. Maybe that can be systemd saving grace?

      anyway, bash’s standard solution has the advantage that those 16mb of command will be in memory for autocomplete, and it will also deal with duplicate lines in some (poor) fashion. Your way has the advantage of timestamps.

    • Igor Khomyakov says:

      Actually in zsh it works quite well

  • Ryan says:

    Bash will write history from multiple terminals just fine. (I just ran your test, on OS X Terminal, to confirm.) Obviously it only writes history after a clean exit, but that’s a separate issue.

    • Daniel says:

      I was wondering the same (why not use HISTSIZE/HISTFILESIZE?), but the point of losing stuff is very valid. On my main machine where I have mult terminals always open I’ve lost more history than I’ve saved.

  • Daniil says:

    # When the shell exits, append to the history file instead of overwriting it
    shopt -s histappend

    # After each command, append to the history file and reread it
    export PROMPT_COMMAND=”${PROMPT_COMMAND:+$PROMPT_COMMAND$’\n’}history -a; history -c; history -r”

    # Adding time stamps to command history
    export HISTTIMEFORMAT=”%d/%m/%y %T ”

    Would do a similar thing(with support of multiple terminals) but use your bash_history.

  • Kevin Lyda says:

    One trick if you use nfs home dirs is to have the history file include the hostname – otherwise corruption is a possibility. I don’t currently have nfs home dirs but have in the past so stick with this.

    I also put my home dir in git. So putting history in separate files by machine sorts out merging issues.

  • Patrick Bacon Patrick Bacon says:

    Daniil, thanks!

    Any downsides to having an infinite history using bash’s history builtin?

    • Dan Esparza says:

      Disk space, but it’s a pretty minor concern.

      Also: you can use grep with history, like this:

      history | grep “thing I want to find”

    • Daniil says:

      Also its easy to search search over your bash_history using ctr+r. Not sure if you can do it by saving it into another file.

  • Bash has native support to log everything to syslog. Why not just use that?

  • Ingo Blechschmidt says:

    @Arthur, Patrick, Nicola: The startup time of Bash gets really noticeable if Bash’s built-in history gets long.

  • Patrick Bacon Patrick Bacon says:

    Thanks Ingo, that’s what I was worried about.

  • – The command must be put in ~/.bashrc, not bash_profile
    – export is not necessary once the line is defined in every shells from bashrc
    – The root check is not necessary on every log as a process such as bash does not changes owner during its life.
    – The command must activated only for interactive shells

    Modified code:

    if [[ $- = *i* ]] && (( EUID != 0 )); then
    [[ -d ~/.logs ]] || mkdir ~/.logs
    PROMPT_COMMAND=’echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $(history 1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log’
    fi

  • luca says:

    Having a big .history file is a problem because it is read every time you open a new bash prompt and it will become very slow very fast.

    export HISTTIMEFORMAT=”%h/%d/%y – %H:%M:%S ”
    export HISTCONTROL=ignoreboth
    export HISTSIZE=5000
    shopt -s histappend
    PROMPT_COMMAND=”${PROMPT_COMMAND:+$PROMPT_COMMAND ; }”‘echo $$ $USER “$(history 1)” >> ~/.bash_eternal_history’

  • Ryan says:

    Hey Patrick, I started an app called https://bashhub.com to do this across systems and have it based in the cloud. Check it out! Would love your feedback. It’s open source as well at https://github.com/rcaloras/bashhub-client

    You can find me at https://bashhub.com/u/rccola :D

  • Brian Olson says:

    Slight tweak: {date}\t{pwd}\t{cmd}
    Just starting this now, hoping tab-separated-values will make it easier to parse when needed

    export PROMPT_COMMAND=’if [ “$(id -u)” -ne 0 ]; then echo “$(date “+%Y-%m-%d.%H:%M:%S”)’$’\t”$(pwd)’$’\t”$(history 1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; fi’

  • yegle says:

    The date is duplicated in the filename and in the file content. Maybe use only time in the file?

  • Louis Ritchie says:

    Thanks for the tip Patrick, I’ve made it work on my Macbook Pro. From the comments I can see there’s more than one way to do this.

  • Thomas says:

    @Ryan
    Nice.
    Where is “the cloud”? Can I set it up to work on my own DigitalOcean droplet?

  • ashu says:

    Good tip Patrick,

    I’m using zsh now, Can you specify the same command for zsh

  • What about a multi-machine history sync?

    Install something on all your servers and pcs and share all shell history across all your user accounts!

    I am not saying I have this, I am saying I want this.

  • Here the function for Zsh (to put in the ~/.zshrc file), don’t forget to mkdir ~/.logs

    preexec() { echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $1″ >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; }

    Thanks mister Bacon

  • Rland says:

    Hi, maybe you should try hstr by dvorak:

    http://www.mindforger.com/projects/hh.html
    or
    https://github.com/dvorka/hstr

    It impoves ctrl-r and can be used with bash and zsh.

  • Conor Hackett says:

    Awesome!

    At first I was thinking about bash history also but this is more neat imho. I have set the output to my Dropbox folder so I can store forever.

    Also, as an added bonus, if my laptop is ever stolen or misused there is a chance Dropbox could collect the evidence.

    Conor

  • Daryl Tucker says:

    You can set your configuration so that bash will store history after execution of the command, vs after the session is terminated: http://askubuntu.com/a/67306

  • MattPr says:

    Neat use of PROMPT_COMMAND. I haven’t managed to get the flexibility and customization I want using normal bash_history. Some further ideas…

    Whenever you open a new terminal window this will log whatever is on the history stack from before…so you get some extra cruft in there. That’s because PROMPT_COMMAND runs right before displaying a prompt…so this will run before the first prompt is shown.

    A small modification is to add another if/else that checks-or-sets an environmental variable so you can avoid logging on the first prompt.

    Also. Logging the history index numbers don’t seem to add much benefit since when you are running many different terminal windows. I would cut this out.

    Including $TERM_SESSION_ID might be interesting in order to extract the series of commands that took place in a given terminal window.

  • Lloyd Moore says:

    I love this approach. There are a lot of naysayers here remarking on the fact that there are other ways to do this, however, this is super simple, and the log files can be analysed and processed in any text editor. Thanks for this.

  • Kenju says:

    Really inspiring post. Thanks, Patrick.

    And here is how I improved your idea:

    1. Distinguish logs per computers using $(hostname)
    2. Manage all logs files in the git repository
    3. git push on startup ( or when logging in to the computer )

    Give it a try :)

  • Ram says:

    If the user is switching between tmux sessions, then maybe we can log tmux session name as well in the history log .

  • rorx says:

    For the multi session problem, why not just use good ole tcsh which supports concurrent shell sessions by optionally merging each sessions history with what is in the history file already. This is a lot nicer than bash’s solution which just amounts to appending to the existing history. How is that of any use to anyone? With tcsh, all history entries are time stamped as well, making the resulting history file be chronologically sorted. It’s not a great option for scripting, but for interactive use it’s tough to beat.

  • Chris Lample says:

    I did this on my mac and noticed that new terminal tabs would no longer be opened to the same working directory. To fix this, you need to not overwrite the old value of PROMPT_COMMAND. In ~/.bash_profile:

    function log_bash_history { if [ “$(id -u)” -ne 0 ]; then echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $(history 1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; fi }

    export PROMPT_COMMAND=”log_bash_history; $PROMPT_COMMAND”

  • Another benefit of this would be a crude auditing system. Log the IP ($SSH_CLIENT) too.

  • Todd says:

    Why “$(id -u)” -ne 0?

  • Pavel Šmerk says:

    Why PROMPT_COMMAND, and not (only) bash builtin history & comp.? Well, simply because it may be two different tasks: I want to “log” all my commands with timestamps, working directory, terminal pids and hostnames (NFS shared among many hosts with different software packages, i.e., not all commands can be run from all hosts), even from killed terminals, etc. — and I want to have a “standard” history of an individual terminal, deduplicated, not mixed with all other terminals, etc.

    My solution is as follows:

    export HISTTIMEFORMAT=’%y%m%d%H%M%S ‘
    PROMPT_COMMAND=’
    HISTLOG_COMMON=”`date +”%y%m%d%H%M%S”` $$ ${HOSTNAME%%.*}”
    if [ -z ${HISTLOG_STARTED+x} ]
    then echo “$HISTLOG_COMMON ${PWD/$HOME/~} [start]”; HISTLOG_STARTED=OK
    elif [ “$HISTLOG_LAST” != “`history 1`” ]; then
    echo -n “`history 1 | sed -E “s/^ +[0-9]+ +([0-9]+).*/\1/”`-$HISTLOG_COMMON ${HISTLOG_WD/$HOME/~}:”
    fc -nl -1 | sed -E “s/^\t //”
    fi >> ~/.history
    export HISTLOG_LAST=`history 1`
    HISTLOG_WD=`pwd`’

    — everything goes to one file: size does not matter and grep will do the work
    — without HISTLOG_STARTED, some random last command from history would be logged during a new terminal startup
    — without HISTLOG_WD, cd newdir run from olddir would log newdir as wd, not olddir
    — without HISTLOG_LAST, pressing enter (without any command) would re-log the last command with the current time as “finish time”
    — PROMPT_COMMAND is evaluated after the previous command is finished. In other words, if someone runs some long-running command (e.g., editor :-), the time of its finish will appear in the log, which could be rather counterintuitive. If we want to add also the start time, we have to employ HISTTIMEFORMAT and history 1, which displays the timestamp of the last command

    Finally, all the above logs only the finished commands. If one would like to log also the long-running commands which run in the moment of some interruption of the terminal, it’s possible to use trap as follows to log commands interrupted by some signals sent to the terminal.

    for I in {0..16} {18..64}; do trap ‘HISTLOG_COMMON=”`date +”%y%m%d%H%M%S”` $$ ${HOSTNAME%%.*}$TITLE_UTF” ;\
    if [ “$HISTLOG_LAST” != “`history 1`” ]; then \
    echo -n “`history 1 | sed -E “s/^ +[0-9]+ +([0-9]+).*/\1/”`-$HISTLOG_COMMON ${HISTLOG_WD/$HOME/~}: [SIG’$I’] ” ;\
    fc -nl -1 | sed -E “s/^\t //” ;\
    else \
    echo “$HISTLOG_COMMON ${PWD/$HOME/~} [SIG’$I’]” ;\
    fi >> ~/.history’ $I; done

    — 17 is SIGCHLD, we don’t want to log it :-)

  • Comments are closed.