Article summary
For the last three and a half years, every single command I’ve run from the command line on my MacBook Pro has been logged to a set of log files.
Uncompressed, these files take up 16 MB of disk space on my laptop. But the return I’ve gotten on that small investment is immense. Being able to go back and find any command you’ve run in the past is so valuable, and it’s so easy to configure, you should definitely set it up today. I’m going to share how to do this so you can take advantage of it as well.
Bash Configuration File
You’ll need to configure an environment variable so that it’s loaded in every command line session. On my MacBook Pro, I use the .bash_profile file. On other operating systems, the .bashrc file is an option. See this blog post on .bash_profile vs .bashrc for more on the differences.
PROMPT_COMMAND
The Bash Prompt HOWTO describes the PROMPT_COMMAND environment variable as follows:
Bash provides an environment variable called PROMPT_COMMAND. The contents of this variable are executed as a regular Bash command just before Bash displays a prompt.
We’re going to set the PROMPT_COMMAND variable to be something that logs the most recent line of history to a file. To do this, add the following to your chosen Bash configuration file (.bash_profile for me):
export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'
First, this checks to make sure we’re not root.
If that checks out, it appends a line that includes the current timestamp, the current working directory, and the last command executed to a log file that includes the current date in the filename.
Having the commands stored in separate files like this really helps when you’re trying to find a command you ran sometime last month, for example.
> grep -h logcat ~/.logs/bash-history-2016-04*
2016-04-01.10:18:03 /Users/me 66555 adb logcat
2016-04-01.10:19:56 /Users/me 66555 adb logcat
2016-04-01.11:01:36 /Users/me 66555 adb logcat
2016-04-05.09:50:25 /Users/me/git/android-project 66368 adb logcat
2016-04-05.13:42:54 /Users/me/git/android-project 66349 adb -s emulator-5554 logcat
2016-04-06.10:40:08 /Users/me/git/android-project 66390 adb logcat
2016-04-06.10:48:54 /Users/me/git/android-project 66342 adb logcat
Conclusion
It will only take a few seconds to update your PROMPT_COMMAND so that it logs every command to a file.
And the next time you’re trying to remember the command line options you used with find that one time (but can’t find in your current session’s history), you’ll be able to look it up in the log files.
Oh, and if you want to know how many times you’ve done a git push in the last three and a half years, you can look that up, too (5,585 git pushes for me)!
Why not use your normal bash history support? Adding a export HISTSIZE=”” to your bashrc should give you infinite history AND you get to search it with crtl+r.
To get the timestamps, you can add a export HISTTIMEFORMAT=”%d/%m/%y %T ” orso.
I was wondering the same thing actually. Is there any benefit to this technique?
set
“`
HISTSIZE=-1
HISTFILESIZE=-1
“`
I don’t think normal bash history works across multiple terminals. You can easily test this by opening two terminals side by side, trying a few commands, and running the history command. On OS X using iTerm the history diverges in each terminal.
Searching reveals some possible ways to force history to be saved and reloaded after every command (http://unix.stackexchange.com/a/48113) but that doesn’t strike me as any cleaner than what I’ve described above. Maybe just a personal preference to have it stored in files at that point.
yep. you have to call `history -a; history -r` on bash_command, because bash tries to optimize things by never reading the file, while it also tries to make it resilient to power failures by always writing the file from memory to the disk.
the elegant solution is to have a central service, akin to syslog. Maybe that can be systemd saving grace?
anyway, bash’s standard solution has the advantage that those 16mb of command will be in memory for autocomplete, and it will also deal with duplicate lines in some (poor) fashion. Your way has the advantage of timestamps.
Actually in zsh it works quite well
Bash will write history from multiple terminals just fine. (I just ran your test, on OS X Terminal, to confirm.) Obviously it only writes history after a clean exit, but that’s a separate issue.
I was wondering the same (why not use HISTSIZE/HISTFILESIZE?), but the point of losing stuff is very valid. On my main machine where I have mult terminals always open I’ve lost more history than I’ve saved.
# When the shell exits, append to the history file instead of overwriting it
shopt -s histappend
# After each command, append to the history file and reread it
export PROMPT_COMMAND=”${PROMPT_COMMAND:+$PROMPT_COMMAND$’\n’}history -a; history -c; history -r”
# Adding time stamps to command history
export HISTTIMEFORMAT=”%d/%m/%y %T ”
Would do a similar thing(with support of multiple terminals) but use your bash_history.
One trick if you use nfs home dirs is to have the history file include the hostname – otherwise corruption is a possibility. I don’t currently have nfs home dirs but have in the past so stick with this.
I also put my home dir in git. So putting history in separate files by machine sorts out merging issues.
Daniil, thanks!
Any downsides to having an infinite history using bash’s history builtin?
Disk space, but it’s a pretty minor concern.
Also: you can use grep with history, like this:
history | grep “thing I want to find”
Also its easy to search search over your bash_history using ctr+r. Not sure if you can do it by saving it into another file.
Bash has native support to log everything to syslog. Why not just use that?
Because logrotate.
@Arthur, Patrick, Nicola: The startup time of Bash gets really noticeable if Bash’s built-in history gets long.
Thanks Ingo, that’s what I was worried about.
– The command must be put in ~/.bashrc, not bash_profile
– export is not necessary once the line is defined in every shells from bashrc
– The root check is not necessary on every log as a process such as bash does not changes owner during its life.
– The command must activated only for interactive shells
Modified code:
if [[ $- = *i* ]] && (( EUID != 0 )); then
[[ -d ~/.logs ]] || mkdir ~/.logs
PROMPT_COMMAND=’echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $(history 1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log’
fi
Having a big .history file is a problem because it is read every time you open a new bash prompt and it will become very slow very fast.
export HISTTIMEFORMAT=”%h/%d/%y – %H:%M:%S ”
export HISTCONTROL=ignoreboth
export HISTSIZE=5000
shopt -s histappend
PROMPT_COMMAND=”${PROMPT_COMMAND:+$PROMPT_COMMAND ; }”‘echo $$ $USER “$(history 1)” >> ~/.bash_eternal_history’
Hey Patrick, I started an app called https://bashhub.com to do this across systems and have it based in the cloud. Check it out! Would love your feedback. It’s open source as well at https://github.com/rcaloras/bashhub-client
You can find me at https://bashhub.com/u/rccola :D
Slight tweak: {date}\t{pwd}\t{cmd}
Just starting this now, hoping tab-separated-values will make it easier to parse when needed
export PROMPT_COMMAND=’if [ “$(id -u)” -ne 0 ]; then echo “$(date “+%Y-%m-%d.%H:%M:%S”)’$’\t”$(pwd)’$’\t”$(history 1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; fi’
The date is duplicated in the filename and in the file content. Maybe use only time in the file?
Thanks for the tip Patrick, I’ve made it work on my Macbook Pro. From the comments I can see there’s more than one way to do this.
@Ryan
Nice.
Where is “the cloud”? Can I set it up to work on my own DigitalOcean droplet?
Hey @Thomas the cloud is https://bashhub.com. Bashhub’s server implementation is not currently open sourced, however the client is and can be installed on all kinds of boxes. If you’re concerned about privacy feel free to give https://github.com/rcaloras/bashhub-client/wiki/Security-and-Privacy a read.
Would love to hear your feedback.
Good tip Patrick,
I’m using zsh now, Can you specify the same command for zsh
See http://superuser.com/questions/735660/whats-the-zsh-equivalent-of-bashs-prompt-command
I use this command for ZSH:
export PROMPT_COMMAND=’if [ “$(id -u)” -ne 0 ]; then echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $(fc -ln -1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; fi’
precmd() { eval “$PROMPT_COMMAND” }
Actually you do not need something like that for zsh
those two lines in your ~/.zshrc do what you whant
setopt inc_append_history
setopt share_history
http://askubuntu.com/a/23631
What about a multi-machine history sync?
Install something on all your servers and pcs and share all shell history across all your user accounts!
I am not saying I have this, I am saying I want this.
https://bashhub.com
Here the function for Zsh (to put in the ~/.zshrc file), don’t forget to mkdir ~/.logs
preexec() { echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $1″ >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; }
Thanks mister Bacon
Hi, maybe you should try hstr by dvorak:
http://www.mindforger.com/projects/hh.html
or
https://github.com/dvorka/hstr
It impoves ctrl-r and can be used with bash and zsh.
Awesome!
At first I was thinking about bash history also but this is more neat imho. I have set the output to my Dropbox folder so I can store forever.
Also, as an added bonus, if my laptop is ever stolen or misused there is a chance Dropbox could collect the evidence.
Conor
You can set your configuration so that bash will store history after execution of the command, vs after the session is terminated: http://askubuntu.com/a/67306
Neat use of PROMPT_COMMAND. I haven’t managed to get the flexibility and customization I want using normal bash_history. Some further ideas…
Whenever you open a new terminal window this will log whatever is on the history stack from before…so you get some extra cruft in there. That’s because PROMPT_COMMAND runs right before displaying a prompt…so this will run before the first prompt is shown.
A small modification is to add another if/else that checks-or-sets an environmental variable so you can avoid logging on the first prompt.
Also. Logging the history index numbers don’t seem to add much benefit since when you are running many different terminal windows. I would cut this out.
Including $TERM_SESSION_ID might be interesting in order to extract the series of commands that took place in a given terminal window.
I love this approach. There are a lot of naysayers here remarking on the fact that there are other ways to do this, however, this is super simple, and the log files can be analysed and processed in any text editor. Thanks for this.
Really inspiring post. Thanks, Patrick.
And here is how I improved your idea:
1. Distinguish logs per computers using $(hostname)
2. Manage all logs files in the git repository
3. git push on startup ( or when logging in to the computer )
Give it a try :)
If the user is switching between tmux sessions, then maybe we can log tmux session name as well in the history log .
For the multi session problem, why not just use good ole tcsh which supports concurrent shell sessions by optionally merging each sessions history with what is in the history file already. This is a lot nicer than bash’s solution which just amounts to appending to the existing history. How is that of any use to anyone? With tcsh, all history entries are time stamped as well, making the resulting history file be chronologically sorted. It’s not a great option for scripting, but for interactive use it’s tough to beat.
I did this on my mac and noticed that new terminal tabs would no longer be opened to the same working directory. To fix this, you need to not overwrite the old value of PROMPT_COMMAND. In ~/.bash_profile:
function log_bash_history { if [ “$(id -u)” -ne 0 ]; then echo “$(date “+%Y-%m-%d.%H:%M:%S”) $(pwd) $(history 1)” >> ~/.logs/bash-history-$(date “+%Y-%m-%d”).log; fi }
export PROMPT_COMMAND=”log_bash_history; $PROMPT_COMMAND”
Another benefit of this would be a crude auditing system. Log the IP ($SSH_CLIENT) too.
Why “$(id -u)” -ne 0?
Why PROMPT_COMMAND, and not (only) bash builtin history & comp.? Well, simply because it may be two different tasks: I want to “log” all my commands with timestamps, working directory, terminal pids and hostnames (NFS shared among many hosts with different software packages, i.e., not all commands can be run from all hosts), even from killed terminals, etc. — and I want to have a “standard” history of an individual terminal, deduplicated, not mixed with all other terminals, etc.
My solution is as follows:
export HISTTIMEFORMAT=’%y%m%d%H%M%S ‘
PROMPT_COMMAND=’
HISTLOG_COMMON=”`date +”%y%m%d%H%M%S”` $$ ${HOSTNAME%%.*}”
if [ -z ${HISTLOG_STARTED+x} ]
then echo “$HISTLOG_COMMON ${PWD/$HOME/~} [start]”; HISTLOG_STARTED=OK
elif [ “$HISTLOG_LAST” != “`history 1`” ]; then
echo -n “`history 1 | sed -E “s/^ +[0-9]+ +([0-9]+).*/\1/”`-$HISTLOG_COMMON ${HISTLOG_WD/$HOME/~}:”
fc -nl -1 | sed -E “s/^\t //”
fi >> ~/.history
export HISTLOG_LAST=`history 1`
HISTLOG_WD=`pwd`’
— everything goes to one file: size does not matter and grep will do the work
— without HISTLOG_STARTED, some random last command from history would be logged during a new terminal startup
— without HISTLOG_WD, cd newdir run from olddir would log newdir as wd, not olddir
— without HISTLOG_LAST, pressing enter (without any command) would re-log the last command with the current time as “finish time”
— PROMPT_COMMAND is evaluated after the previous command is finished. In other words, if someone runs some long-running command (e.g., editor :-), the time of its finish will appear in the log, which could be rather counterintuitive. If we want to add also the start time, we have to employ HISTTIMEFORMAT and history 1, which displays the timestamp of the last command
Finally, all the above logs only the finished commands. If one would like to log also the long-running commands which run in the moment of some interruption of the terminal, it’s possible to use trap as follows to log commands interrupted by some signals sent to the terminal.
for I in {0..16} {18..64}; do trap ‘HISTLOG_COMMON=”`date +”%y%m%d%H%M%S”` $$ ${HOSTNAME%%.*}$TITLE_UTF” ;\
if [ “$HISTLOG_LAST” != “`history 1`” ]; then \
echo -n “`history 1 | sed -E “s/^ +[0-9]+ +([0-9]+).*/\1/”`-$HISTLOG_COMMON ${HISTLOG_WD/$HOME/~}: [SIG’$I’] ” ;\
fc -nl -1 | sed -E “s/^\t //” ;\
else \
echo “$HISTLOG_COMMON ${PWD/$HOME/~} [SIG’$I’]” ;\
fi >> ~/.history’ $I; done
— 17 is SIGCHLD, we don’t want to log it :-)