First create a new bash process by running this command on your command line.
ssh-agent bash
It will ask you to enter your password once, then won't ask again.
Now add your ssh keys like this:
ssh-add /home/$USER/.ssh/id_rsa
Note: You may not even need to run 'ssh-agent bash' first step. 'ssh-add' may work fine as in my case.
Since Ubuntu 13.10 this works until your machine is on. Once you restart you need to go through those steps again.
Update: after adding below command on the bottom of ~/.bashrc it will continue to work even after restarting my ubuntu virtualbox machine.
eval `gnome-keyring-daemon --start`
You need to install gnome-keyring for above command to work if it's not already installed on your linux box:
sudo apt-get install gnome-keyring
But on my office laptop I get below error when I try to run first step: ssh-agent bash
** Message: couldn't connect to dbus session bus: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11
So, far I am not sure how to fix that, so I reverted my ~/.bashrc by removing line "eval `gnome-keyring-daemon --start`". Now I'll need to go through first 2 steps every time I restart my machine. But that's OK for now than asking password every time.
Let me know if you know how to fix that last issue.