In Mac OS X, if you have a password protected ssh private key then every time you use this key to connect to a remote server, a password prompt will appear. If you want to avoid this popup then pass the below option to ssh in the command line:
$ ssh -o AskPassGUI=no <serverToConnect>
If you dont want to mention this every time then you can also mention this in ssh_config file.