You may experience problems using ssh after reinserting the card on 64-bit platforms, caused by a unexpected sequence of implicit casts in the C language when assigning aritmetic results to 64-bit variables (or other types larger than an default int).
There is a slightly annoying bug in gpg-agent up to and including version 2.0.0 where it will permanently loose the pcsc connection if any operation is attempted while the card is removed. The cause is an quite unexpected 64-bit issue in the C language which is far from obvious even to the most experienced C programmer.
Patch to gnupg-2.0.0 can be found in the GnuPG BTS Issue #724, and hopefully it will make it into the 2.0.1 release.
The root cause is a quite unexpected implicit sign extension when assigning the results of << operations to long variables.
(long)(((unsigned char)0x80) << 24)
or
(unsigned long)(((unsigned char)0x80) << 24)
becomes 0xffffffff80000000 due to sign extension of the result instead of 0x80000000 what most people expect from the input being unsigned. The reason is that the (unsigned char) gets automatically promoted to an int as part of the evaluation of <<, and int is signed.. then when this is cast to a long or unsigned long it's sign extended making a bit of a mess..
If you are bitten by this issue you can recover functionality until the next time the card is removed by forcing scdaemon to restart
pkill -9 scdaemon