More info. I can install to ~/.gem as non-root, even without gemhome pointing to ~/.gem/... in .gemrc. However, I can only uninstall from there if I add an entry for gemhome pointing to ~/.gem in .gemrc.