You're right. My main carry is an LCP, so it's much smaller. I also bought an XD subcompact for carry, but I've never actually carried it. I still haven't bought a good holster. That and the LCP in my wallet holster is so easy to carry it's ridiculous.
Pics for reference
EDIT: Also to the OP, you may want to consider something a little more simplified, like this LCP or a glock. You don't want to
need to use your gun in a defense situation and have to mess with a safety. Just my personal opinion. The instructor for our CCW class recommended guns with no safety and always have a bullet chambered and ready to fire.