I'm a BJJ guy and I feel the same way, I practice taking off my jacket quickly as the lack of mobility would be awful in an altercation.
But a leather jacket will protect your back if you pull guard! Without a leather jacket you are going to stab your back full of needles, glass shards and legos!