To pick the most comfortable size I would have to say 7 1/4 including the headbandAll good felts benefit by an occasional brushing. The "hairs" you are seeing are likely rabbit to create that distinct fuzzy finish, and may not necessarily need a brushing... Yet!
My question for you, sir, is your hat size. Which one of the four sizes between 7 and 7-3/8 are you?