Do not use glue! That is the worst you can do.
+1. I am not sure about the sweatband either. I seem to recall that Art Fawcett was chasing some down thin kangaroo leather sweats a lá Champ Featherweight, for use on his ArtLite hats. I don't recall reading about the end result, but he may well have sourced them.