How much clothing depends on the season. If it's in the dead of winter, I'll be packing far more in the way of thermal duds.
But 'everyday' clothes like shirts, pants etc... I don't pack any in the bag. Skivvies and socks, yes.
I do maintain an 'accessory' bag that holds many ancillary items: Extra food, water, ammo, meds and clothing. But it's more of a gym bag. I'd grab it if I'm bugging out by vehicle.
If your bag is too full, you either are trying to carry too much, or your bag itself is too small. Either obtain a larger bag (remember... you gotta be able to carry it), or start making choices about what's really needing to be in it.