Freezing our balls off!
The saying "it's so cold out there it could freeze the balls off a brass monkey" came from when they had old cannons like ones used in the Civil War. The cannonballs were stacked in a pyramid formation, called a brass monkey. As pictured in the Puerto Rican castle below:
When it got really cold (not likely here in Puerto Rico, but in the North) the poorly constructed brass balls would begin to shimmy and crack. Some soldiers would use the term to describe a certain temperature. You knew it was bad when the balls were so cold they froze, cracked, and fell right off the brass monkey.
So there you have it... it was never testicles, even though I imagine most modern adaptations have a lot more to do with male genitalia than with 18th C. war-lingo.
Cheers to weird expression with even weirder origins.