The cannon balls on a man o' war were stacked and held in place by a brass ring on the deck known as a brass monkey. In cold weather the sea spray would turn to ice thus freezing the cannon balls together and onto the ring.
Hence the expression "It's cold enough to freeze balls on a brass monkey" or simply "brass monkey weather". Note that the correct phrase is "on" NOT "off" a "brass monkey". The expression is often corrupted to hint at an association with a man's testicles balls