Species living in a "harem" structure with one male mating with several females had larger hyoids and smaller testes. Species living with multiple males and females in a group, all mating with one another, had larger testes and smaller hyoids.
It's likely that the harem-heading howler monkeys use their deep calls to attract more females, or perhaps to scare off males that might steal their mates, Knapp said. As a result, they can get away with a lower sperm load - they compete for females before the actual mating act.
For male monkeys living with other males, though, the competition comes after copulation. A greater volume of sperm may give these monkeys a fertilization advantage in a female monkey that has mated with multiple males.
It's probably impossible for howler monkeys to evolve both a large hyoid and large testes, Knapp said.
"If they could, they would," she said. "I think, because they don't, it suggests that you can have one or the other, but you can't have both."
To really nail down the notion that monkeys make this reproductive trade-off, the researchers will have to look within species to see if large hyoids or large testes confer real advantages for males.
"We kind of have answered a question that Charles Darwin pondered in the 1800s, so that is exciting for us," Knapp said. "It also opens the doors to more questions, like what is going on within species? Is there evidence that males with the very largest hyoids are the ones having the most kids?"
The study also raises questions about what humans might have in common with their distant monkey cousins. In people, men with deeper voices are considered more attractive by women than men with higher-pitched voices, studies have found, but men with deeper voices don't have better-quality semen.
More from LiveScience:
Animal Sex: 7 Tales of Naughty Acts in the Wild 5 Things a Person's Voice Can Tell You Boa Constrictor Eats Howler Monkey - First Look | Video Copyright 2015 LiveScience, a Purch company. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.
Article originally appeared on LiveScience.