His wife Catherine was unable to have children. Charles "volunteered" her for a eugenics programme to create advanced soldiers. The resulting child, Emma, was largely kept away from her. Charles also had a large number of affairs in later years but the Waverly's lived together for appearences.
In early 1965, he attempted to gain control of the Intrusion Countermeasures Group through underhanded means and put his Emma (under an alias) in charge. When Catherine found out what her husband had done to Emma, she killed him. (AUDIO: Manhunt)