Abomination was a term used most commonly by the Bene Gesserit to describe individuals who could not control the ego-memories that had surfaced within them. These ego-memories usually surfaced after consumption of the Water of Life, or through direct genetic inheritance in pre-born children.
Abominations, along with those who possessed genetic memory, had access to vast knowledge. This, when combined with their psychological instability, those deemed Abomination were feared and considered very dangerous by the Sisterhood. Standing orders existed for Bene Gesserit that anyone believed to be Abomination were to be immediately killed.
The classic example of Abomination was Alia Atreides, who was a fetus when her mother the Lady Jessica converted the Water of Life and became a Reverend Mother. Being too young and weak to control the plethora of dynamic personalities that invaded her body via genetic memory, Alia was never able to adjust to her situation and eventually succumbed to the personality of Vladimir Harkonnen.
Other individuals manifested similar qualities, and thus were temporarily considered Abomination by some in the Bene Gesserit order. Leto Atreides II and his twin sister Ghanima Atreides were examples of this. As the children of the Kwisatz Haderach Paul Atreides, they possessed similarly formidable knowledge from their inception (they were pre-born) and would even play games together and temporarily allow the ego-memories of the Duke Leto Atreides and the Lady Jessica to surface and interact with each other. Unlike their aunt, Alia Atreides, Leto Atreides II and Ghanima Atreides were able to control these memories and avoid the depths of Abomination exhibited by Alia using disparate methods (Ghanima through the use of self-hypnosis, Leto II through a complex council/spokesperson). While Ghanima's escape of Abomination appears to have been successful, Leto II hints at the end of Children of Dune of the possibility that an ancient personality, Harum, may either have or share power over the actions of Leto II.