Mostly guesswork. Mom always figured he had a mistress, she'd go around kicking up a fuss and ordering servants to track him down when he disappeared. Plus he spread some rumour about getting a pile of gold from 'Beatrice the golden witch'. Then when he got to be old and senile he started ranting about being in love with her.