He did not lose his harem, they just went back to their country to give birth to the last 2 guys you see in the last page. They're his sons. Many years have passed and the TV shot is a recording. They came back in the last panel and you see their pregnant bellies in the second last page. In the last page, they're like "HOOOOO we're baaaackkkk. Brought some souvenirs~" There was 3 pregnant bellies, but we only saw 2 kids. I wonder what happened to the third one.
edit:MISCARRIAGE is the logical conclusion, but that would be too macabre.