Hmm, there were some E4 member doing something off-duty like Lorelei dealing with team rocket on fourth island , Bruno regularly visisting Sevi islands Spa , as of lance it isn't clear if was so free spirited as E4-member , Koga is said to patrol the Safari zones, whcih are all not places of their representative reign
and in the manga and anime the E4 members are even more off-duty then gym leaders.
E4 Phoebe often trains in Mt. Pyre, Flint visits his friend volkens for a sparring match, Lucian is seen at the Canalave Libary for some reading. they're prety much spending free time and in some cases even go out region