Yeah, you're able to do so. 'Cause everytime you save in the castle, you start on the gate that leads to the garden before Hyrule Castle.
So you can leave anytime. And the Fortune Teller in Castly Town tell you where you can find the heartpieces. =3
So you can leave anytime. And the Fortune Teller in Castly Town tell you where you can find the heartpieces. =3