[Slike]

Huh, I'm surprised nobody's ever asked that before!

What you actually need to do is go through what you can of Lavender Tower. The NPC at the end will trigger the flag that lets you go through to Saffron.