I just want to give you a tip. On the map you have there. The "cliffs" there should go all around the town, because it is kinda awkward when they are so large and they are only there they are.
If you still want to keep the design I would suggest using ledges:)
There are also many flaws here...