Well seeing how the map is right now I would say behind the fortress would be most logical, but it could be elsewhere as the fortress could simply be build there becuase it is an easely defendable position. That would explain why there would be two watchtowers watching towards the exit of the tavern. It would mean the guards in the fortress would serve as a detection and attack in the back unit rather then a keep the enemy on the other side of the doors, but that is a tactic often used so wouldnt be that strange.