If you're referring to the Watchtower ladders in the Bronze doors region, climbing them is simply a matter of jumping a bit to get up. My only problem is that sometimes I hop too much and go overboard.
I remember at one point that I flew straight into the abyss
Workarounds:
Most ladders thought can be climbed by simply walking on them. One exception - the tower of Hydlaa, where a last step is missing. There, you have to be very careful and jump up and to the left a bit. Changing cameras also helps a bit to visualize your position better.
Going down ladders is very easy - walk onto one and move a tad sideways. Then, use the strafe button to go down that ladder just as would expect it to happen normally.
Hope it helps!