Insta360 x3 - Proper way of using direction lock?

Went out for a jog with my new x3. Tried using the direction lock function by turning it on then set a keyframe at the start of the footage to the direction I want.

This is how it is supposed to work am I right? But I find myself having to set new keyframes along the way because it will 'lose' direction and start showing the side view, then I have to correct it by setting additional keyframes.

Maybe I move my camera too much during recording, and the editor gets confused because of this?