While making that codepen i noticed that my issue is not present there and that made me realize that it may have to do, as usual, with the url bar, which is not hidden/shown but always present in full page view.
The problem is that when the navbar appears and disappears it triggers "adjustments" after the scrolling is done while using gsap to change backgroundPositionY .