[Kireien]

This has happened to me very rarely and I don't know what causes it, but ways to prevent it from happening is to unclick "Gather Gardening." Very few seeds and soil are worth after all.
Also try using "Valid Slots" instead of an item priority if you only want one thing and to focus on it. It will keep on picking at that very same spot for that marker no matter what and won't wander like what is being shown here.