So I got it working now, but the issue is that one of the markers in The Iron Feast makes me face in a way where the fishing skill doesn't light up. I have to manually change my facing direction for it to start fishing.
The marker name is "The Iron Feast: Thunderbolt Eel"
The marker name is "The Iron Feast: Thunderbolt Eel"