You have to use dummy profiles for the followers that contain at least the ["MapID"] entry that corresponds to the dungeon you are doing. If they detect they are in the wrong map, they will attempt to leave.
Example:
https://www.dropbox.com/s/5m9vamfxh8y4ct....info?dl=0
Example:
https://www.dropbox.com/s/5m9vamfxh8y4ct....info?dl=0