I don't recall it being such a big annoyance before. But it seems like the NPCs that have any kind of schedule / travel pattern don't like to stop when you want to talk to them. I ran into an Adventure Trainer just as he was heading to bed. I stopped him 4x along the way... and finally got him to train me in some fishing skills as he was laying in bed. Similarly for Kinga or Ronin. If they feel like walking, conversations break.