When rightclicking on another Avatar the context menu offers as the first options to: Examine Duel which, as they are close together, can easily be misclicked, especially when clicking faster.
I have complained about this many times in the past to no avail. I personally feel that if you have Auto-Decline Duels turned on in the settings then Duel should not even be available.