Is this right?

Many times when I sit down with people to have a conversation or just happen to become part of one, I find myself wandering off. After a while, some conversations just become words to me. Sometimes, I even start thinking of other things and feel like an idiot when caught wandering. I don't do this purposely. It just happens. No, I don't have ADHD and I doubt there is any "label" for my weirdness. Some conversations are just not worth it to me. Am I disrespecting these individuals by not listening? Or is it acceptable to do so when the conversation holds no value to you?