I'm going to be that one person who goes against everything.
The reason this wouldn't work is it could be abused to annoy people, for example someone could add you to a group with lots of people in and they can keep doing it if you leave, as well as this like stated by Shreya, there are many communication apps out there which you could make a group on with your friends - Slack, Discord, Skype - they are just a few.
I'm not saying this is a bad idea, I'm just giving some criticism so this suggestion can be improved to patch these, you could add a button to stop a user from adding you to conversations possible? This way people who do spam you with adding to conversations can not add you anymore.
+-0 from me