Living with other humans is about compromise. There are trade offs. You give, you get. Sometimes the deal is equal and fair, but more often it's not.
Eventually, no matter how much you care about them, no matter how much they care about you, they are going to do something that pisses you off. You are going to do something that pisses them off.
Eventually, you reach a point where you don't care enough to let it slide anymore. Eventually, they reach a point where they don't care enough to even think about whether it pisses you off or not.
This goes both ways, for every person who has other people in their lives. Be it individual, family, or just society.
Everyone deals with this in their own way, in their own time. Some cope and adjust, some don't.
There is a reason some folks just say fuck it, and live their lives alone.