One of Libra's flaws is their dislike of confrontation. As an Air Sign, you're capable of direct communication, but secretly want everyone to get along. You are just that kind and thoughtful!
So you're often loath to voice a conflict, or push the issue until it ends up in a fight. Part of you is too sensitive for that. Learning to say no in a firm but kind way is one of your sign's challenges.
You don't have to shout, or draw super firm boundaries. A simple no should let the other person know not to push you anymore. You may feel empowered by your new-found skill!