Without a doubt it's a router issue,your brothers system is hogging the port that's needs to be open for party chat.
Running two systems of the same type on the same network can be a struggle with most ISP supplied routers especially when both are connected wirelessly. Connecting one or both systems by ethernet cable or powerline network adapters may help,but both over wireless will almost certainly cause problems.