1) Any child would be your son/daughter, half brother/sister to the existing kids, and nephew/niece to your brother. I don't think I would lay this on the kids, the family or anyone else. 2) I have a brother who had an open marriage agreed to on both sides, but things started to get tense when a second woman and her child arrived in the house. They now get along well enough, though his first wife has another steady. The financial ramifications got tricky. 3) There's no reason you can't go out to dinner or on vacation as a threesome. As long as nobody is feeling guilty or jealous about the s_xual arrangements there's no reason to obsess about that either - just make sure to do a verbal check each time you try this.
I think it is important for you to figure out what problems they have been having with their marriage - probably something s_xual. How do they feel your participation will help - is one man not enough for her - does he have his eyes on someone else? You may end up stuck with your brother's family. When their problems are resolved will you then be dumped? There are any number of ways that someone could eventually get hurt. I think you would be well advised to use conraception so you know the next child is not yours.
The parents will not take kindly to this information - it will make the family look odd to their friends. Let them figure it out on their own if they want. Best of luck!