I forage for food as a hobby, and noticed that the attaching parts of sumac stays with the berries sometimes, so it got me to wondering if the wood of edible sumac is poisonous for human consumption.
For a side note I live in MN, USA
The simple, common-sense answer is no, a small amount of sumac wood (such as could be accidentally consumed with sumac tea) is not poisonous for human consumption. We can conclude this from the fact that A) normal sumac tea preparation methods can include some of the wood, and B) people don't get sick from sumac tea prepared by these methods.
People have been consuming sumac tea (or sumac "lemonade") for hundreds, possibly thousands of years. As you described, it's normal that when you break the cluster of berries off the plant, you get some of the woody stem with it. It's also normal to include the entire berry cluster (stem and all) when you steep the tea. If the wood was poisonous, people would certainly have realized that it was making them sick, and they would have stopped making sumac tea, or at least started removing the stem from the berries before steeping them.
There are numerous descriptions of how to harvest sumac berries and make them into tea. Here's one from Edible Wild Plants of Eastern/Central North America by Lee Allen Peterson (copyright 1977):
Use: Cold drink. When ripe, the hard berries are covered with acidic red hairs. Collect the entire fruit cluster, rub gently to bruise the berries, and soak for 10-15 min in cold water. Remove the cluster, and pour the pink juice through cheesecloth to strain out the hairs and any loose berries. Sweeten to taste and chill; tastes like pink lemonade. Gather the clusters before heavy rains wash out most of the acid.
As you can see, Peterson makes no mention of removing the woody stem before steeping.
In The Forager's Harvest by Samuel Thayer, Thayer goes spends several pages describing how to harvest and prepare good sumac tea. The description is clearly based on extensive personal experience. He makes no mention of any need to avoid the woody stem. In fact, he actually says that the raw shoots of sumac are edible and delicious.
In Incredible Wild Edibles: 36 Plants that can Change your Life by Samuel Thayer, Thayer mentions using sumac wood to make taps for collecting maple sap:
You can also make your own wooden spigots... Sumac, elderberry and highbush cranberry are traditionally used for this; they have pithy centers and are commonly found at about the right diameter. (Some people warn against using elderberry because of a toxin in the bark, but I don't think this has any merit. Just don't eat your spigots.)
It's notable that he mentions the possible toxicity of elderberry wood, but makes no mention of any similar concerns for sumac wood.
I have personally prepared and consumed sumac tea without getting sick. I didn't have any cheesecloth to strain it through, so the final product was rather fuzzy, but definitely not poisonous.
Note: The edible sumac I'm referring to here is any of several red-berried species of sumac (Rhus spp.) common throughout North America, including smooth sumac (R. glabra), staghorn sumac (R. typhina) and fragrant sumac (R. aromatica). It does not include poison sumac (Toxicodendron vernix), which has white berries. Poison sumac is closely related to poison ivy, and gives a similar (although much more severe) skin rash if you touch it. Poison sumac is not edible.