Most answers curiously omit a very dangerous reason why removing anything should be illegal, unless designated "affirmative" (that is, you are explicitly given permission to take away). You don't mention where you intend to take your mementos, and dangers lurk here as well.
All rocks, animal parts, wood, seeds, flowers, food, etc contain things local to the environment.
Removing an item can have the deleterious effect of defacing the area from which you take things. Burial sites may contain items that appear to any, other than archaeologists, as "just a stone". You can alter the movement of streams and other waterways, damage an animal/bird/reptile habitat. As mentioned, if everyone took a flower or a rock, then eventually, the area would be scavenged into oblivion.
Importing an item to your own area can have the deleterious effect of bringing in blight/disease, or an invasive species (whether insect, microbial, or plant). In many of the scout camps we go to, there are strict rules that require us to use only local wood for burning - we are forbidden from using wood we bring from elsewhere, including that which we bought from a supermarket. We either use what we find on the ground, or, as in many camps, the forester will cut wood for us and we can use that. In New York, you may not bring in wood from further than 50 miles, and if you bring in any wood at all (within 50 miles), you must provide source documentation.
Frequently Asked Questions for Firewood Regulation and 2012 Revision (Department of Environmental Conservation)
Forests (Department of Environmental Conservation)
There's even a quarantine:
EAB Regulations and Quarantines (Department of Environmental Conservation)
Just the mere attempt to get at artifacts - even pictures! - can deface an area that is historically or ecologically important or sensitive. Human presence near some wildlife sites can have negative effects, like bird nests which can cause birds to abandon eggs. Turtle and other reptile eggs can be destroyed. Damage to a beaver dam can have downstream effects.
And sometimes, merely touching things can be dangerous, particularly with animal parts. Here, rabies, parasites, anthrax, and other diseases are high-risk to human interaction. So, leave the antlers, skulls, snake heads, and turtle shells alone.
Oh, and animals? Good luck trying to take a dead mountain lion or eagle you find on a trail head and stuff it. Taxidermists are forbidden by law to process a protected animal - even if you find it already dead. There's permits, paper work, red tape, and bureaucracy that are required to do this.
Taxidermists & Federal Law (U.S. Fish & Wildlife Service)
You specifically mention taking a live plant and replanting it at home. You may be allowed to remove from one area, but be prohibited from importing it to another. In fact, transport across country boundaries is generally illegal. It's not unusual to try to take home some plant or animal product you purchased at a gift shop - like a sea shell, starfish, pine cone, snake head, or some other animal or plant by-product, only to be told by customs you cannot bring it on the plane or boat. Even some foods are generally illegal to take across borders. Food which is processed into oblivion - like soda or milk - is usually okay. Or canned foods. But raw food? Or dried jerky or grain? Conch shells from Barbados? Dried starfish from Australia? Sand dollars from Martinique? Or a bottle of wine with a dead snake in it from Vietnam? Or that dried corn husk from Mexico to make tamales? It all could be confiscated, even if you purchased it from a super market or gift shop.
Conversely, you may be encouraged to take (or rather, destroy) things, but would generally be prohibited from taking anywhere else. Kudzu is an infamous example, such as in South Carolina where at one point a few years back the governor declared war on the invasive plant. All culling by any means available was encouraged. So you were allow to kill/destroy as much of it as you find, you just aren't allowed to bring it anywhere - because those places likely have kudzu infestations themselves.
And as to food: Sometimes, it's not about the danger to the environment, or to yourself or others. Sometimes, it's about the damage to corporations. Yep: Try going to a foreign country and bring in a food product (like seeds) which are not genetically engineered to produce fruit without seeds. Monsanto spent billions convincing us that seedless watermelons are better because, well, who wants to eat seeds?? - when the real issue is they want you to be buying their seeds year after year. Seeds will be confiscated at customs checkpoints.
Along the lines of damage to corporate bottom lines, there's also the damage to government bottom lines, too. Sure, alcohol is not a memento you're going to find. But tobacco leaves? Our car got searched because a dog hit on tobacco. We were allowed to keep the cigarettes because the woman proved she bought them legally. But the customs officer (or whatever they're called when they're on the reservations) told us that all tobacco products are subject to confiscation if we can't prove we paid tax on it. And then there's the drug plants, like cocoa, marijuana, poppies, and various mushrooms, which I'm sure you didn't intend to include as mementos, but they nevertheless fall into your category of taking live plants and transporting them somewhere else.
In the end, you may think you are contributing to a species by propagating them, or making your house guests aware of what you've found. But you may be doing more harm to the area you took them from, or the area into which you brought them, or to the people whose livelihoods rely on them and you took or stole from them.
So, besides mentioning your intentions with your mementos, you need to be up on the local laws of the place you're taking things from; the laws of the place you're taking things to; and the laws of the places through which you travel to transport things, all in order to know whether such a thing is allowed to be taken to or from. It's just not feasible to answer here with anything more reasonable other than "you should not remove anything". (And be careful about those pictures).
take nothing but pictures
is part of the leave no trace philosophy.