I have an old Marlin 38.40 lever action rifle and am having trouble finding ammo for it. Can I fire modern .38 cartridges in this rifle?
No, I don't believe so. The .38-40, perhaps despite its name is a 40 caliber round, while the .38 Special, certainly despite its name, is a much smaller .357 round. The case diameter dimensions are also much smaller, so a .38 Special round is unlikely to fit properly in your rifle.
In addition, it's generally considered to be a bad idea to fire modern ammunition in guns designed for much older ammunition, since modern ammunition tends to be loaded to far higher pressures that its earlier counterparts. This can cause catastrophic failures in firearms that weren't designed for it. In this specific case, you might be unlikely to have overpressure issues because of the .38 Special is so much smaller than a .38-40 round, allowing gas to escape every which way, but it's not a certainty since because gas will be escaping every which way it can potentially put pressure on components of the rifle that should be sealed from such forces by the shell case.
In any event, I wouldn't try this combination. It can only result in bad things.
The long and the short of it is no - definitely not. If you check wikipedia (or other reputable sources such as SAAMI) you will see that the parent case of the .38-40 is the .44-40 Wincheseter, which on the face of it makes them completely incompatible. To see why these rounds are actually incompatible you can search the SAAMI specifications for the .38 and the .38-40 rounds:
As you can see above, these rounds are not even remotely close. In general, though, the only rounds that will be compatible are rounds that share a common "ancestor", meaning that they have a common parent design. Basically what this means is that the new round is a necked-down or necked-up version of the parent design. For example, the .25-45 Sharps, and the .223 Remington. The .223 is the parent design, and the .25-45 is simply a necked-up version.
Typically You will find that rounds which are compatible are only necked-down rounds, meaning that you could use .25-45 Sharps brass and convert it to .223 Remington by running the round through your resizing dies, and then trimming the brass to the proper length. Working brass in such a manner will inevitably result in a very poor success rate, and is generally not recommended because it may weaken the metal; I can't remember whether brass work hardens. However, if you have a firearm chambered in a caliber which is no longer manufactured, it may be your only alternative.