In the aftermath of Rollout storage, I managed to hypothesize a phenomenon that, upon testing, seems to be accurate: specifically, the game only stores 32 bits at a time in the middle of damage calculation, and results that overflow that range by only a slight margin can wrap around to pretty low values.
Testing with
DaWoblefet produced the following:
Battle Video 1: GBXG-WWWW-WWWL-YQNC
Battle Video 2: AHMW-WWWW-WWWL-YQN7
Battle 1 is based on the calculation
42 * 15840 * 6456 = 4,295,047,680, where that last number represents 2^32 plus 80,384. I found that breakdown by looking through multiples of 17280 (a factor that setups of this type are highly likely to have) to find one that breaks down into three components that can reasonably exist separately in the same damage calculation. Specifically,
42 is what level 100 gets morphed into before it gets multiplied by anything,
15840 represents a 220-power move that has two Helping Hand boosts applied plus the full 32x effect of Rollout storage, and
6456 can be obtained as an attack stat by starting with a stat of 269 and applying +6, Light Ball, Huge Power, and Flower Gift in sun. Since it had to be a physical move to get all those multipliers in place, Power Trip (with exactly 10 boosts, 6 of which are already accounted for in attack) immediately came to mind as the move to use. Pikachu is incapable of naturally going to to 269 on its attack stat, so it had to receive the benefit of one Power Split.
As I assumed, if the register could only hold 32 bits there, then instead of 4 billion-plus, the formula was dealing with just 80384 at that point. The next step in the damage formula was a division by the opponent's defense (which was 1 in this case, thus a no-op) then a division by 50, reducing it to 1607 (which then meeds the static +2 to go to 1609), even if it "rightfully" should have been around 86 million if the system used 64-bit registers. Still too big to be compared against a health bar, but at least within striking distance.
The goal, then, instead of taking Hoopa's Psychic/Ghost typing and generally going overboard on positive multipliers, was to squeeze in lots of fractional multipliers to
reduce the damage, but only ones that specifically do not show up until after the division by 50. First up is the random roll, which diffuses from a straight 1609 to a range between 1367 and 1609. Then there's typing, using Pangoro's type to get a double resist (341 to 402), and the fact that Pikachu could be given a burn. Since the transformed Mew kept Smeargle's low defense stats but kept its own HP, then, I figured it would survive with a damage range between 170 and 201. (Reflect could be added on as well, but that would have made the setup longer and wasn't necessary to get into a range that Mew survives.) If not for the overflow in mid-calculation, the damage would have been around 10 million--clearly too much for anything to handle.
Instead, the actual damage was
74, which evidently agrees with the fact that an overflow of some kind was happening, but doesn't agree with where I had it positioned.
One thing I wasn't certain about here was whether Power Trip's variable power even plays nicely with the Rollout multiplier that got stored. After all, Smogon's documentation on the damage formula dates back to G5, a time when Power Trip didn't even exist, so I don't know the mechanism behind their modifier, or where it lands in modifier order (the three most logical guesses would be next to Punishment, next to Stored Power, or at the very end of the move-specific code). As such, I wanted to develop at least one scenario that didn't depend on Power Trip as the move.
The numbers I found for that, giving rise to battle 2, were
40 * 18720 * 5736 = 4,295,116,800, which is 2^32 + 149,504--a larger margin of overrun than in battle 1, but still possibly low enough to work with. That breaks down as
40 calling for an attacker's level of 95, 96, or 97 (instead of 100),
18720 as before but where the move power is 260 instead of 220, and
5736 calling for Pikachu's starting attack stat to be 239 instead of 269. That's still too high to reach innately, but it's just one Power Split away all the same, and the infrastructure was already there so it's no more of a hassle. 260 seems pretty high to reach with a move as well, but instead of using Power Trip + 12 boosts, I noticed that it could also be realized as Bolt Strike + Charge.
With that in place, we assume that the register wraps around to 149504 after multiplying all those factors together, then divide by 1 again, then 50 (2990), then add 2 (2992). The random roll gives the range 2543-2992, then the double resist gives 635-748, and the burn makes it 317-374 which is just enough reduction to fit within Mew's 404 HP.
As it turns out, the actual damage in this case was
347, which is right in line with that estimate; in fact it corresponds to the 93% damage roll.
Based on the results of these tests, I'm inclined to believe that I have the overflow mechanism spot-on (if an intermediate result of a calculation at any point in the damage formula goes over 4294967295, take the mod-2^32 at that point and proceed with only that portion), but since the damage from Power Trip was so much lower than expected (by less than half), there seems to be something about that move, and possibly also Stored Power, that isn't adequately captured in our current understanding of the damage formula.