So I think this is evidence that the bootloader has at least something to do with the issue.
I think it's time to debug the bootloader - daunting!
Since this issue only seems to occur with really big sketches, e.g. 100k or more (every time I try to isolate the issue in a small sketch, it disappears), I suspect it's got something to do with register or memory addressing, and the bootloader simply assumes some addresses, which maybe are not reliably set to 0 by the atmega reset function. These addresses would normally be 0 with smaller sketches. At this point however, my theory is only speculation, just good as a starting point for debugging.