just for curiosity: isn't this the old issue of printing on Windows? As QGIS on win is 32bit then it can use just 4gb of ram (actually much less) and then priting of large/complex layouts fails. In my experience the same layouts do print ok on 64bit Linux/Mac boxes.