I recommend re-creating the LUKS device with --pbkdf pbkdf2 to switch to the "old" (used to be default in LUKS1) key derivation function PBKDF2 which doesn't use extra memory. Alternatively you can also use --pbkdf-memory to force lower amount of memory for the default Argon2i.