You may have to make use of the gpu_memory_limit and/or lora_on_cpu config alternatives to avoid managing outside of memory. If you continue to operate out of CUDA memory, you'll be able to try to merge in process RAM https://www.asgdfx.com/