Background

I've recently run into a storage panic when my DIY-built Machine Learning desktop is running out of storage in the system SSD drive (128 GB). After desperately searching online for various options, I found this approach to be by far the best and the easiest/fastest way to seamlessly move your entire conda environment to a new/larger storage in no time.

Environment

This approcah is written for Linux/Ubuntu machine, but you might be able to leverage the same methodology on other operating systems.

  • Ubuntu 22 LTS
  • Conda env: miniconda
  • Current storage: system ("/")
  • Target new storage: newly mounted SATA SSD ("/data/")

How to

The key is: "symbolic link". Yes it's that simple but effective.

Here are the steps

  1. Copy your entire conda environment (under "/home/[username]/.conda" or "/home/[username]/miniconda3" -- this is what I use) to your new storage drive via

    cp -r /home/[username]/miniconda3 /data/

    The command will create a new a new folder under /data/ with the same name ("miniconda3")

  2. Remove/rename your old conda environment

    # for remove
rm -rf /home/[username]/miniconda3
# for rename (for archive/backup purpose)
mv /home/[username]/miniconda3 home/[username]/miniconda3_backup

  3. Create a symbolic link in your working environment folder (but the actual environment is already moved to new storage)

    # command usage: ln -s [data_storage_path] [symbolic link path]
ln -s /data/miniconda3 /home/[username]/miniconda3

  4. Done! Later when you create new conda environments, the symbolic link will automatically use the new storage location to storage the files, and you still have your previous environments intact!

Reference Link

https://saturncloud.io/blog/how-to-specify-a-new-environment-location-for-conda-create-a-guide/
https://stackoverflow.com/questions/37926940/how-to-specify-new-environment-location-for-conda-create