Shell and DoC Systems

Department of Computing | Imperial College London

home

Shell and DoC Systems, Lab 1

The Linux Terminal and Scientia

Chapter 3: Copying, Moving, Removing

Temporary Storage of Large Files

face Tom Crossland

In addition to your personal home directory, there is another storage area available to all members of DoC, located at /vol/bitbucket/.

Unlike your home directory, /vol/bitbucket/ is not backed up, and so there’s a chance data stored here could be lost if something goes wrong (it hasn’t happened in the department yet, but still).

So this area is for temporary storage of files that can be recreated if necessary, such as downloads of raw data, local installations of software (e.g. Python environments), or other non-essential files.

Do not store code files or other essential data in /vol/bitbucket/. The department cannot guarantee that data stored there can be recovered if there are issues with storage.

Also, there are limited file protections on data in bitbucket, so do not use it for storing any sensitive or copyrighted data (e.g. that you may have access to during your project work). The department has other potential storage solutions for these cases.

So, after that dire warning, what should you store in /vol/bitbucket/?

  1. Python environments (and make sure you have a requirements.txt stored elsewhere, so you can easily recreate the environment – which you should do anyway, even if you’re not using bitbucket!).
  2. Large data downloads (which you might have to do for projects later). This might include pretrained ML models, from repositories like HuggingFace.

How to use bitbucket

Each user should create their own directory within /vol/bitbucket/ to store their data. Use your username as the directory name, so the directory can be properly identified. We can use mkdir for this!

So, if your username was abc123, you would do the following:

username@MACHINE:notes$ mkdir /vol/bitbucket/abc123

If you ls inside /vol/ before accessing bitbucket, you may find that there is no bitbucket directory in there! Likewise if you use tab autocomplete when typing the path.

This is because /vol/bitbucket/ is accessed on an as-needed basis by the system, and so is only visible if you ask for it directly – for example, by cd-ing into it, or ls-ing its contents.

Once you’ve accessed bitbucket once within a login session, it should be visible for the rest of that session.

Exercise 10

Create a directory, using your username as the directory name, inside /vol/bitbucket/. You can use this directory throughout your time with DoC.

Why not add a quick file to it, and list the contents to make sure it’s there!