Skip to main content

Uploading Datasets

To upload datasets, you will need to create an account at OpenCSG Community. Datasets are Git-based repositories, which give you versioning, branches, discoverability and sharing features. You can upload anything you want to the dataset repository.

We support two ways to upload files currently: using Git or web interface.

Upload Files to a Repository Using Git​

Downloading Repository​

After creating the dataset repository in the web interface, clone the dataset repository to your local machine and upload your dataset files with the git CLI.

  • Downloading dataset repositories using HTTP:
git lfs install
git clone https://portal.opencsg.com/datasets/demo/test_dataset_1.git
  • Downloading dataset repositories using SSH:
git lfs install
git clone git@hub.opencsg.com:datasets_demo/test_dataset_1.git

You will need to add your SSH public key to your user settings to push changes or access private repositories.

Uploading Dataset Files​

Assuming that your dataset files are located in the /work/my_dataset_dir local directory, you can upload your files to the dataset repository with the following command:

cd test_dataset_1
cp -rf /work/my_dataset_dir/* .
git add .
git commit -m "commit message"
git push

[Note]​

Files with the following suffixes are automatically uploaded with git-lfs:
.7z,.arrow,.bin,.bz2,.ckpt,.ftz,.gz,.h5,.joblib,.lz4,.mlmodel,.model,.msgpack,.npy,.npz,.onnx,.ot,.parquet,.pb,.pickle,.pkl,.pt,.pth,.rar,.safetensors,.tar,.tflite,.tgz,.wasm,.xz,.zip,.zst,.tfevents,.pcm,.sam,.raw,.aac,.flac,.mp3,.ogg,.wav,.bmp,.gif,.png,.tiff,.jpg,.jpeg,.webp

If there are other types of large files, run the following command to make them upload as lfs:

git lfs track <your_file_name>

Upload Files to a Repository Using Web Interface​

To add files to your repository with the web interface, start by selecting the Files tab, and then clicking Add file. You will be given the option to create a new file or upload a file.

Add new file

Creating a New File​

Click Create new file, add the contents and click Create File to save your file. Add new create

Uploading a File​

Click Upload file, you can choose a local file to upload.

Add new upload