Building a container on sylabs in layers

Sometimes it is necessary to build very large containers that include many packages and dependencies.

Most HPC systems (such as Sylabs/Apptainer) impose a build time limit (BTL) of around 1 hour.
If your container requires many packages with complex dependencies, building it all at once may exceed this limit.

A practical solution is to build the container in layers:

Install the most essential dependencies first
Save this intermediate container
Add remaining packages in a second build

This approach keeps each build under the time limit and makes debugging easier.

Example: Seurat 5 container

Below is a definition file I used to build a container for Seurat 5.
Seurat 5 has new functionality for efficiently analyzing large single-cell datasets (up to 1M cells) and requires several R packages.

Full definition file

~~~bash Bootstrap: docker From: bioconductor/bioconductor_docker:devel %labels Version v0.0.1 %help This is a container to run Seurat 5 on Hawk %post # Update the image apt update apt upgrade -y # for igraph apt install -y glpk-utils libglpk-dev # for sctransform apt install -y libicu-dev # for BPCells apt install -y libhdf5-dev # Install R packages R --no-echo -e 'remotes::install_github("bnprks/BPCells")' R --no-echo -e 'BiocManager::install("glmGamPoi")' R --no-echo -e 'install.packages("RPresto")' R --no-echo -e 'install.packages("Seurat")' R --no-echo -e 'setRepositories(ind=1:3)' # Needed to automatically install Bioconductor dependencies for Signac R --no-echo -e 'install.packages(c("R.utils", "Signac"))' R --no-echo -e 'remotes::install_github("satijalab/seurat-wrappers")' R --no-echo -e 'remotes::install_github("satijalab/azimuth")' R --no-echo -e 'BiocManager::install(c("scuttle", "scater"))' R --no-echo -e 'install.packages("scCustomize")' R --no-echo -e 'install.packages("readxl")' apt clean ~~~

Instead of running the entire thing at once, which exceeds the BTL, I built the container up until the `R --no-echo -e 'install.packages("Seurat")'` line first:

### Layered build approach Instead of building everything at once, we split the container into **two layers**.

Layer 1 definition

This takes about 45 minutes to build. Once this base container is built, it can be stored and used as the root for the next layer.

Layer 2 definition

~~~bash Bootstrap: library From: library://dazcam/containers/seurat5a:latest %labels Version v0.0.1 %help This is a container to run Seurat 5 on Hawk %post # Install R packages R --no-echo -e 'setRepositories(ind=1:3)' # Needed to automatically install Bioconductor dependencies for Signac R --no-echo -e 'install.packages(c("R.utils", "Signac"))' R --no-echo -e 'remotes::install_github("satijalab/seurat-wrappers")' R --no-echo -e 'remotes::install_github("satijalab/azimuth")' R --no-echo -e 'BiocManager::install(c("scuttle", "scater"))' R --no-echo -e 'install.packages("scCustomize")' R --no-echo -e 'install.packages("readxl")' ~~~

*** Notes: 1. **`apt` commands** The `apt` commands in layer 1 install system-level libraries that some R packages require. Examples: - `glpk-utils` and `libglpk-dev` for `igraph` - `libicu-dev` for `sctransform` - `libhdf5-dev` for `BPCells` 2. **Base vs layered container** - Layer 1 uses a Bioconductor Docker image from Docker Hub as the base - Layer 2 uses the Layer 1 container stored in a Singularity library as its base *** Move back to [Build a basic container](/Bioinformatics-tutorials/pages/containers/01_build_basic_container.html), or back to [index page](/Bioinformatics-tutorials/index.html).