Yuntao Zhang

Yuntao Zhang

he/him

Senior Scientist | AI & Mass Spectrometry for Biologics

Domino Data Lab Environment Setup & Docker Container Lessons

7 minute read

Published:

Topic: Setting up Python 3.11 + CUDA environment in Domino Data Lab

Table of Contents

  1. Initial Problem & Requirements
  2. Understanding Domino Compute Environments
  3. Environment Customization Strategy
  4. Technical Issues & Corporate Proxy Resolutions
  5. Production Solution: Official System-Level Installation
  6. Key Lessons Learned

Initial Problem

User’s Requirement:

  • Need Python 3.10+ with CUDA support for machine learning work
  • Working in Domino Data Lab platform
  • Confused about available compute environments
  • Initially thought Spark was needed (turned out to be unnecessary)

Initial Confusion:

  • What is “Domino 6.0 Spark compute environment”?
  • Which environment to choose for Python 3.10+ and CUDA?
  • Difference between various compute environment types

Understanding Domino Environments

Environment Types in Domino

  1. Domino Standard Environment (DSE) - Complete set of libraries and packages
  2. Domino Minimal Environment (DME) - Lighter with fewer packages
  3. Custom Environments - User-built environments for specific needs

Available Environment Categories

  • Standard Data Science environments
  • GPU/CUDA environments
  • Spark environments (for distributed computing)
  • Custom Docker-based environments

Key Insight: Spark is NOT needed

  • Spark = Distributed computing across multiple machines
  • User’s need = Individual ML work with GPU acceleration
  • Solution = Standard Python + CUDA environment

Environment Creation Process

Initial Approach: Search for Existing

  • Looked for environments with both Python 3.10+ and CUDA
  • Problem: No pre-built environment matched exact requirements
  • Discovery: Domino allows custom environment creation

Solution: Duplicate and Modify Existing Environment

Steps Taken:

  1. Found base environment: domino-dse5.3-cuda11.8 (had CUDA 11.8 but Python 3.9)
  2. Duplicated environment using Domino’s interface
  3. Renamed to: domino-dse5.3-cuda11.8-py3.11
  4. Added custom Dockerfile instruction to upgrade Python

Environment Configuration Interface

Domino Environment Editor:

  • ✓ Environment Base (inherit from existing)
  • ✓ Supported Cluster Settings (none/Spark/Ray/Dask/MPI)
  • ✓ Dockerfile Instructions (custom modifications)
  • ✓ Pluggable Workspace Tools
  • ✓ Run Setup Scripts
  • ✓ Environment Variables
  • ✓ Advanced Settings

Technical Issues & Solutions

Issue 1: SSL Certificate Verification Error & Enterprise CA Trust

Error Message: CondaSSLError: Encountered an SSL error. Most likely a certificate verification issue. Exception: HTTPSConnectionPool (host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/main/linux-64/current_repodata.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (ssl.c:1129)')))

Root Cause: Corporate enterprise networks frequently deploy egress proxies or firewalls that perform deep packet inspection (DPI) and SSL re-encryption using private corporate root CAs. Because standard base Linux images only package public public-key infrastructure (PKI) roots in /etc/ssl/certs/, package managers like Conda and Pip reject incoming intercepted certificates with CERTIFICATE_VERIFY_FAILED.

Security Anti-Pattern Warning (ssl_verify: false): A common quick-fix is disabling verification entirely:

# INSECURE ANTI-PATTERN - ISOLATED DIAGNOSTIC PROBE ONLY
RUN conda config --set ssl_verify false && conda install -c conda-forge python=3.11 -y

Setting ssl_verify false completely eliminates TLS cryptographic validation, exposing container builds to Man-in-the-Middle (MITM) attacks and malicious package spoofing. This flag must strictly be treated as an ephemeral diagnostic check in an isolated sandbox to confirm that proxy interception is the failure cause, and should never be retained in production Dockerfiles.

The Enterprise-Grade Resolution (Inject Corporate Root CA): The correct production pattern is to inject the enterprise root CA certificate directly into the container’s trusted certificate store:

  1. System Trust Store Update:
    USER root
# Copy corporate proxy / firewall root CA into system certificates
COPY corporate-ca.crt /usr/local/share/ca-certificates/corporate-ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/corporate-ca.crt && \
    update-ca-certificates
  2. Explicit Conda & Python Environment Configuration: For package managers or Python tools that do not automatically inspect the OS-level store:
    # Configure Conda to validate against the updated system bundle
RUN conda config --set ssl_verify /etc/ssl/certs/ca-certificates.crt

# Set environment variables for Pip and Requests
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

Issue 2: Long Build Times & Conda Solver Bottlenecks

Observed Behavior:

  • Build process took 30+ minutes
  • Multiple dependency resolution attempts
  • “failed with initial frozen solve. Retrying with flexible solve”

Explanation:

  • Python 3.9 -> 3.11: Major version upgrade
  • Complex dependencies in scientific computing stack
  • CUDA compatibility checks required
  • Network latency downloading packages

Normal Build Process:

  1. Collecting package metadata (current_repodata.json)
  2. Solving environment (frozen solve) ❌
  3. Solving environment (flexible solve) ☑️
  4. Collecting package metadata (repodata.json)
  5. Final dependency resolution… (in progress)

Issue 3: Jupyter Notebook Startup Failure

Error Message: ModuleNotFoundError: No module named 'notebook.notebookapp'

Root Cause:

  • Python 3.11 upgrade changed Jupyter architecture
  • Jupyter Notebook 7.0+ restructured/relocated notebook.notebookapp module during architectural changes
  • Default Domino workspace configuration expects classic Notebook interface

Solutions Attempted:

  1. Updated Dockerfile - Added specific Jupyter packages:
    RUN pip install jupyterlab notebook nbconvert ipykernel
  2. IDE Selection - Switched from “Jupyter” to “JupyterLab” in workspace creation

Resolution:

  • JupyterLab works perfectly with Python 3.11
  • Classic Jupyter Notebook fails due to module incompatibility
  • Root issue: Jupyter ecosystem evolution and backward compatibility

Production Solution: Official System-Level Installation

Discovery of Official Documentation

After the in-place conda solver upgrade failed, official Domino documentation provided a dramatically cleaner architecture: installing Python 3.11 via system-level APT packages while redirecting conda symlinks.

  • Source: Installing Python 3.11 in a Domino Compute Environment

Why the Official Method Works Better

| Aspect | Conda In-Place Upgrade (Failed) | Official System Overlay (Success) | | :— | :— | :— | | Strategy | Upgrade existing Python inside conda env | Install Python 3.11 via system APT packages | | Dependencies| Must resolve 200+ pinned scientific packages | Bypasses brittle conda SAT solver resolution | | CUDA Handling| Tries to maintain complex conda ABI matrix | Preserves existing validated base CUDA drivers | | Build Time | 1+ hours (failed / timed out) | ~3 minutes | | Success Rate | Failed | ✓ Successful |

Final Working Dockerfile

USER root

# Clean problematic repositories and install Python 3.11
RUN rm -f /etc/apt/sources.list.d/pgdg.list && \
    apt update && \
    apt install software-properties-common -y && \
    add-apt-repository ppa:deadsnakes/ppa && \
    apt update && \
    apt install python3.11 python3.11-distutils -y

# Set Python 3.11 as default python and python3 commands
RUN update-alternatives --install /opt/conda/bin/python python /usr/bin/python3.11 1
RUN update-alternatives --install /opt/conda/bin/python3 python3 /usr/bin/python3.11 1

# Install pip for Python 3.11
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11

# Install PyTorch with CUDA 11.8 support and scientific packages
RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 && \
    pip install jupyterlab notebook nbconvert ipykernel && \
    pip install numpy pandas matplotlib scikit-learn

USER ubuntu

Key Implementation Techniques

Rather than struggling with conda solver conflicts across hundreds of pre-installed packages, this Dockerfile relies on two essential system-level techniques:

  • ppa:deadsnakes/ppa for Clean Python 3.11 Installation: Adds Ubuntu’s trusted deadsnakes PPA to install python3.11 and python3.11-distutils directly at the OS level without breaking system libraries.
  • update-alternatives for Seamless Path Redirection: Configures /opt/conda/bin/python and /opt/conda/bin/python3 to point to /usr/bin/python3.11. This ensures Domino’s workspace launcher and Jupyter kernels execute Python 3.11 while retaining the existing base CUDA 11.8 runtime.

Build Results: Success!

  • Total build time: ~13 minutes
  • Total process time: 3 minutes coding + 10 minutes Docker build/push

Successfully Installed:

  • Python 3.11.13
  • PyTorch 2.7.1+cu118
  • All NVIDIA CUDA packages (cublas, cudnn, etc.)
  • Complete scientific computing stack
  • Jupyter notebook environment

Final Status:

  • ✔ Environment builds successfully (~13 min total)
  • ✔ JupyterLab starts without errors
  • ✔ Python 3.11 + CUDA 11.8 + PyTorch working with full GPU acceleration (NVIDIA A10G)

Key Takeaways

  1. Vendor Guidance First: Always check official vendor documentation before attempting custom package builds. The official system-level recipe took ~3 minutes of configuration versus over an hour of failed conda solver resolution.
  2. System-Level Over Solver Conflicts: In pre-baked ML containers, installing Python via system packages (apt + deadsnakes) and redirecting symlinks avoids the brittle cascade of re-resolving hundreds of pinned scientific packages.
  3. Preserve the CUDA Foundation: Choose a base image with the desired CUDA and driver version already validated. Rebuilding CUDA from inside an unprivileged container is impractical, whereas Python runtime versions can easily be overlaid on top.

Leave a Comment

You May Also Enjoy