Acc-Py

Accelerating Python (Acc-Py) represents a comprehensive Python distribution and ecosystem developed specifically for CERN’s Accelerator and Technologies sector. The system addresses the operational requirements of accelerator control environments while providing access to Python’s extensive third-party package ecosystem.

System Architecture and Purpose

Acc-Py emerged from the need to balance Python’s flexibility with the stability requirements of operational control-room environments. The traditional approach of centrally-managed, fully-featured software distributions becomes problematic when dealing with diverse third-party Python packages, as all applications would require re-validation at each iteration, and individual applications with upgrade issues could block updates for the entire community.

The platform operates on the principle of empowering users through virtual environments and pip package management, allowing applications to manage upgrades independently within reasonable constraints. This approach derives from Python’s inherently permissive philosophy while maintaining operational stability through well-placed safety mechanisms.

Target Platform and Compatibility

The target platform for Acc-Py services encompasses x86-64 Linux systems, specifically RHEL9 and binary-compatible variants. Container images are based on Alma Linux 9 for licensing purposes. The system currently supports the manylinux_2_28_x86_64 standard, covering operational consoles, servers, and virtual machines. Other platforms such as Windows or macOS may function with certain services but receive no active testing or support. Python usage on Front-End Computers (FECs) is discouraged except for boot sequences, and Acc-Py provides no support for such environments.

Base Distribution System

The Acc-Py base distribution functions as the fundamental Python distribution for all accelerator sector activities. This distribution differs from system Python installations by providing several key advantages. The base distribution enables updating Python versions independently of the operating system, supports multiple Python versions side-by-side on the same machine, and allows installation to any directory by users with write access.

The base distribution is accessible on Technical Network trusted virtual machines through a central NFS installation. Users activate the distribution by sourcing the setup script:

source /acc/local/share/python/acc-py/base/pro/setup.sh

Other versions of the distribution are available and documented in the Python distribution documentation. The system logs each invocation of Acc-Py Python for community service improvements, with full details available in the usage logging documentation.

The distribution maintains a lightweight approach, containing only essential components: the Python interpreter, numpy, scipy, and Acc-Py pip configuration. Unlike predecessors, the base distribution excludes compiler toolchains and extensive pre-installed libraries. This design requires package authors to avoid relying on user compiler environments and to declare dependencies correctly, as the smaller base distribution reduces the likelihood of accidentally having dependencies already installed.

Virtual environments are created using the acc-py tool available after enabling the distribution:

acc-py venv /path/to/my/venv

Users can activate virtual environments either by running the executable directly (/path/to/my/venv/bin/python) or through activation:

source /path/to/my/venv/bin/activate

Package installation follows standard pip practices, with the recommendation to use python -m pip rather than pip directly:

python -m pip install pyda

This approach prevents confusion where the pip executable might not originate from the same directory as the python executable, particularly in PyCharm environments. Further details about pip usage are available in the Using pip documentation.

Version Management Strategy

Acc-Py maintains multiple concurrent versions to facilitate iterative adoption of newer Python versions. The current release schedule includes version 2020.11 with Python 3.7, version 2021.12 with Python 3.9, and version 2023.06 with Python 3.11. A version 2024.XX with Python 3.13 is planned. Each version remains suitable for operations until the end of Run 3.

The installation includes a “pro” symbolic link pointing to the recommended version for operational use. The existence of “pro” does not diminish the operational grade of other available versions. The “pro” version evolves over time, but released applications and virtual environments continue using their original version until explicitly upgraded.

Virtual environments are not transferable between Python versions, requiring recreation from scratch during upgrades. The upgrade process for existing virtual environments involves:

# Activate the new distribution
source /acc/local/share/python/acc-py/base/2021.12/setup.sh
 
# Remove the current virtual environment
rm -rf /path/to/venv
 
# Re-create the virtual environment at the same location
acc-py venv /path/to/venv
 
# Activate the environment
source /path/to/venv/bin/activate
 
# Install project and dependencies into the new virtual environment
pip install -e /path/to/project

For applications using acc-py-deploy, the upgrade process involves:

# Activate the new distribution
source /acc/local/share/python/acc-py/base/2021.12/setup.sh
 
# Navigate to the project
cd /path/to/project/
 
# Lock application with the new distribution
acc-py app lock . --base-python=2021.12
 
# Deploy the application
acc-py app deploy . --base-python=2021.12

Detailed version information and release notes are available in the Acc-Py base releases documentation.

Container Image Infrastructure

The container image system provides OCI-compatible images for use with GitLab CI, Podman, Docker, and Kubernetes. Images are built on containerisation project base images and follow a structured naming convention:

registry.cern.ch/acc/acc-py_{operating_system}_{java}_{variant}:{version}

The system offers multiple image variants addressing different use cases. Base images (acc-py_<BASE>) include only the necessary Python distribution. CI images (acc-py_<BASE>_ci) add the acc-py command, development tools, and packages useful for continuous integration such as git. App-Exec images (acc-py_<BASE>_app_exec) contain materials for building container images that embed Python applications.

GUI variants include libraries for running graphical applications. GUI images (acc-py_<BASE>_gui) provide GUI support, while GUI CI images (acc_py_<BASE>_gui_ci) combine GUI capabilities with tools for running and testing headless applications in CI environments. GUI App-Exec images (acc-py_<BASE>_gui_app_exec) support building images that embed Python GUI applications.

Example usage with Podman on managed machines:

podman run \
    --rm \
    -ti \
    registry.cern.ch/acc/acc-py_el9:pro \
    python

Building custom images involves creating a Containerfile:

FROM registry.cern.ch/acc/acc-py_el9_ci:2023.06
 
RUN pip install \
        --no-cache \
        pyjapc \
    && python -m cmmnbuild_dep_manager resolve
 
CMD python

This can be built with:

podman build \
    --file=/path/to/Containerfile \
    /path/to/context

GUI applications require X11 socket forwarding for local display:

# Grant permissions (required after each desktop login)
xhost +local:
 
# Run GUI application with X11 forwarding
docker run --rm -ti \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -v /path/to/my/app:/path/to/my/app \
    registry.cern.ch/acc/acc-py_el9_gui:2023.06 \
    python

GUI CI images support headless testing through the run_headless script:

run_headless 'my command here'

Troubleshooting headless applications can be accomplished by taking screenshots:

# Install screenshot tools
yum install -y ImageMagick
 
# Take screenshot (adjust display number as needed)
import \
    -display :99 \
    -window root \
    /path/to/image.png

Images provide the Acc-Py base distribution installed at /opt/acc-py/base/$VERSION/ with a convenience symlink at /opt/acc-py/base/installed/. The distribution is enabled by default for interactive usage, and a pre-created virtual environment is available at /opt/acc-py/venvs/acc-py-venv/. Additional details are available in the Container Images documentation.

Interactive Environment

The interactive environment provides a pre-configured Python environment containing commonly used packages for ad-hoc analysis and prototyping. This environment is activated through:

source /acc/local/share/python/acc-py/pro/setup.sh

The interactive environment explicitly does not provide stability guarantees. Contents can and do change during operational runs, often coinciding with technical stops and system administration days. The environment serves quick access needs but is unsuitable for regular or programmatic operational use. Applications requiring stable package versions must use virtual environments from the Acc-Py base distribution. Changes to the interactive environment are documented in the acc-py-rich-environment documentation.

Package Repository System

Acc-Py maintains a Python Package Index at acc-py-repo.cern.ch, providing a repository for package installation and release. The system recommends releasing Python packages in both source distribution and manylinux-compatible wheel formats. This approach provides the advantages of wheel format (faster installation, increased security and predictability) in operational contexts while maintaining flexibility through source distributions for all other scenarios.

Package releases occur via continuous integration using Acc-Py GitLab CI templates that provide automated wheel building upon release through git tags. For packages containing native code, wheels must meet the manylinux2014 standard at minimum and support all currently operational Acc-Py base versions.

Control System Libraries

The platform includes several specialized libraries for accelerator control operations. These libraries are maintained by BE-CSS and provide essential functionality for accelerator operations.

Core Device and Data Access Libraries

PyDA provides device access and control functionality with comprehensive documentation available. The library enables direct parameter access and device control within the accelerator complex. The source repository is maintained at https://gitlab.cern.ch/acc-co/devops/python/prototypes/pyda.

PyTimber enables access to CERN’s logging database for historical data analysis. Originally developed by BE-ABP and now maintained by BE-CSS, the library provides comprehensive access to logged accelerator data. Documentation is available on Confluence, with the source repository at https://gitlab.cern.ch/scripting-tools/pytimber.

PyJAPC offers Java API for Parameter Control integration, facilitating communication with Java-based control systems. Originally developed by EN-EA and now maintained by BE-CSS, the library provides seamless integration between Python and Java control environments. Documentation and source code are available at https://gitlab.cern.ch/scripting-tools/pyjapc.

Configuration and Security Libraries

PyCCDA provides access to controls configuration databases through the Controls Configuration Data API. The library enables configuration management for accelerator control systems. Documentation is available, with source code maintained at https://gitlab.cern.ch/controls-configuration-service/controls-configuration-data-api/accsoft-ccs-ccda/-/tree/dev/accsoft-ccs-pyccda.

PyRBAC implements role-based access control for authentication and authorization in control systems. The library provides essential security functionality for operational environments. Documentation and source code are available at https://gitlab.cern.ch/acc-co/cmw/cmw-core-cpp/-/tree/develop/cmw-rbac-py/.

Additional Control System Tools

pyLogbook enables electronic logbook integration, originally developed by SY-BI and now maintained by BE-CSS. The library provides programmatic access to CERN’s electronic logbook system. Documentation and source code are available at https://gitlab.cern.ch/scripting-tools/pylogbook.

PyRDA3 provides remote device access for non-operational use during development and testing. The library enables safe device interaction outside operational contexts. Documentation and source code are maintained at https://gitlab.cern.ch/acc-co/cmw/cmw-core-cpp/-/tree/develop/cmw-rda3-py.

PjLSA provides LSA (LHC Software Architecture) integration capabilities. The source repository is maintained at https://gitlab.cern.ch/scripting-tools/pjlsa by BE-CSS.

CMMNBUILD-DEP-MANAGER handles dependency management for CMW applications. The package is available through the Acc-Py repository, with source code at https://gitlab.cern.ch/scripting-tools/cmmnbuild-dep-manager.

Example usage of control system libraries:

import pyda
import pytimber
import pyjapc
 
# Device access through PyDA
device = pyda.Device("LHC.BOFSU:BEAM_ENERGY")
current_energy = device.read()
device.write("SETPOINT", 6500.0)
 
# Historical data access through PyTimber
ldb = pytimber.LoggingDB()
data = ldb.get("LHC.BOFSU:BEAM_ENERGY", 
               "2024-01-01 00:00:00", 
               "2024-01-01 23:59:59")
 
# Java control system integration through PyJAPC
japc = pyjapc.PyJapc()
japc.getParam("LHC.BOFSU:BEAM_ENERGY")

The Controls libraries documentation provides comprehensive information about all available libraries.

Development Tools and Templates

GitLab CI templates provide zero-configuration solutions for common Python development workflows. The templates aim to address build, test, and release stages of development pipelines. Users can generate default CI file definitions using the acc-py command line tool through the acc-py init-ci command.

The template system includes jobs for multi-version Python testing, code quality checks, documentation building and deployment, binary wheel building for multiple platforms, and automated releases to package repositories. Templates are customizable and their source code is available on GitLab for expert modifications.

Example GitLab CI configuration using templates:

include:
  - project: 'acc-co/devops/python/acc-py-gitlab-ci-templates'
    ref: 'v2'
    file: 'python-package.yml'
 
variables:
  PYTHON_VERSIONS: "3.9,3.11"
  PYPI_REPOSITORY: "acc-py-repo"

Complete template documentation is available in the full documentation.

Documentation Service

The documentation service enables automatic publishing of project documentation built as part of GitLab CI pipelines. The service integrates with standard documentation building tools and provides version-specific hosting under the acc-py.web.cern.ch domain. Projects can configure automatic documentation deployment through GitLab CI integration.

Example documentation CI configuration:

include:
  - project: 'acc-co/devops/python/acc-py-gitlab-ci-templates'
    ref: 'v2'
    file: 'build-docs.yml'

Graphical User Interface Development

PyQt6 serves as the standard choice for GUI development within the Acc-Py ecosystem. All new GUI projects use PyQt6 with Python 3.11 or later (Acc-Py Base 2023.06 or later versions). Additional tooling includes PyQt6-stubs for mypy type checking validation and PyQt6-designer for Qt Designer compatibility.

Existing codebases using PyQt5 can migrate using the provided PyQt6 migration guide. Online training materials are available in beta form to support PyQt development for accelerator controls applications.

Example PyQt6 application:

from PyQt6.QtWidgets import QApplication, QMainWindow, QVBoxLayout, QWidget, QPushButton
from PyQt6.QtCore import pyqtSignal
import pyda
 
class AcceleratorControlWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.device = pyda.Device("LHC.BOFSU:BEAM_ENERGY")
        self.init_ui()
    
    def init_ui(self):
        central_widget = QWidget()
        layout = QVBoxLayout(central_widget)
        
        read_button = QPushButton("Read Energy")
        read_button.clicked.connect(self.read_energy)
        layout.addWidget(read_button)
        
        self.setCentralWidget(central_widget)
        self.setWindowTitle("Accelerator Control")
    
    def read_energy(self):
        energy = self.device.read()
        print(f"Current beam energy: {energy} GeV")
 
app = QApplication([])
window = AcceleratorControlWindow()
window.show()
app.exec()

Type checking for PyQt6 applications requires proper configuration:

python -m pip install pyqt6-stubs mypy
mypy --strict my_gui_application.py

Additional GUI development information is available in the GUI documentation, including GUI testing and troubleshooting guides.

Project Structure and Packaging

The development guidelines strongly recommend that all applications, services, and libraries be deployed as Python packages rather than collections of scripts. The acc-py command line interface provides project initialization through:

acc-py init name-of-package

This command creates a directory structure with setup.py metadata file and an importable package directory. The setup.py file requires editing to include proper metadata and declare all third-party dependencies, even those shipped as part of the Acc-Py distribution.

Example generated project structure:

name-of-package/
├── setup.py
├── name_of_package/
│   ├── __init__.py
│   └── __main__.py
└── tests/

Package structure conversion from scripts involves moving all files into the package directory and converting absolute imports to relative imports. For example, import script_b becomes from . import script_b and from script_b import thing becomes from .script_b import thing.

Projects typically include a __main__.py file for application entry points:

if __name__ == '__main__':
    import script_a
    script_a.main()

This enables execution through python -m name_of_package. Installation for development testing follows standard practices:

# Create virtual environment
acc-py venv ./env
source ./env/bin/activate
 
# Install in editable mode
pip install --editable ./name-of-package

Package data inclusion requires configuration in setup.py:

setup(
    name="name_of_package",
    packages=find_packages(),
    package_data={'': '*.png'},
    install_requires=[
        "numpy>=1.20.0",
        "pyda>=2.0.0",
        "pyqt6>=6.4.0",
    ],
)

Comprehensive guidance is available in the creating a Python package documentation and best practices guidelines.

Virtual Environment Management

Virtual environments represent the supported mechanism for extending the Acc-Py base distribution. The base distribution enables the acc-py command for convenience:

acc-py venv /path/to/venv

Alternatively, direct calls to python -m venv --system-site-packages /path/to/venv work when the acc-py tool is unavailable. Virtual environments provide dependency isolation, reproducible environments across development and production, and safe experimentation with package versions.

Standard virtual environment workflow:

# Create environment
acc-py venv ./project-env
 
# Activate environment
source ./project-env/bin/activate
 
# Install dependencies
python -m pip install -r requirements.txt
 
# Install project in development mode
python -m pip install --editable .
 
# Work with project
python -m my_project
 
# Deactivate when finished
deactivate

The documentation strongly recommends avoiding PYTHONPATH modifications or user space installations (.local directory) due to fragility, particularly in operational contexts where these approaches are not permitted.

Best Practices and Guidelines

The documentation emphasizes several development practices. PEP8 provides the foundation for Python coding style, with Acc-Py guidelines extending rather than contradicting PEP8. Project layout should follow established patterns with proper testing, documentation, and release procedures.

Type hints are encouraged for better code quality and maintainability. Testing should be comprehensive using pytest or similar frameworks. Documentation should clearly describe all public APIs and usage patterns. Code review processes help maintain quality standards across projects.

Example type-hinted code following best practices:

from __future__ import annotations
 
from typing import Dict, List, Optional
import numpy as np
import pyda
 
class BeamEnergyMonitor:
    """Monitor beam energy for LHC operations."""
    
    def __init__(self, device_name: str) -> None:
        self.device = pyda.Device(device_name)
        self._energy_history: List[float] = []
    
    def read_current_energy(self) -> float:
        """Read current beam energy in GeV."""
        energy = self.device.read()
        self._energy_history.append(energy)
        return energy
    
    def get_energy_statistics(self) -> Dict[str, float]:
        """Calculate energy statistics from history."""
        if not self._energy_history:
            return {}
        
        history_array = np.array(self._energy_history)
        return {
            'mean': float(np.mean(history_array)),
            'std': float(np.std(history_array)),
            'min': float(np.min(history_array)),
            'max': float(np.max(history_array)),
        }

Additional best practices are documented in the development advice, code style, and testing guidelines.

Community and Support Structure

The Acc-Py community operates through multiple communication channels. The primary mailing list serves general community discussions, with subscription available through CERN e-groups. Technical support is available through [email protected]. The Mattermost Python Talks channel provides real-time communication and discussions.

Documentation feedback and suggestions can be submitted to [email protected] for integration into the official documentation. The community maintains example projects, training materials, and template repositories to support development activities.

The community follows a code of conduct to ensure inclusive and productive collaboration. Community information and guidelines are available in the community documentation.

Migration and Compatibility

Migration between Acc-Py versions requires careful planning and execution. Development environments require virtual environment recreation due to Python version incompatibilities. Production applications using acc-py-deploy tools follow specific upgrade procedures involving version locking and deployment with new base Python versions.

Container image migration involves updating base image references in Dockerfile configurations and rebuilding application dependencies:

# Update base image version
FROM registry.cern.ch/acc/acc-py_el9_ci:2023.06
 
# Rebuild dependencies with new base
RUN python -m pip install --no-cache-dir -r requirements.txt

The platform provides migration guides and compatibility matrices to support smooth transitions between versions. Specific migration procedures are documented for different scenarios, including PyQt5 to PyQt6 migration and setup.py to pyproject.toml migration.

Usage Logging and Telemetry

The system implements usage logging for each invocation of Acc-Py Python to better serve the community. This logging provides insights into usage patterns and helps guide future development decisions. Full details of the logging implementation are documented in the usage logging documentation.

Operational Deployment Strategies

Applications can be deployed using multiple strategies depending on operational requirements. Container-based deployment uses app-exec images with embedded applications. Traditional deployment employs acc-py-deploy tools for systematic application management. Configuration management and service integration support various operational contexts.

Example application deployment using containers:

FROM registry.cern.ch/acc/acc-py_el9_app_exec:2023.06
 
# Copy requirements and install dependencies
COPY requirements.txt /app/
RUN python -m pip install --no-cache-dir -r /app/requirements.txt
 
# Copy application code
COPY . /app/
RUN python -m pip install --no-deps /app/
 
# Set runtime configuration
USER appuser
WORKDIR /app
 
CMD ["python", "-m", "my_accelerator_app"]

The deployment walk-through provides comprehensive guidance for operational deployment scenarios.

Monitoring and maintenance procedures include scheduled update windows, rollback capabilities for problematic updates, and testing protocols for production deployments. The platform balances operational stability requirements with the need for security updates and feature improvements. Additional deployment guidance is available in the deployment documentation.