This course focuses on the machinery surrounding a scientific analysis: configuration, data, provenance, workflow execution, and computational environment. Participants develop workflows that make analysis steps explicit and repeatable, then extend them from local execution toward portable, automated pipelines that can run across different machines and omputing environments.

Course Outline

The course meets weekly. Each session lasts 3.5 hours, for a total of 14 hours of scheduled course time.

Session 1: Configurable Analyses

Separate analysis code from parameters using command-line interfaces and configuration files. Explore how notebook-based analyses can become repeatable workflow steps and how automated notebook execution can support reproducible scientific work.

Session 2: Data and Provenance Management

Organize scientific data in versioned datasets that can be copied, shared, and connected to online repositories. Run analysis commands while recording their provenance and examine ways to check whether a workflow produces the expected outputs.

Session 3: Automated Scientific Workflows

Represent an analysis as an explicit workflow with defined inputs, outputs, and task dependencies. Use a workflow manager such as Snakemake to move from a fixed pipeline toward reusable patterns that can process different files, parameters, or datasets.

Session 4: Portable Computing Environments

Examine how remote data hosting with datalad, Linux environments, virtual machines, and containers support execution across operating systems and computing infrastructure. Consider how these approaches help move workflows from local machines toward shared or high-performance computing systems.

Learning and participation

Topics

  • Configurable Scientific Analyses: command-line interfaces, configuration files, and notebook-based workflows.
  • Data and Provenance Management: versioned datasets, related dataset copies, provenance tracking, and online repositories.
  • Automated Scientific Workflows: explicit task dependencies, reusable workflow patterns, and generalized analysis pipelines.
  • Portable Computing Environments: Linux environments, virtualization, containers, and computing infrastructure.

Intended participants

  • Researchers and students from all universities are welcome.
  • Comfort writing small Python scripts.
  • Comfort using the command line.
  • Comfort using Git for version control.

Certification

Participants who attend at least 75% of the course will receive a participation certificate by email at the end of the course.