Intro to File- and Database-oriented Neuroscience Data Management With Python, SQL, and HDF5

–
Event completed

What is all this hype about databases, and how can I use them to make my analysis work simpler? In this workshop, you’ll build databases and query them in Python and SQL using DuckDB, store complex data in HDF5, and host it with Git and GitHub.

Workshop

Neuroscience is evolving rapidly, with experimental data becoming increasingly complex. How can you seamlessly integrate vast and diverse datasets for insightful analysis and easy sharing? And how would your process improve if, instead of having to write long scripts, you could analyze data with just a few lines of code?

In this workshop, discover the power of database management systems, a game-changer in neuroscience research. We will dive into the world of SQL and learn about DuckDB SQL engine, which makes it easy to apply industry-standard data organization methods to research data as a relational database – no server management needed! You’ll also gain hands-on experience with HDF5 and JSON for key-value data storage and learn how to combine various management techniques for optimal convenience and performance by building hybrid database systems.

By the course’s end, you’ll be adept at writing Python scripts to create and extract data from databases, query large databases in SQL, store complex data in HDF5, manage your work with Git, and publish your projects on GitHub.

Learning and participation

Topics

  • Navigating Local and Remote Filesystems with Pathlib and FsSpec
  • Parsing and Extracting Metadata from Filenames and JSON Files
  • Querying JSON, CSV, and Parquet files with SQL in DuckDB

Intended participants

  • Researchers and students from all universities are welcome.
  • Participants of all skill levels and backgrounds are welcome.

Useful resources for preparation

Certification

Students who attend at least 75% of the course will receive a participation certificate by email at the end of the course.