Home 2019Jan01
  • Site
      • 00 - Introduction
        • First steps
          • Installing python
          • Installing extra dependencies
        • Course objectives
        • Motivation (editorial)
          • Concurrency vs. parallelism
          • Threads and processes
            • Threads and processes in Python
            • Python global intepreter lock
          • Thread scheduling
            • Note that these three threads are taking turns, resulting in a computation that runs slightly slower (because of overhead) than running on a single thread
          • Releasing the GIL
      • 01 - Introduction to joblib
        • Creating a process or thread pool with joblib
          • Running a threadsafe function
          • Setup logging so we can know what process and thread we are running
          • Create two functions, one to print thread and process ids, and one to run the wait_for loop
          • Now repeat this holding the GIL
          • Now repeat with processes instead of threads
          • Summary
      • 02: xarray, netcdf and zarr
        • The current defacto standard in atmos/ocean science
        • Some challenges with netcdf
        • create an xarray
        • Download toy model data
        • Sort in numeric order
        • Make an xarray
        • Dump to a zarr file
      • 03 - Using dask and zarr for multithreaded input/output
        • zarr
        • dask
          • Example, write and read zarr arrays using multiple threads
          • Create 230 Mbytes of fake data
          • Copy to a zarr file on disk, using multiple threads
          • Add some attributes
          • Create an array of zeros – note that compression shrinks it from 230 Mbytes to 321 bytes
          • copy input to output using chunks
          • Create a dask array from a zarr disk file
          • The following calculation uses numpy, so it releases the GIL
          • Note that result hasn’t been computed yet
          • Now do the calculation
          • You can evaluate your own functions on dask arrays
      • 04 - Using numba to release the GIL
        • Timing python code
          • Define a function that does a lot of computation
          • now time it with pure python
        • Now try this with numba
        • Make two identical functions: one that releases and one that holds the GIL
        • now time wait_loop_withgil
        • not bad, but we’re only using one core
      • 05 - Using conda to manage C++ libraries
      • Using conda to manage python libraries
        • Accessing the functions from python using CFFI
          • Running this in a threadpool
      • 06 - Using conda to manage C++ libraries with pybind11
  • Page
      • 00 - Introduction
      • 01 - Introduction to joblib
      • 02: xarray, netcdf and zarr
      • 03 - Using dask and zarr for multithreaded input/output
      • 04 - Using numba to release the GIL
      • 05 - Using conda to manage C++ libraries
      • Using conda to manage python libraries
      • 06 - Using conda to manage C++ libraries with pybind11
  • 00 - Introduction »
  • Source
  • 00 - Introduction
  • 01 - Introduction to joblib
  • 02: xarray, netcdf and zarr
  • 03 - Using dask and zarr for multithreaded input/output
  • 04 - Using numba to release the GIL
  • 05 - Using conda to manage C++ libraries
  • Using conda to manage python libraries
  • 06 - Using conda to manage C++ libraries with pybind11

Parallel Python¶

(Note – this material is under construction and might change significantly between now and June 14)

  • 00 - Introduction
    • First steps
    • Course objectives
    • Motivation (editorial)
  • 01 - Introduction to joblib
    • Creating a process or thread pool with joblib
  • 02: xarray, netcdf and zarr
    • The current defacto standard in atmos/ocean science
    • Some challenges with netcdf
    • create an xarray
    • Download toy model data
    • Sort in numeric order
    • Make an xarray
    • Dump to a zarr file
  • 03 - Using dask and zarr for multithreaded input/output
    • zarr
    • dask
  • 04 - Using numba to release the GIL
    • Timing python code
    • Now try this with numba
    • Make two identical functions: one that releases and one that holds the GIL
    • now time wait_loop_withgil
    • not bad, but we’re only using one core
  • 05 - Using conda to manage C++ libraries
  • Using conda to manage python libraries
    • Accessing the functions from python using CFFI
  • 06 - Using conda to manage C++ libraries with pybind11

Back to top

© Copyright 2019, Philip Austin.
Created using Sphinx 1.8.3.