{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"toc": true
},
"source": [
"Table of Contents
\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 05 - Using conda to manage C++ libraries\n",
"\n",
"The [cffi_funs](https://github.com/phaustin/parallel_python_course/tree/master/code_examples/cffi_funs) folder shows how to set up a C++ file so that it compiled with [cmake](http://preshing.com/20170511/how-to-build-a-cmake-based-project/) and also [here](http://preshing.com/20170522/learn-cmakes-scripting-language-in-15-minutes/)\n",
"\n",
"The folder contains [these files](https://github.com/phaustin/parallel_python_course/blob/master/code_examples/cffi_funs/Readme_cffi_funs.rst)\n",
"\n",
"**Demo**\n",
"\n",
"Show how to build a conda archive with:\n",
"\n",
" conda build .\n",
" \n",
"and then upload it to your [Anaconda channel](https://anaconda.org/phaustin/dashboard)\n",
"\n",
"Where it can be installed into a conda environment with:\n",
"\n",
" conda install -c phaustin cffi_funs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using conda to manage python libraries\n",
"\n",
"The [cffi_practice](https://github.com/phaustin/parallel_python_course/tree/master/code_examples/cffi_practice)\n",
"folder shows how to install a simple python project using conda and [setuptools](http://setuptools.readthedocs.io/en/latest/setuptools.html)\n",
"\n",
"**Demo**\n",
"\n",
"Show how to build and upload cffi_practice to my conda channel\n",
"\n",
"The purpose of this module is to provide one function:\n",
"\n",
" cffi_practice.get_paths()\n",
" \n",
"That can be used to locate the library and header from cff_funs\n",
"\n",
"The get_paths function is defined in this package in the [__init__.py](https://github.com/phaustin/parallel_python_course/blob/master/code_examples/cffi_practice/cffi_practice/__init__.py) module.\n",
"\n",
"Try:\n",
"\n",
" conda install cff_practice -c phaustin\n",
" \n",
"Then:\n",
"\n",
" python -c 'import cffi_practice;print(cffi_practice.get_paths())'\n",
" \n",
"Which should output something like:\n",
"\n",
" {'libfile': '/Users/phil/mini36/lib/libcffi_funs.so', 'libdir': '/Users/phil/mini36/lib', 'includedir': '/Users/phil/mini36/include'}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Accessing the functions from python using CFFI\n",
"\n",
"The [C foreign function interface](https://cffi.readthedocs.io/en/latest/overview.html) provides a way to call the cffi_funs from python\n",
"\n",
"Here is an example that exposes the get_thread_id and get_proces_id functions from the cffi_fun package"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"found these functions in module: ['get_process_id', 'get_thread_id']\n",
"Here is the thread id in hex: -4fdf76a5-\n",
"here is the process ide in base 10: -16502-\n"
]
}
],
"source": [
"from cffi import FFI\n",
"from joblib import Parallel\n",
"from cffi_practice import get_paths\n",
"#\n",
"# locate the library\n",
"#\n",
"lib_so_file=get_paths()['libfile']\n",
"ffi=FFI()\n",
"ffi.cdef(\"\"\"\n",
" void get_thread_id(char *thread_id);\n",
" void get_process_id(char *process_id);\n",
"\"\"\")\n",
"#\n",
"# open the library file\n",
"#\n",
"lib = ffi.dlopen(lib_so_file)\n",
"print('found these functions in module: ',dir(lib))\n",
"#\n",
"# create a 25 character C array to hold the ouput\n",
"#\n",
"arg_thread = ffi.new(\"char[]\",25) #(C++)\n",
"#\n",
"# copy the bytes into arg_thread (C++)\n",
"#\n",
"lib.get_thread_id(arg_thread) \n",
"#\n",
"# get the bytes into a python byte object\n",
"#\n",
"out_thread=ffi.string(arg_thread) #C++ to python\n",
"#\n",
"# turn the bytes into a utf-8 string\n",
"#\n",
"str_out=out_thread.decode('utf-8') #python\n",
"#\n",
"# print it out\n",
"#\n",
"print(f\"Here is the thread id in hex: -{str_out}-\")\n",
"#\n",
"# repeat for the process\n",
"#\n",
"arg_process = ffi.new(\"char[]\",25) \n",
"lib.get_process_id(arg_process)\n",
"out_process=ffi.string(arg_process)\n",
"str_out=out_process.decode('utf-8')\n",
"print(f\"here is the process ide in base 10: -{str_out}-\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Running this in a threadpool\n",
"\n",
"The following script uses joblib to create 10 jobs to call the cffi functions in parallel, returning\n",
"the pointers to the character array and convertingthem to python strings."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"here are the pointers to hold the ids: [[], [], [], [], [], [], [], [], [], []]\n",
"\n",
"here are the thread ids\n",
"thread id: 1bb9329b\n",
"thread id: 72b7e656\n",
"thread id: 7f162e70\n",
"thread id: c9742b94\n",
"thread id: 9e63eefb\n",
"thread id: 6b8afbdc\n",
"thread id: c3945c37\n",
"thread id: 425f89f8\n",
"thread id: e612e1c6\n",
"thread id: f9e7dfdc\n"
]
}
],
"source": [
"nprocs=10\n",
"arg_list=[]\n",
"fun_list=[]\n",
"dict_list=[]\n",
"for i in range(nprocs):\n",
" fun_list.append(lib.get_thread_id)\n",
" result_var=ffi.new(\"char[]\",25)\n",
" arg_list.append(result_var)\n",
" dict_list.append({})\n",
"ptr_list=[[ffi.cast(\"char*\",item)] for item in arg_list]\n",
"jobs=list(zip(fun_list,ptr_list,dict_list))\n",
"print(f'here are the pointers to hold the ids: {ptr_list}\\n')\n",
"with Parallel(n_jobs=nprocs,backend='threading') as parallel:\n",
" parallel(jobs)\n",
"print('here are the thread ids')\n",
"for item in ptr_list:\n",
" out_thread=ffi.string(item[0]).decode('utf-8')\n",
" print('thread id: ',out_thread)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
},
"toc": {
"nav_menu": {},
"number_sections": true,
"sideBar": false,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {},
"toc_section_display": "block",
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}