Table of Contents

  • 1  05 - Using conda to manage C++ libraries

  • 2  Using conda to manage python libraries

    • 2.1  Accessing the functions from python using CFFI

      • 2.1.1  Running this in a threadpool

05 - Using conda to manage C++ libraries

The cffi_funs folder shows how to set up a C++ file so that it compiled with cmake and also here

The folder contains these files

Demo

Show how to build a conda archive with:

conda build .

and then upload it to your Anaconda channel

Where it can be installed into a conda environment with:

conda install -c phaustin cffi_funs

Using conda to manage python libraries

The cffi_practice folder shows how to install a simple python project using conda and setuptools

Demo

Show how to build and upload cffi_practice to my conda channel

The purpose of this module is to provide one function:

cffi_practice.get_paths()

That can be used to locate the library and header from cff_funs

The get_paths function is defined in this package in the init.py module.

Try:

conda install cff_practice -c phaustin

Then:

python -c 'import cffi_practice;print(cffi_practice.get_paths())'

Which should output something like:

{'libfile': '/Users/phil/mini36/lib/libcffi_funs.so', 'libdir': '/Users/phil/mini36/lib', 'includedir': '/Users/phil/mini36/include'}

Accessing the functions from python using CFFI

The C foreign function interface provides a way to call the cffi_funs from python

Here is an example that exposes the get_thread_id and get_proces_id functions from the cffi_fun package

[1]:

from cffi import FFI
from joblib import Parallel
from cffi_practice import get_paths
#
#  locate the library
#
lib_so_file=get_paths()['libfile']
ffi=FFI()
ffi.cdef("""
    void get_thread_id(char *thread_id);
    void get_process_id(char *process_id);
""")
#
#  open the library file
#
lib = ffi.dlopen(lib_so_file)
print('found these functions in module: ',dir(lib))
#
# create a 25 character C array to hold the ouput
#
arg_thread = ffi.new("char[]",25)  #(C++)
#
# copy the bytes into arg_thread  (C++)
#
lib.get_thread_id(arg_thread)
#
# get the bytes into a python byte object
#
out_thread=ffi.string(arg_thread)  #C++ to python
#
# turn the bytes into a utf-8 string
#
str_out=out_thread.decode('utf-8')  #python
#
# print it out
#
print(f"Here is the thread id in hex: -{str_out}-")
#
# repeat for the process
#
arg_process = ffi.new("char[]",25)
lib.get_process_id(arg_process)
out_process=ffi.string(arg_process)
str_out=out_process.decode('utf-8')
print(f"here is the process ide in base 10: -{str_out}-")

found these functions in module:  ['get_process_id', 'get_thread_id']
Here is the thread id in hex: -4fdf76a5-
here is the process ide in base 10: -16502-

Running this in a threadpool

The following script uses joblib to create 10 jobs to call the cffi functions in parallel, returning the pointers to the character array and convertingthem to python strings.

[2]:

nprocs=10
arg_list=[]
fun_list=[]
dict_list=[]
for i in range(nprocs):
    fun_list.append(lib.get_thread_id)
    result_var=ffi.new("char[]",25)
    arg_list.append(result_var)
    dict_list.append({})
ptr_list=[[ffi.cast("char*",item)] for item in arg_list]
jobs=list(zip(fun_list,ptr_list,dict_list))
print(f'here are the pointers to hold the ids: {ptr_list}\n')
with Parallel(n_jobs=nprocs,backend='threading') as parallel:
    parallel(jobs)
print('here are the thread ids')
for item in ptr_list:
    out_thread=ffi.string(item[0]).decode('utf-8')
    print('thread id: ',out_thread)

here are the pointers to hold the ids: [[<cdata 'char *' 0x7faa09f64970>], [<cdata 'char *' 0x7faa09f01f20>], [<cdata 'char *' 0x7faa09f2fb90>], [<cdata 'char *' 0x7faa09fbd840>], [<cdata 'char *' 0x7faa09f7aee0>], [<cdata 'char *' 0x7faa09f08c40>], [<cdata 'char *' 0x7faa09f08c90>], [<cdata 'char *' 0x7faa09f7b6a0>], [<cdata 'char *' 0x7faa09f7b6f0>], [<cdata 'char *' 0x7faa09f0ef90>]]

here are the thread ids
thread id:  1bb9329b
thread id:  72b7e656
thread id:  7f162e70
thread id:  c9742b94
thread id:  9e63eefb
thread id:  6b8afbdc
thread id:  c3945c37
thread id:  425f89f8
thread id:  e612e1c6
thread id:  f9e7dfdc

[ ]: