C/C++ Extension for Python
Recently I need to write a very fast and flexible crypto lib. And C++ is one of the richest languages with many crypto modules and libs, and also one of the fastest languages. But the problem was that I needed that lib with different languages, mainly Python, but writing the same code with other languages was not the idea I expected for. So the best solution was just called C++ code from Python. And here I want to share some little techniques and “challenges” I met on my way of doing that, as there are many cases when C++’s speed and riches are indispensable and unquestionable.
Let's start from very beginning.
For calling C++ from Python we first need to write some wrapper code which can “translate” C++ into Python.
Let’s have a classic and a bit bored function hello in hello.c file.
char * hello(){ return "Hello!";}Let’s write a wrapper (Python callable)function for hello function in a wrapper.cpp file.
#include "Python.h"
#include <stdio.h>
#include "hello.c"static PyObject * hello_wrapper(PyObject * self, PyObject * args)
{
char * input; // param for hello
char * result; // return value from hello
PyObject * ret; // python object which should be returned from
// wrapper function
// parse arguments
if (!PyArg_ParseTuple(args, "s", &input)) {
return NULL;
}
// run the actual function
result = hello(input);
// build the resulting string into a Python object.
ret = PyBytes_FromString(result);
free(result);
return ret;
}
Note that in our first line we include header file called “Python.h”. To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros, and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header file “Python.h”.
After writing a wrapper function we have to register this function within a module’s symbol table (all Python functions live in a module, even if they’re actually C functions!).
static PyMethodDef HelloMethods[] = {
{ "hello", hello_wrapper, METH_VARARGS},
{ NULL, NULL, 0, NULL}
};And we need to write a module definition and init function for the module.
static struct PyModuleDef hello_definition = {
PyModuleDef_HEAD_INIT,
"hello",
"A Python module extension for C++ lib",
-1,
HelloMethods
};PyMODINIT_FUNC PyInit_hello(void) {
Py_Initialize();
return PyModule_Create(&hello_definition);
}
This was a basic example, where C++ code was as short as we could keep it with our Python package. Now let’s look a case, where we have a very long C++ code that it is not rational to drag with our Python package.
Let’s imagine we have C++ code and all function which should be called from Python are defined in a call.h header file.
// include all header files void return_nothing();
void* return_by_value(void *ptr);
void return_by_reference(void* ptr, char** buffer, int* number);
void* return_by_value_and_by_reference(void* ptr, char** buffer, int* number);
Now, let’s write wrapper functions in a wrapper.cpp file.
#include "Python.h"
#include <stdio.h>
#include "call.h"
#include <string>static PyObject* return_nothing_wrapper(PyObject *self, PyObject *args)
{
proxylib_init();
Py_RETURN_NONE;
}static PyObject* return_by_value_wrapper(PyObject *self, PyObject *args)
{
PyObject* obj; if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
return NULL; void* input_ptr = PyCapsule_GetPointer(obj, "obj");
void* output_Ptr = return_by_value(input_Ptr);
PyObject* output = PyCapsule_New(output_Ptr, "outPtr");
return output;
}static PyObject* return_by_reference_wrapper(PyObject *self, PyObject * args)
{
PyObject* obj; if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
return NULL; char *buffer;
int length; return_by_reference(PyCapsule_GetPointer(obj, "obj"), &buffer, &length);
return PyByteArray_FromStringAndSize(buffer, length);
}static PyObject* return_by_value_and_by_reference_wrapper(PyObject *self, PyObject * args)
{
PyObject* obj; if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
return NULL; char *buffer;
int length; void* ptr = return_by_value_and_by_reference(PyCapsule_GetPointer(obj, "obj"), &buffer, &length);
PyObject* capsule_Ptr = PyCapsule_New(ptr, "ptr");PyObject* py_buffer = PyByteArray_FromStringAndSize(buffer, length);
PyObject* tuple_Ptr = PyTuple_Pack(2, capsule_Ptr, py_buffer); return tuple_Ptr;}
And of course, we have to register this function within a module’s symbol table
static PyMethodDef callMethods[] = {
{ "return_nothing", return_nothing_wrapper, METH_NOARGS},
{ "return_by_value", return_by_value_wrapper, METH_VARARGS},
{ "return_by_reference", "return_by_reference_wrapper", METH_VARARGS},
{ "return_by_value_and_by_reference", "return_by_value_and_by_reference_wrapper", METH_VARARGS}, { NULL, NULL, 0, NULL }
};
And finally module definition and init function for the module.
// Module definition
// The arguments of this structure tell Python what to call your extension,
// what it's methods are and where to look for it's method definitionsstatic struct PyModuleDef call_definition = {
PyModuleDef_HEAD_INIT,
"calllib",
"A Python module extension for C++ lib",
-1,
callMethods
};PyMODINIT_FUNC PyInit_call(void) {
Py_Initialize();
return PyModule_Create(&call_definition);
}
after we just have to write setup script.
#
import os
import shutil#
from distutils.core import setup, Extension
from distutils.command.build_py import build_py
from distutils.command.clean import clean
from subprocess import check_output#
proxylib_module = Extension('wrapper.cpp', sources = ['.cpp'], extra_compile_args=["-fPIC", "-std=c++11"], language="c++", extra_link_args=['callLib.a'] #call lib file)#
class BuilderClass(build_py):
LIBNAME="callLib"
LIBFILE=LIBNAME + ".a"
SOURCE_DIR=os.getcwd()def run(self):
os.system("git clone call lib gt repo path {}".format(self.LIBNAME))
os.chdir("{}/{}".format(self.SOURCE_DIR, self.LIBNAME))
if os.path.exists("build"):
shutil.rmtree("build", ignore_errors=True)
os.mkdir("build")
os.chdir("build") os.system("cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE='ON'")
os.system("make -j4")#
setup(name='your_project_name',
version='0.1.0',
python_requires='>3',
description='Python SDK for call lib.',
ext_modules=[callModule],
packages=[your packages inside python project, 'tests'],
cmdclass={'build_py': BuilderClass}
)
after running setup script it generates calllib.so shared library. Now we can open a Python shell and call our wrapper functions.
>>> import calllib
>>> obj = return_by_value()
>>> buffer = return_by_reference(obj)
...Now let’s again return to a wrapper.cpp file, to look through PyObject and its subtypes - PyObject*, PyCapsule_GetPointer, PyCapsule_New, PyByteArray_FromStringAndSize. PyObject is a python readable basic object. All others are functions for working with PyObject, mainly for type check and type conversion. For a full list and more info please see
Thanks for reading !!!
If you find this article interesting and helpful, please share and 👏!