C/C++ Extension for Python

Recently I need to write a very fast and flexible crypto lib. And C++ is one of the richest languages with many crypto modules and libs, and also one of the fastest languages. But the problem was that I needed that lib with different languages, mainly Python, but writing the same code with other languages was not the idea I expected for. So the best solution was just called C++ code from Python. And here I want to share some little techniques and “challenges” I met on my way of doing that, as there are many cases when C++’s speed and riches are indispensable and unquestionable.

Let's start from very beginning.

For calling C++ from Python we first need to write some wrapper code which can “translate” C++ into Python.
Let’s have a classic and a bit bored function hello in hello.c file.

char * hello(){ return "Hello!";}

Let’s write a wrapper (Python callable)function for hello function in a wrapper.cpp file.

#include "Python.h"
#include <stdio.h>
#include "hello.c"
static PyObject * hello_wrapper(PyObject * self, PyObject * args)
{
char * input; // param for hello
char * result; // return value from hello
PyObject * ret; // python object which should be returned from
// wrapper function

// parse arguments
if (!PyArg_ParseTuple(args, "s", &input)) {
return NULL;
}

// run the actual function
result = hello(input);

// build the resulting string into a Python object.
ret = PyBytes_FromString(result);
free(result);

return ret;
}

Note that in our first line we include header file called “Python.h”. To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros, and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header file “Python.h”.

After writing a wrapper function we have to register this function within a module’s symbol table (all Python functions live in a module, even if they’re actually C functions!).

static PyMethodDef HelloMethods[] = {
{ "hello", hello_wrapper, METH_VARARGS},
{ NULL, NULL, 0, NULL}
};

And we need to write a module definition and init function for the module.

static struct PyModuleDef hello_definition = {
PyModuleDef_HEAD_INIT,
"hello",
"A Python module extension for C++ lib",
-1,
HelloMethods
};
PyMODINIT_FUNC PyInit_hello(void) {
Py_Initialize();
return PyModule_Create(&hello_definition);
}

This was a basic example, where C++ code was as short as we could keep it with our Python package. Now let’s look a case, where we have a very long C++ code that it is not rational to drag with our Python package.

Let’s imagine we have C++ code and all function which should be called from Python are defined in a call.h header file.

// include all header files void return_nothing();
void* return_by_value(void *ptr);
void return_by_reference(void* ptr, char** buffer, int* number);
void* return_by_value_and_by_reference(void* ptr, char** buffer, int* number);

Now, let’s write wrapper functions in a wrapper.cpp file.

#include "Python.h"
#include <stdio.h>
#include "call.h"
#include <string>
static PyObject* return_nothing_wrapper(PyObject *self, PyObject *args)
{
proxylib_init();
Py_RETURN_NONE;
}
static PyObject* return_by_value_wrapper(PyObject *self, PyObject *args)
{
PyObject* obj;
if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
return NULL;
void* input_ptr = PyCapsule_GetPointer(obj, "obj");
void* output_Ptr = return_by_value(input_Ptr);
PyObject* output = PyCapsule_New(output_Ptr, "outPtr");

return output;
}
static PyObject* return_by_reference_wrapper(PyObject *self, PyObject * args)
{
PyObject* obj;
if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
return NULL;
char *buffer;
int length;
return_by_reference(PyCapsule_GetPointer(obj, "obj"), &buffer, &length);
return PyByteArray_FromStringAndSize(buffer, length);
}
static PyObject* return_by_value_and_by_reference_wrapper(PyObject *self, PyObject * args)
{
PyObject* obj;
if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
return NULL;
char *buffer;
int length;
void* ptr = return_by_value_and_by_reference(PyCapsule_GetPointer(obj, "obj"), &buffer, &length);
PyObject* capsule_Ptr = PyCapsule_New(ptr, "ptr");
PyObject* py_buffer = PyByteArray_FromStringAndSize(buffer, length);
PyObject* tuple_Ptr = PyTuple_Pack(2, capsule_Ptr, py_buffer);
return tuple_Ptr;}

And of course, we have to register this function within a module’s symbol table

static PyMethodDef callMethods[] = {
{ "return_nothing", return_nothing_wrapper, METH_NOARGS},
{ "return_by_value", return_by_value_wrapper, METH_VARARGS},
{ "return_by_reference", "return_by_reference_wrapper", METH_VARARGS},
{ "return_by_value_and_by_reference", "return_by_value_and_by_reference_wrapper", METH_VARARGS},
{ NULL, NULL, 0, NULL }
};

And finally module definition and init function for the module.

// Module definition
// The arguments of this structure tell Python what to call your extension,
// what it's methods are and where to look for it's method definitions
static struct PyModuleDef call_definition = {
PyModuleDef_HEAD_INIT,
"calllib",
"A Python module extension for C++ lib",
-1,
callMethods
};
PyMODINIT_FUNC PyInit_call(void) {
Py_Initialize();
return PyModule_Create(&call_definition);
}

after we just have to write setup script.

#
import os
import shutil
#
from distutils.core import setup, Extension
from distutils.command.build_py import build_py
from distutils.command.clean import clean
from subprocess import check_output
#
proxylib_module = Extension('wrapper.cpp', sources = ['.cpp'], extra_compile_args=["-fPIC", "-std=c++11"], language="c++", extra_link_args=['callLib.a'] #call lib file)
#
class BuilderClass(build_py):
LIBNAME="callLib"
LIBFILE=LIBNAME + ".a"
SOURCE_DIR=os.getcwd()
def run(self):
os.system("git clone call lib gt repo path {}".format(self.LIBNAME))
os.chdir("{}/{}".format(self.SOURCE_DIR, self.LIBNAME))
if os.path.exists("build"):
shutil.rmtree("build", ignore_errors=True)
os.mkdir("build")
os.chdir("build")
os.system("cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE='ON'")
os.system("make -j4")
#
setup(name='your_project_name',
version='0.1.0',
python_requires='>3',
description='Python SDK for call lib.',
ext_modules=[callModule],
packages=[your packages inside python project, 'tests'],
cmdclass={'build_py': BuilderClass}
)

after running setup script it generates calllib.so shared library. Now we can open a Python shell and call our wrapper functions.

>>> import calllib
>>> obj = return_by_value()
>>> buffer = return_by_reference(obj)
...

Now let’s again return to a wrapper.cpp file, to look through PyObject and its subtypes - PyObject*, PyCapsule_GetPointer, PyCapsule_New, PyByteArray_FromStringAndSize. PyObject is a python readable basic object. All others are functions for working with PyObject, mainly for type check and type conversion. For a full list and more info please see

Thanks for reading !!!
If you find this article interesting and helpful, please share and 👏!

--

--

I'm mathematician and programmer. Currently working at Skycryptor startup, which focuses on data encryption. Chess lover and bicyclist.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gegham Jivanyan

I'm mathematician and programmer. Currently working at Skycryptor startup, which focuses on data encryption. Chess lover and bicyclist.