C/C++ Extension for Python

Recently I need to write a very fast and flexible crypto lib. And C++ is one of the richest languages with many crypto modules and libs, and also one of the fastest languages. But the problem was that I needed that lib with different languages, mainly Python, but writing the same code with other languages was not the idea I expected for. So the best solution was just called C++ code from Python. And here I want to share some little techniques and “challenges” I met on my way of doing that, as there are many cases when C++’s speed and riches are indispensable and unquestionable.

Let's start from very beginning.

For calling C++ from Python we first need to write some wrapper code which can “translate” C++ into Python.
Let’s have a classic and a bit bored function hello in hello.c file.

char * hello(){ return "Hello!";}

Let’s write a wrapper (Python callable)function for hello function in a wrapper.cpp file.

#include "Python.h"
#include <stdio.h>
#include "hello.c"
static PyObject * hello_wrapper(PyObject * self, PyObject * args)
{
  char * input; // param for hello 
  char * result; // return value from hello
  PyObject * ret; // python object which should be returned from                 
                  // wrapper function

  // parse arguments
  if (!PyArg_ParseTuple(args, "s", &input)) {
    return NULL;
  }

  // run the actual function
  result = hello(input);

  // build the resulting string into a Python object.
  ret = PyBytes_FromString(result);
  free(result);

  return ret;
}

Note that in our first line we include header file called “Python.h”. To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros, and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header file “Python.h”.

After writing a wrapper function we have to register this function within a module’s symbol table (all Python functions live in a module, even if they’re actually C functions!).

static PyMethodDef HelloMethods[] = {
  { "hello", hello_wrapper, METH_VARARGS},
  { NULL, NULL, 0, NULL}
};

And we need to write a module definition and init function for the module.

static struct PyModuleDef hello_definition = {
    PyModuleDef_HEAD_INIT,
    "hello",
    "A Python module extension for C++ lib",
    -1,
    HelloMethods
};
PyMODINIT_FUNC PyInit_hello(void) {
    Py_Initialize();
    return PyModule_Create(&hello_definition);
}

This was a basic example, where C++ code was as short as we could keep it with our Python package. Now let’s look a case, where we have a very long C++ code that it is not rational to drag with our Python package.

Let’s imagine we have C++ code and all function which should be called from Python are defined in a call.h header file.

// include all header files 
void return_nothing();
void* return_by_value(void *ptr);
void return_by_reference(void* ptr, char** buffer, int* number);
void* return_by_value_and_by_reference(void* ptr, char** buffer, int* number);

Now, let’s write wrapper functions in a wrapper.cpp file.

#include "Python.h"
#include <stdio.h>
#include "call.h"
#include <string>
static PyObject* return_nothing_wrapper(PyObject *self, PyObject *args)
{
    proxylib_init();
    Py_RETURN_NONE;
}
static PyObject* return_by_value_wrapper(PyObject *self, PyObject *args)
{
    PyObject* obj;
    if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
        return NULL;
    void* input_ptr = PyCapsule_GetPointer(obj, "obj");
    void* output_Ptr = return_by_value(input_Ptr);
    PyObject* output = PyCapsule_New(output_Ptr, "outPtr");

    return output;
}
static PyObject* return_by_reference_wrapper(PyObject *self, PyObject * args)
{
    PyObject* obj;
    if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
        return NULL;
    char *buffer;
    int length;
    return_by_reference(PyCapsule_GetPointer(obj, "obj"), &buffer, &length);
    return PyByteArray_FromStringAndSize(buffer, length);
}
static PyObject* return_by_value_and_by_reference_wrapper(PyObject *self, PyObject * args)
{
    PyObject* obj;
    if (! PyArg_UnpackTuple( args, "obj",0,1, &obj))
        return NULL;
    char *buffer;
    int length;
    void* ptr = return_by_value_and_by_reference(PyCapsule_GetPointer(obj, "obj"), &buffer, &length);
    PyObject* capsule_Ptr = PyCapsule_New(ptr, "ptr");
PyObject* py_buffer = PyByteArray_FromStringAndSize(buffer, length);
    PyObject* tuple_Ptr = PyTuple_Pack(2, capsule_Ptr, py_buffer);
    return tuple_Ptr;
}

And of course, we have to register this function within a module’s symbol table

static PyMethodDef callMethods[] = {
    { "return_nothing", return_nothing_wrapper, METH_NOARGS},
    { "return_by_value", return_by_value_wrapper, METH_VARARGS},
    { "return_by_reference", "return_by_reference_wrapper", METH_VARARGS},
    { "return_by_value_and_by_reference", "return_by_value_and_by_reference_wrapper", METH_VARARGS},
  { NULL, NULL, 0, NULL }
};

And finally module definition and init function for the module.

// Module definition
// The arguments of this structure tell Python what to call your extension,
// what it's methods are and where to look for it's method definitions
static struct PyModuleDef call_definition = {
    PyModuleDef_HEAD_INIT,
    "calllib",
    "A Python module extension for C++ lib",
    -1,
    callMethods
};
PyMODINIT_FUNC PyInit_call(void) {
    Py_Initialize();
    return PyModule_Create(&call_definition);
}

after we just have to write setup script.

#
import os
import shutil
#
from distutils.core import setup, Extension
from distutils.command.build_py import build_py
from distutils.command.clean import clean
from subprocess import check_output
#
proxylib_module = Extension('wrapper.cpp', sources = ['.cpp'], extra_compile_args=["-fPIC", "-std=c++11"], language="c++", extra_link_args=['callLib.a'] #call lib file)
#
class BuilderClass(build_py):
    LIBNAME="callLib"
    LIBFILE=LIBNAME + ".a"
    SOURCE_DIR=os.getcwd()
def run(self):
      os.system("git clone call lib gt repo path {}".format(self.LIBNAME))
      os.chdir("{}/{}".format(self.SOURCE_DIR, self.LIBNAME))
      if os.path.exists("build"):
          shutil.rmtree("build", ignore_errors=True)
      os.mkdir("build")
      os.chdir("build")
      os.system("cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE='ON'")
      os.system("make -j4")
#
setup(name='your_project_name',
      version='0.1.0',
      python_requires='>3',
      description='Python SDK for call lib.',
      ext_modules=[callModule],
      packages=[your packages inside python project, 'tests'],
      cmdclass={'build_py': BuilderClass}
      )

after running setup script it generates calllib.so shared library. Now we can open a Python shell and call our wrapper functions.

>>> import calllib
>>> obj = return_by_value()
>>> buffer = return_by_reference(obj)
...

Now let’s again return to a wrapper.cpp file, to look through PyObject and its subtypes - PyObject*, PyCapsule_GetPointer, PyCapsule_New, PyByteArray_FromStringAndSize. PyObject is a python readable basic object. All others are functions for working with PyObject, mainly for type check and type conversion. For a full list and more info please see

Python/C API Reference Manual - Python 3.4.9 documentation

Thanks for reading !!!
If you find this article interesting and helpful, please share and 👏!