Performance & Native Acceleration

Optimise only what you have measured. This page covers the profiling tools that find the real hotspot, and the escape hatches when Python itself is the bottleneck: vectorisation, JIT and native compilation, and the buffer protocol that lets you cross into C without copying data.

Performance Profiling and Native Acceleration

Optimise only what you have measured. The expert workflow is profile first, then act on the hotspot -- never guess.

import cProfile, pstats

with cProfile.Profile() as pr:
    run_workload()
pstats.Stats(pr).sort_stats("cumulative").print_stats(20)

When profiling proves Python itself is the bottleneck (tight numeric loops, not I/O), escape to native code:

# 1. Vectorise first -- often the biggest win is staying in NumPy/Polars
import numpy as np
result = np.where(arr > 0, arr * 2, 0)   # C-speed loop, no Python per-element

# 2. Cython / numba for hot numeric kernels
from numba import njit
@njit                                     # JIT-compiles to machine code
def mandelbrot_escape(c: complex, max_iter: int) -> int:
    z = 0j
    for i in range(max_iter):
        z = z * z + c
        if (z.real * z.real + z.imag * z.imag) > 4.0:
            return i
    return max_iter

# 3. Rust via PyO3 + maturin for production-grade native extensions
#    (this is how pydantic-core, polars, and ruff are built)

The decision order: vectorise → cache/algorithm → numba/Cython → Rust/PyO3. Reaching for C before profiling, or before trying NumPy, is the classic premature-optimisation mistake.

C Interoperability and Zero-Copy

Python's performance escape hatch is calling into C, and the buffer protocol lets you do it without copying data. memoryview exposes another object's memory directly -- you can slice and reshape gigabytes without duplicating them:

data = bytearray(b"large payload ...")
view = memoryview(data)       # zero-copy window onto the same bytes
chunk = view[10:20]           # still zero-copy -- no new buffer allocated
chunk[0] = 0x41               # writes through to `data` (it's a view)
# Pass `view` to socket.send, file.write, struct -- they consume it without a copy.

For calling existing native libraries, ctypes (stdlib, no build step) loads a shared object and declares signatures at runtime:

import ctypes
libc = ctypes.CDLL("libc.so.6")
libc.strlen.restype = ctypes.c_size_t        # ALWAYS set argtypes/restype --
libc.strlen.argtypes = [ctypes.c_char_p]     # the default int return truncates pointers
libc.strlen(b"hello")                         # 5

The honest guidance on which tool:

The recurring theme across all of it: the boundary between Python and native code is where bugs and crashes live -- mismatched argtypes segfault, a freed buffer behind a memoryview corrupts memory, and the GIL must be released explicitly for native threads to run in parallel. Cross it deliberately, and only after profiling proves you must.