Setting up python confluent-kafka for macOS

All steps and links are for version 3.0 of confluent and macOS Sierra.

First thing we need is having confluent kafka services up and running. Download tar file under Zip and Tar archives section from here. Uncompress downloaded file and start/test services. All the steps to do so can be found in confluent quickstart guide. It is quite straightforward and no problems should rise here.

After trying to install confluent-kafka python libraries with pip install confluent-kafka I’ve seen it crashes in macOS with below error

fatal error: ‘librdkafka/rdkafka.h’ file not found

It seems is not finding librdkafka headers and the way I found to solve it is building and installing everything from scratch.

We will need librdkafka to build and install confluent-kafka python library so lets do that.

git clone https://github.com/edenhill/librdkafka
cd librdkafka
./configure
make
sudo make install

By default headers of librdkafka will be in /usr/local/include/librdkafka and libraries in /usr/local/lib. We will need these paths to install python libraries so make sure everything is there.

Next step is install confluent-kafka python library, lets get it from git and build it.

git clone https://github.com/confluentinc/confluent-kafka-python

We will need to set C_INCLUDE_PATH and LIBRARY_PATH environment variables to the path where librdkafka headers and libs are before building it.

export C_INCLUDE_PATH=/usr/local/include/
export LIBRARY_PATH=/usr/local/lib/lib

Lets install it

python setup.py build
sudo python setup.py install

We are done, now confluent python library should be ready to be imported from our project.