TLDR – Compile librdkafka from source
Both Ubuntu 20.04 and AMI Linux 2 (effectively CentOS) have an older version of the librdkafka. My Ubuntu Vagrant VM shows v1.2.1 where as php-simple-kafka-lib shows the requirements as:
php: ^7.3|^8.0
ext-simple_kafka_client: >=0.1.0
librdkafka: >=1.4.0
In order to install the PHP simple_kafka_client extension (using Pecl or complied from source) it needs a newer version of librdkafka than what comes packaged. You’ll need to compile it from source.
Compiling from source:
echo "-- Compiling librdkafka from source"
sudo mkdir /opt
cd /opt
sudo git clone https://github.com/confluentinc/librdkafka
cd librdkafka
sudo ./configure --install-deps
sudo make
sudo make install
echo "-- Pecl Installing simple_kafka_client"
sudo pecl channel-update pecl.php.net
sudo pecl install simple_kafka_client
echo "PHP ini for Kafka now it should be installed:"
php -ini | grep kafka
# Example output $ php -ini | grep kafka
# simple_kafka_client
# kafka support => enabled
# librdkafka version (runtime) => 2.1.1-3-g68455a
# librdkafka version (build) => 2.1.1.255
More details
As technical co-founder of Drivible I figured it’s time to create an event streaming system which we could use for both background tasks (we have a database heavy task system but need something light) and also something we can use for things like Webhooks and eventually a whole lot of event and streaming data.
Whilst SQS, Redis Queues or RabbitMQ would be fine for basic task queuing, I’d heard a lot about Kafka. Both that it was great, but also a pain to run.
Thankfully AWS has MSK – Managed Streaming for Apache Kafka. So the hosting isn’t much of an issue.
We don’t need much right now, just an instance in 2 regions, so 2 nodes. About $50/month each.
It’ll still cost more than SQS which would probably be $2/month for our usage, if that. But there’s a lot more we can do with Kafka.
I did some research, setup the Cluster and it took me a little while to do things like change from IAM based Auth, which seems like it’s only supported by Java clients and needs special source code monkey patching (although I might be wrong), to the more traditional SASL/SCRAM.
I then had to setup a symmetric encryption KMS key and then logins to MSK using the AWS Secrets Manager.
The cluster was now sitting, waiting to be used. Of course it’s only available on the VPC (Virtual Private Cloud) that the EC2 servers are sitting on.
I needed to test it. As per the PHP Simple Kafka Client docs there’s some example files. So I tried to use those.
I git cloned the example repo, changed into the directory and ran composer install, only to get the errors about the PHP extension.
git clone https://github.com/php-kafka/php-kafka-examples.git
cd php-kafka-examples/src/ext-php-simple-kafka-client/
composer installIf you don’t have PHP composer already installed you can use the latest command line installation instructions as per https://getcomposer.org/download/ and install it globally by running
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php -r "if (hash_file('sha384', 'composer-setup.php') === '55ce33d7678c5a611085589f1f3ddf8b3c52d662cd01d4ba75c0ee0459970c2200a51f492d557530c71c15d8dba01eae') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
php composer-setup.php
php -r "unlink('composer-setup.php');"
sudo mv composer.phar /usr/local/bin/composer or skip the last line and use php composer.phar instead of just composer
When I ran this initially I got:
composer install No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information. Loading composer repositories with package information Info from https://repo.packagist.org: #StandWithUkraine Updating dependencies Your requirements could not be resolved to an installable set of packages. Problem 1 - Root composer.json requires php-kafka/php-simple-kafka-lib dev-main -> satisfiable by php-kafka/php-simple-kafka-lib[dev-main]. - php-kafka/php-simple-kafka-lib dev-main requires ext-simple_kafka_client ^0.1 -> it is missing from your system. Install or enable PHP's simple_kafka_client extension.
So I tried the installation steps as per the installation guide.
sudo pecl install simple_kafka_clientUnfortunately I got a big wall of text with an error that ‘make’ failed and the initial error line looked to be:
/root/pear/temp/simple_kafka_client/php_simple_kafka_client_int.h:192:51: error: unknown type name ‘rd_kafka_error_t’
Unfortunately Googling that didn’t provide many useful results. One person said they’d given up with trying to use Kafka and PHP.
I tried to compile the php-simple-kafka-client from source as per the documentation.
git clone https://github.com/php-kafka/php-simple-kafka-client.git
cd php-simple-kafka-client
phpize && ./configure && make -j5 all && make installBut it also failed.
Pecl install (or manual compile from source install) error
$ sudo pecl install simple_kafka_client
WARNING: channel "pecl.php.net" has updated its protocols, use "pecl channel-update pecl.php.net" to update
downloading simple_kafka_client-0.1.4.tgz ...
Starting to download simple_kafka_client-0.1.4.tgz (35,991 bytes)
..........done: 35,991 bytes
45 source files, building
running: phpize
Configuring for:
PHP Api Version: 20190902
Zend Module Api No: 20190902
Zend Extension Api No: 320190902
building in /root/pear/temp/pear-build-roote0vsXG/simple_kafka_client-0.1.4
running: /root/pear/temp/simple_kafka_client/configure --with-php-config=/bin/php-config
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking for pkg-config... /bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for cc... cc
....
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
configure: patching config.h.in
configure: creating ./config.status
config.status: creating config.h
config.status: executing libtool commands
running: make
/bin/sh /root/pear/temp/pear-build-roote0vsXG/simple_kafka_client-0.1.4/libtool --mode=compile cc -I. -I/root/pear/temp/simple_kafka_client -DPHP_ATOM_INC -I/root/pear/temp/pear-build-roote0vsXG/simple_kafka_client-0.1.4/include -I/root/pear/temp/pear-build-roote0vsXG/simple_kafka_client-0.1.4/main -I/root/pear/temp/simple_kafka_client -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /root/pear/temp/simple_kafka_client/simple_kafka_client.c -o simple_kafka_client.lo
libtool: compile: cc -I. -I/root/pear/temp/simple_kafka_client -DPHP_ATOM_INC -I/root/pear/temp/pear-build-roote0vsXG/simple_kafka_client-0.1.4/include -I/root/pear/temp/pear-build-roote0vsXG/simple_kafka_client-0.1.4/main -I/root/pear/temp/simple_kafka_client -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /root/pear/temp/simple_kafka_client/simple_kafka_client.c -fPIC -DPIC -o .libs/simple_kafka_client.o
In file included from /root/pear/temp/simple_kafka_client/simple_kafka_client.c:43:0:
/root/pear/temp/simple_kafka_client/php_simple_kafka_client_int.h:192:51: error: unknown type name 'rd_kafka_error_t'
void create_kafka_error(zval *return_value, const rd_kafka_error_t *error);
^~~~~~~~~~~~~~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c: In function 'zim_SimpleKafkaClient_Kafka_setOAuthBearerTokenFailure':
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:282:11: warning: implicit declaration of function 'rd_kafka_oauthbearer_set_token_failure'; did you mean 'rd_kafka_header_get_all'? [-Wimplicit-function-declaration]
err = rd_kafka_oauthbearer_set_token_failure(intern->rk, error_string);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rd_kafka_header_get_all
/root/pear/temp/simple_kafka_client/simple_kafka_client.c: In function 'zim_SimpleKafkaClient_Kafka_setOAuthBearerToken':
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:333:11: warning: implicit declaration of function 'rd_kafka_oauthbearer_set_token'; did you mean 'rd_kafka_header_get_all'? [-Wimplicit-function-declaration]
err = rd_kafka_oauthbearer_set_token(intern->rk, token, lifetime_ms, principal_name, extensions, extension_size, errstr, errstr_size);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rd_kafka_header_get_all
In file included from /usr/include/php/main/php.h:441:0,
from /root/pear/temp/simple_kafka_client/simple_kafka_client.c:40:
/root/pear/temp/simple_kafka_client/simple_kafka_client.c: In function 'zm_startup_simple_kafka_client':
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:382:19: error: 'RD_KAFKA_PURGE_F_QUEUE' undeclared (first use in this function); did you mean 'RD_KAFKA_MSG_F_FREE'?
COPY_CONSTANT(RD_KAFKA_PURGE_F_QUEUE);
^
/usr/include/php/Zend/zend_constants.h:53:105: note: in definition of macro 'REGISTER_LONG_CONSTANT'
#define REGISTER_LONG_CONSTANT(name, lval, flags) zend_register_long_constant((name), sizeof(name)-1, (lval), (flags), module_number)
^~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:382:5: note: in expansion of macro 'COPY_CONSTANT'
COPY_CONSTANT(RD_KAFKA_PURGE_F_QUEUE);
^~~~~~~~~~~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:382:19: note: each undeclared identifier is reported only once for each function it appears in
COPY_CONSTANT(RD_KAFKA_PURGE_F_QUEUE);
^
/usr/include/php/Zend/zend_constants.h:53:105: note: in definition of macro 'REGISTER_LONG_CONSTANT'
#define REGISTER_LONG_CONSTANT(name, lval, flags) zend_register_long_constant((name), sizeof(name)-1, (lval), (flags), module_number)
^~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:382:5: note: in expansion of macro 'COPY_CONSTANT'
COPY_CONSTANT(RD_KAFKA_PURGE_F_QUEUE);
^~~~~~~~~~~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:383:19: error: 'RD_KAFKA_PURGE_F_INFLIGHT' undeclared (first use in this function); did you mean 'RD_KAFKA_PURGE_F_QUEUE'?
COPY_CONSTANT(RD_KAFKA_PURGE_F_INFLIGHT);
^
/usr/include/php/Zend/zend_constants.h:53:105: note: in definition of macro 'REGISTER_LONG_CONSTANT'
#define REGISTER_LONG_CONSTANT(name, lval, flags) zend_register_long_constant((name), sizeof(name)-1, (lval), (flags), module_number)
^~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:383:5: note: in expansion of macro 'COPY_CONSTANT'
COPY_CONSTANT(RD_KAFKA_PURGE_F_INFLIGHT);
^~~~~~~~~~~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:384:19: error: 'RD_KAFKA_PURGE_F_NON_BLOCKING' undeclared (first use in this function); did you mean 'RD_KAFKA_PURGE_F_INFLIGHT'?
COPY_CONSTANT(RD_KAFKA_PURGE_F_NON_BLOCKING);
^
/usr/include/php/Zend/zend_constants.h:53:105: note: in definition of macro 'REGISTER_LONG_CONSTANT'
#define REGISTER_LONG_CONSTANT(name, lval, flags) zend_register_long_constant((name), sizeof(name)-1, (lval), (flags), module_number)
^~~~
/root/pear/temp/simple_kafka_client/simple_kafka_client.c:384:5: note: in expansion of macro 'COPY_CONSTANT'
COPY_CONSTANT(RD_KAFKA_PURGE_F_NON_BLOCKING);
^~~~~~~~~~~~~
make: *** [simple_kafka_client.lo] Error 1
ERROR: `make' failed
I nearly gave up, but figured I’d already gone this far, I need to go a level deeper.
Hence I checked how to compile the librdkafka from source as per the instructions at the start of this post and Bam! It worked!
> sudo pecl install simple_kafka_client downloading simple_kafka_client-0.1.4.tgz … Starting to download simple_kafka_client-0.1.4.tgz (35,991 bytes) ……….done: 35,991 bytes 45 source files, building running: phpize Configuring for: PHP Api Version: 20190902 Zend Module Api No: 20190902 Zend Extension Api No: 320190902 building in /root/pear/temp/pear-build-rootOC54Bd/simple_kafka_client-0.1.4 running: /root/pear/temp/simple_kafka_client/configure --with-php-config=/bin/php-config ... Build process completed successfully Installing '/usr/lib64/php/modules/simple_kafka_client.so' install ok: channel://pecl.php.net/simple_kafka_client-0.1.4 Extension simple_kafka_client enabled in php.ini
I can now run the pecl install and the extension worked and so does the composer install.
Ohh, I spoke to soon. It’s working on my Ubuntu machine, but not on my EC2 servers running AMI Linux 2
Running PHP I get:
PHP Warning: PHP Startup: Unable to load dynamic library 'simple_kafka_client.so' (tried: /usr/lib64/php/modules/simple_kafka_client.so (/usr/lib64/php/modules/simple_kafka_client.so: undefined symbol: rd_kafka_error_is_retriable),
There’s a suggestion I should update the config.m4 however the link to the example fails and when I tried I couldn’t get it to work. Turns out that’s a red herring.
I’ve left an issue on Github and Nick, the creator suggested there was a version mis-match issue between what was compiled at build time and what was being used at run-time.
When I checked on my Ubuntu Vagrant VM which I thought was fine (but didn’t have network access to the Kafka Cluster to test) this turned out to be the case.
$ php -i | grep kafka /etc/php/7.4/cli/conf.d/21-simple_kafka_client.ini simple_kafka_client kafka support => enabled librdkafka version (runtime) => 1.2.1 librdkafka version (build) => 2.1.1.255
So on a fresh EC2 instance I compiled librdkafka from source, without trying the yum install and it worked!
The suggested steps at the top of this post work when you’ve not already tried installing from the distro.
Why is it this hard??
I certainly feels like trying to use Kafka and PHP isn’t something that’s done very much.
Or maybe it’s just a faster moving target than I expected.
Looks like it’ll take a few more moments for the first EC2 deploy build as things are compiled from source, but should be fine after that.
Yay!
