We have been utilizing Apache Pulsar for our messaging needs, however, we've found it to be quite complex and prone to numerous bugs. These issues have unfortunately led to significant downtime within our production environment, impacting our overall operational efficiency. If there is new technology, such as AutoMQ, that can potentially enhance our operational maintenance and utilization of Kafka, why not give it a try?
The table summarizes the differences between various streaming platforms. It highlights how different technical architecture choices can lead to different end results. What are your thoughts on this table?
The following explanation from Strimzi about the difficulty of implementing autoscaling in Kubernetes can also be applied to answer this question. Using spot instances requires the computing layer to be stateless.
For Kafka brokers, auto-scaling is complicated because of their architecture. Adding or removing brokers is simple. But directing some load to them is complicated because they are in a way form of data storage. And moving the whole partitions between brokers is expensive. The partitions often contain huge amounts of data which need to be shifted from one broker to another - that will take time, it will have a performance penalty on the other traffic and possibly cost even real money for the data transfers.
