Rahul Singhai Blog

Let's say we want to read and parse a CSV file containing Key value pairs. We will be using Alpakka's CSVParser for this. A snippet of a file (src/main/resources/CountryNicCurrencyKeyValueMap.csv) that shows mapping from Country NIC code to currency code with pipe (|) as field delimiter: AD|EUR AE|AED AF|AFN AG|XCD AI|XCD AL|ALL AM|AMD AN|ANG AO|AOA AQ|AQD AR|ARS AS|EUR AT|EUR AU|AUD AW|ANG AX|EUR AZ|AZN BA|BAM BB|BBD BD|BDT BE|EUR BF|XOF BG|BGN BH|BHD BI|BIF BJ|XOF BL|EUR BM|BMD BN|BND BO|BOB BR|BRL BS|BSD BT|INR Following is the code: import java.io.File import java.nio.charset.StandardCharsets import akka.actor.ActorSystem import akka.stream._ import akka.stream.alpakka.csv.scaladsl.CsvParsing import akka.stream.scaladsl.{FileIO, Flow, Sink} import akka.util.ByteString import scala.collection.immutable import scala.concurrent.{ExecutionContext, _} import scala.concurrent.duration._ implicit val system: ActorSystem = ActorSystem("TestApplication") implicit ...

Performance Tuning of Kafka is critical when your cluster grow in size. Below are few points to consider to improve Kafka performance: Consumer group ID : Never use same exact consumer group ID for dozens of machines consuming from different topics. All of those commits will end up on the same exact partition of __consumer_offsets , hence the same broker, and this might in turn cause performance problems. Choose the consumer group ID to group_id+topic_name . Skewed : A broker is skewed if its number of partitions is greater that the average of partitions per broker on the given topic. Example: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2). Try to make sure that none of the brokers is skewed. Spread : Brokers spread is the percentage of brokers in the cluster that has partitions for the given topic. Example: 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic. Try to achieve 100% broker spread...