Rahul Singhai Blog

Check MirrorMaker.scala for more details. Target cluster setup Download and install Kafka (target cluster). Select appropriate version and download its tgz from Kafka Downloads page. tar -zxf kafka_2.11-0.10.0.1.tgz cd kafka_2.11-0.10.0.1 Configure Target Kafka cluster's ZooKeeper vim ./config/zookeeper.properties # the directory where the snapshot is stored. dataDir=/work/kafka_2.11-0.10.0.1/zookeeper-data # the port at which the clients will connect clientPort=2181 # disable the per-ip limit on thseparatedof connections since this is a non-production config maxClientCnxns=0 Start Target Kafka cluster's ZooKeeper ./bin/zookeeper-server-start.sh config/zookeeper.properties Configure Target Kafka cluster's Server vim ./config/server.properties # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 # The number of threads handling network requests num.network.threads=3 # The number of threads doing disk I/O num.io.threads=8 # The sen...

The Subject Alternative Name field lets you specify additional host names (sites, IP addresses, common names, etc.) to be protected by a single SSL Certificate, such as a Multi-Domain (SAN) or Extend Validation Multi-Domain Certificate. Benefits Secure Host Names on Different Base Domains in One SSL Certificate: A Wildcard Certificate can protect all first-level subdomains on an entire domain, such as *.example.com. However, a Wildcard Certificate cannot protect both www.example.com and www.example.net. Virtual Host Multiple SSL Sites on a Single IP Address: Hosting multiple SSL-enabled sites on a single server typically requires a unique IP address per site, but a Multi-Domain (SAN) Certificate with Subject Alternative Names can solve this problem. Microsoft IIS and Apache are both able to Virtual Host HTTPS sites using Multi-Domain (SAN) Certificates. Greatly Simplify Your Server's SSL Configuration: Using a Multi-Domain (SAN) Certificate saves you the hassle and time invo...

ElasticSearch has a concept of bucket selection generated from aggregation. This works as a pipeline, where first aggregation generates buckets, and then bucket selection further filters out buckets. We have an ElasticSearch index ' daily_reports ', where a row represents a particular version of report. When a report is created a new row is inserted in the index with a new ' reportId ' field value and ' publishDate ' field representing the UNIX timestamp. Each report/row has multiple other fields representing properties of the report, for e.g., ' title ', ' activity ', ' reportStatus ', ' reportLevel ', etc. When the report is edited/deleted, a new row is inserted into the index, with same ' reportId ', but different '_id', 'publishDate', 'reportLevel' etc. Now if user wants to get the latest version for each report matching a particular filter criterion ( reportLevel = Monitoring AND repor...

Suppose we have a JSON payload (may be a stream coming from Kafka) that looks like this: {"regionName":"asia","nodes":[{"node":"router.dc1.singhaiuklimited.com","weight":"0"},{"node":"router.dc2.singhaiuklimited.com","weight":"100"}]} {"regionName":"asia","nodes":[{"node":"router.dc1.singhaiuklimited.com","weight":"0"},{"node":"router.dc1.singhaiuklimited.com","weight":"0"},{"node":"router.dc2.singhaiuklimited.com","weight":"100"},{"node":"router.dc2.singhaiuklimited.com","weight":"100"}]} To loop through the nested fields and generate extra fields from the calculations while using Logstash, we can do something like this: input { kafka { bootstrap_servers => "kafka.singhaiukl...

Refrences: 1. http://www.cyberciti.biz/tips/howto-linux-unix-bash-shell-setup-prompt.html 2. http://www.cyberciti.biz/faq/bash-shell-change-the-color-of-my-shell-prompt-under-linux-or-unix/ Add following in ~/.profile (or ~/.bash_profile, or ~/.bashrc, or /etc/profile if you have access): export JAVA_VERSION=1.8 export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home export PATH=$PATH:$JAVA_HOME/bin alias ll='ls -al' # This alias recursively destroys all .DS_Store files in the folder I am currently in alias killDS='find . -name *.DS_Store -type f -delete' # This alias reloads this file alias reload_profile='. ~/.profile' # Mac get stuck very often and are extremely slow and unstable on shutdowns. This forces a shutdown. alias poweroff='sudo /sbin/shutdown -h now' # To ignore space changes while doing git diff alias gitdf='git diff --ignore-space-change' # Show Git branch in prompt parse_git_branch() { git bra...

MongoDB A cross-platform open-source document-oriented NoSQL database with an Enterprise level distribution that includes additional security and management features. MongoDB is supported and steered by the company MongoDB . What do we mean by NoSQL? The NoSQL database category loosely incorporates all databases which do not have a rigid schema and are not queryable via a structured query language. They are normally designed specifically to scale easily to support very large datasets. What is a document based database? Instead of storing information as a table with a series of records or rows, MongoDB stores data as searchable documents. A document is analogous to the object model in object orientated programming languages. How does MongoDB represent documents? MongoDB uses JSON to represent its documents. JSON is a simple key-value pair textual serialization of an object, for example, a bunch of grapes could be represented as the following JSON string: { Name: "Bunch of...

Main resources: Scala Cheat Sheet Reactive Cheat Sheet Spark Cheat sheet Spark Quick start Spark programming guide Spark Streaming : processing real-time data streams Spark SQL and DataFrames : support for structured data and relational queries MLlib : built-in machine learning library GraphX : Spark’s new API for graph processing Scala programming examples: Define a object with main function -- Helloworld. object HelloWorld { def main(args: Array[String]) { println("Hello, world!") } } Execute main function: scala> HelloWorld.main(null) Hello, world! Creating RDDs Parallelized Collections: val data = Array(1, 2, 3, 4, 5) val distData = sc.parallelize(data) External Datasets: val distFile = sc.textFile("data.txt") Above command returns the content of the file: scala> distFile.collect() res16: Array[String] = Array(1,2,3, 4,5,6) SparkContext.wholeTextFiles can return (filename, content). val distFile = sc.wholeTextFiles("/tmp/t...