Siva Nadesan is a Data Engineering Leader with nearly two decades of experience in the data space. His passion includes working on data engineering and writing technical blogs. He likes to learn new technologies and apply his knowledge to build solutions for real world problems.
I have been using self-hosted services for a while now. It all started with …
Overview Not enough teams focus on conceptual modeling of data before …
Overview Snowpipe is Snowflake’s most cost-efficient method for …
Overview There are multiple ways to replicate data from an Amazon RDS …
My journey into smart home automation began with Samsung SmartThings, which …
What is Orchestration ? Orchestration in the context of IT and software …
Overview Transferring data between Amazon S3 buckets is a common …
In this article, we’re diving into a data engineering demo project …
Overview dbt (data build tool) is a popular open-source tool used for …
Unleash the Power of Chatting with Data: A Deep Dive into Snowflake, …
CloudWatch logs provide insight into logs from many AWS services, but …
In this article, we will see how to analyze documents in Snowflake database …
The Terraform Snowflake Provider allows to provision Snowflake resources …
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a managed service …
Federated authentication enables your users to connect to Snowflake using …
A Snowflake object naming convention is a framework for naming objects in a …
Kafka consumer lag gives the offset difference between last produced …
In this article, we will see how to create an Apache Kafka cluster on …
Snowpipe is Snowflake’s continuous data ingestion service. Snowpipe …
In this article, we will see how to use Aiven’s GCS Sink Connector …
Materialize is a streaming database for real-time applications. Materialize …
Materialize is a streaming database for real-time applications. Materialize …
DataStage Next Gen is here now. It is fully managed and cloud-ready data …
In this article, we will see how to install and configure Kubernetes …
In this article, we will see how to load XML from an s3 bucket into …
In this article, we will see how to initialize and configure a new dbt …
In this article, we will see how to get/pull data from an api, store in S3 …
In this article, we will see how to install AWS SAM in a docker container …
In this article, we will see How To Install Discourse In AWS Lightsail. …
In this article, we will see how to stream data from Kafka to apm platforms …
In this article, we will see how to load and use XML files in snowflake. …
In this article, we will see how to use dbt with Snowflake. dbt (data build …
In this article, we will see how to stream data from kafka to snowflake …
In this article, we will see how to build a simple weather alert …
In this article, we will see how to integrate Kafka connect with Amazon …
In this article, we will see how to use the Snowflake connector to stream …
In this article, we will see how to send log files generated by an …
In this article, we will see how to use elk in Docker with the help of logs …
ksqlDB is built on top of Kafka Streams, a lightweight, powerful Java …
In this article, we will see how to install Confluent Kafka using Ansible …
In this article, we will see how to generate a tombstone record in Kafka …
In this article, we will see how to generate test data for Kafka using …
ksql-test-runner is a ksqlDB testing tool to test set of KSQL statements. …
In this article, we will see the usage of some common Java commands for non …
In this article, we will see how to install Multiple Virtual Storage, more …
The Raspberry Pi is a low cost, credit-card sized computer that plugs into …
Pi-hole is a network-wide ad blocker. This will block Ads to all devices …
In this article, we will go over few options to check if a port is open in …
In this article, we will see the steps to redirect naked domain to full …
I have been hosting my blog in Google Blogger for a long time. It is still …
Pi-hole is a network-wide ad blocker. This will block Ads to all devices …
In this article, we will see the steps to add an existing folder to git …
In this article, we will see the steps to create a Zoho email account for …
In this article, we will see the steps to connect your custom google domain …
If you have worked on ksqlDB, you would have come across this scenario …
In this article, we will go over steps to implement Change Data …
In this article, we will go over few scenarios and to explore ARRAY, MAP …
Prerequisites Install Confluent platform for your operating system …
In this article, we will go over the process to create and validate ksqlDB …
Prerequisites Install Confluent platform for your operating system using …
In this article, we will discuss some of the common Confluent Kafka …
This article explains the steps required to make AWS CLI call from local …
Confluent KSQL is the streaming SQL engine that enables real-time data …
Flatten JSON is a simple REST API which takes a JSON object as input and …
Here are the steps to connect to the MongoDB database from DataStage using …
Here are the steps to start All Services in IBM InfoSphere Information …
In this article, we will go over find command in Linux with examples. Find …
Kafka Connect integrates Apache Kafka with other systems and makes it easy …
Overview Article to show the usage of some common Kafka commands. These …
Login into hive Command : hive Logoff from hive Command : quit; Command : …
In this article, we will go over some frequently used HDFS commands Hadoop …
bashrc is the best place to add aliases and Bash related functions. The …
Let’s say we need to FTP a file (say filename.txt) from my local …
Query to find the space used by a table SELECT DS.TABLESPACE_NAME , …
Query to find database uptime SELECT host_name , instance_name , …
Query to find the queries executed by an user in Oracle SELECT VS.USERNAME …
This article explains the steps to find the service name of a database …
This is a utility to scan the DataStage job run logs to find the usage of a …
Introduction DataStage job logs and status files can grow huge at times due …
In most of the applications with a batch cycle, IT and Business …
Notes If you get a command not found for “wget” or …