How to stop AWS EC2 instance automatically when it is idle?

AWS is use and pay model (if it is reserved instance it is pay and use model). During usage of AWS organization needs to control the cost.  Set up automated mechanism whenever EC2 instance  does not running. Here is the steps how to implement this....

amazon-web-services

What is AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy in AWS CodePipeline

AWS becomes the major player in infrastructure needs and you cannot avoid cloud when you are talking about Hadoop Cluster. AWS DevOps is very hot now and in infrastructure segment. If we moved to AWS then we need to utilize the all services that AWS is providing....

What is machine learning in Layman terms?

Machine learning is buzzword and highest salary offered compare with other skills. Need for Machine learning engineer is growing significantly than Data Scientist and Data Architect. Here is the LinkedIn proof for you. What is Machine Learning? Humans we are taking decision based on past experience. We...

boundary-value

Why we need to use IF NOT EXIST command during Hive Table creation?

During Hive Table or Database creation we are usually using “if not exist” command. Hive is not allowing to create the same table name and it throws the error. In real time we are using more number of tables and in “Data lake” concepts...

Boundary Value Query in Sqoop

Sqoop gauges its workload Sqoop has perform parallel imports. The default mappers are 4 that means it took four splitting tasks. Sqoop uses splitting columns of RDBMS table to split the workload. It splits by identify the primary key column of the RDBMS table....

Sqoop import and export

JDBC Drivers The Sqoop import or export operations (The Data from RDBMS import to Sqoop or Data from HDFS export to RDBMS) are done by help of JDBC drivers. In Sqoop the drivers are not bundled because of licensing issue. However most of the...

Incremental Load in Sqoop

Sqoop job in production system

In real-time production system Sqoop job has option file with password protection. Sqoop job can be run with Oozie workflow also that I can write post separately. In this post explaining the steps to creating the password and option files then how to run...

Apache Sqoop

Apache Sqoop Job – How to create and run?

What is Sqoop Job? Incremental load mode option is good in Sqoop. However it has demerits of remember the last successful modified record or time. Next time if we run the incremental load in sqoop we need to run the data where last successful...

Apache Sqoop

Incremental Load in Sqoop

Consistency of Data Apache Sqoop framework is helping to fetch the data from RDBMS to HDFS and/or HDFS to RDBMS. Typically RDBMS the data are keep on incremented or appended with the existing data and existing data has been updated (edit/update or delete). After...