Folks transitioning from java or any object-oriented language to python might struggling or spending their substantial time when assigning the method argument variables to their class level variable every-time they write reusable classes and functions definitions. For example, Imagine the above function, what if I have to long list of arguments and in multiple functions…
Read more
As we all know, the Amazon S3 is an amazing storage to deal with persisting the hot and cold data in this big-data era. It has 99.99% uptime which has been claimed by amazon. You can follow the documentation from amazon for more details. When we all agreeing upon the S3 storage is easiest, resilient…
Read more
Objective The blog is to copy a file from the project path to build path after the project build completes Example Use-case : In my case, I have a Dockerfile in the project root directory. I wanted to copy the file to the build directory once project build completes. Project structure : smartechie-pro | —– src…
Read more
Error : java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty This is the problem I have faced when running build or test command on any java project. The following is the error I got, + make test ./gradlew test Downloading https://services.gradle.org/distributions/gradle-3.3-all.zip Exception in thread “main” javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty…
Read more
Error : “/usr/bin/env: node: No such file or directory Solution : ln -s /usr/bin/nodejs /usr/bin/node Followup issue : Error: EACCES: permission denied, open ‘/home/ubuntu/.config/configstore/bower-github.json’ Solved (click here for solution)
Error : 21:04:36.791 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /home/ubuntu/.gradle/native 21:04:36.817 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. 21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: 21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine java version from ‘9.0.4’. 21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is: 21:04:36.823 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] java.lang.IllegalArgumentException:…
Read more
ERROR : 18/03/02 01:42:17 INFO RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping for 10000ms. com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: Directory ‘bucket/folder/_temporary’ present in the metadata but not s3 at com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.getFileStatus(ConsistencyCheckerS3FileSystem.java:506) Root cause : Mostly the consistent problem comes due to Manual deletion of files and directory from S3 console retry logic in spark and hadoop…
Read more
Connecting to a Database on a DB Instance Running the MySQL Database Engine Once Amazon RDS provisions your DB instance, you can use any standard SQL client application to connect to a database on the DB instance. In this example, you connect to a database on a MySQL DB instance using MySQL monitor commands. One…
Read more
Create table : hive> CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2)); OK Time taken: 1.084 seconds List tables : hive> show tables; OK students values__tmp__table__1 Time taken: 0.023 seconds, Fetched: 2 row(s) Describe table : hive> describe students; OK name varchar(64) age …
Read more
Login into master node : [ec2-user@ip-123-45-67-89 ~]$ sudo hbase shell HBase Shell; enter ‘help<RETURN>’ for list of supported commands. Type “exit<RETURN>” to leave the HBase Shell Version 1.3.1, rUnknown, Fri Sep 22 21:28:57 UTC 2017 hbase(main):001:0> list tables Create table : Command stntax create ‘<table_name>’, ‘<column_family>’ Example hbase(main):004:0> create ’employee_hbase’, ‘cf1’ Insert data into above…
Read more
The emrfs-site.xml is being create if the EMRFS is enabled when creating the EMR in AWS. You can manage other related configurations by logging into the master node and in the following location, [ec2-user@ip-123-45-67-89 ~]$ ls -ltr /usr/share/aws/emr/emrfs/conf/emrfs-site.xml -rw-r–r– 1 root root 609 Feb 6 21:59 /usr/share/aws/emr/emrfs/conf/emrfs-site.xml [ec2-user@ip-123-45-67-89 ~]$ [ec2-user@ip-123-45-67-89 ~]$ cat /usr/share/aws/emr/emrfs/conf/emrfs-site.xml <?xml version=”1.0″?>…
Read more
Problem FAILED: SemanticException Cannot find class ‘parquet.hive.DeprecatedParquetInputFormat’ Solution [hadoop@ip-123-45-67-890 extjars]$mkdir extjars [hadoop@ip-123-45-67-890 extjars]$cd extjars/ Now Download required jars: [hadoop@ip-123-45-67-890 extjars]$for f in parquet-avro parquet-cascading parquet-column parquet-common parquet-encoding parquet-generator parquet-hadoop parquet-hive parquet-pig parquet-scrooge parquet-test-hadoop2 parquet-thrift do curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/${f}/1.2.4/${f}-1.2.4.jar done curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/parquet-format/1.0.0/parquet-format-1.0.0.jar [hadoop@ip-123-45-67-890 extjars]$ ls -ltr total 5472 -rw-rw-r– 1 hadoop hadoop 891821 Dec…
Read more
Exception: smartechie:~ sudhir.pradhan$ hive hiveSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.8.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/hive-common-2.3.1.jar!/hive-log4j2.properties Async: trueException in thread “main” java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize(Path.java:254) at org.apache.hadoop.fs.Path.<init>(Path.java:212) at…
Read more
Problem : [hadoop@spradhan]$ unzip *.zip Archive: a.csv.zip caution: filename not matched: b.csv.zip caution: filename not matched: c.csv.zip caution: filename not matched: d.csv.zip caution: filename not matched: e.csv.zip Solution : [hadoop@spradhan]$ unzip ‘*.zip’ If you run in background, [hadoop@spradhan]$ nohup unzip ‘*.zip’ &
Install aws cli in the ec2 instance [if not installed] $ sudo yum install aws-cli Configure aws cli $ aws configure AWS Access Key ID [None]: <your_access_key> AWS Secret Access Key [None]:<your_secret_key> Default region name [None]: Default output format [None]: Execute sync command in ec2 instance aws s3 sync s3://<path_to_file> <ec2_local_path>
Exception : [2017-11-10 08:32:32,183] ERROR Task hdfs-sink-prqt-stndln-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask:455) java.lang.RuntimeException: java.util.concurrent.ExecutionException: io.confluent.connect.hdfs.errors.HiveMetaStoreException: Invalid partition for default.srp-oracle-jdbc-stdln-raw-KFK_SRP_HDFS_SINK_TEST: partition=0 at io.confluent.connect.hdfs.DataWriter.write(DataWriter.java:226) at io.confluent.connect.hdfs.HdfsSinkTask.put(HdfsSinkTask.java:103) at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:435) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:251) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:180) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:148) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: io.confluent.connect.hdfs.errors.HiveMetaStoreException: Invalid partition for…
Read more
Exception : [ec2-user@ip-123-45-67-890 ~]$ hive Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j2.properties Async: falseException in thread “main” java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=ec2-user, access=WRITE, inode=”/user/ec2-user”:hdfs:hadoop:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1728) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1712) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1695) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3896) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)…
Read more
#!/bin/sh gradle_version=”4.4-rc-5″ #Your Custom Installation Path install_target_path=”/opt/gradle” wget -N https://services.gradle.org/distributions/gradle-${gradle_version}-all.zip mkdir -p ${install_target_path} unzip gradle-${gradle_version}-all.zip mv gradle-${gradle_version} ${install_target_path}/ ln -sfn gradle-${gradle_version} ${install_target_path}/latest chown -R ec2-user:ec2-user ${install_target_path} printf “export GRADLE_HOME=${install_target_path}/latest\nexport PATH=\$PATH:\$GRADLE_HOME/bin” > /etc/profile.d/gradle.sh . /etc/profile.d/gradle.sh # check installation gradle -v
[ec2-user@ip-xxx-xx-xx-xx ~]$ sudo vi /etc/docker/daemon.json [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo vi /etc/docker/daemon.json [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo cat /etc/docker/daemon.json { “bip”: “yyy.yyy.y.y/zz” } [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo service docker stopStopping docker: [ OK ] [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo…
Read more
Install Maven : Following are the set of commands need to be executed sequentially to install maven. sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo sudo yum install -y apache-maven mvn –version And you all set to run any “mvn” (maven) command in ec2 instance. Output :
Step-1: Install Java Runtime – java 1.8 sudo yum install java-1.8.0 if you need a java compiler and other developer tools: sudo yum install java-1.8.0-openjdk-devel Step-2: If you have multiple versions and one of those default , use the alternatives command as follows and enter the selection number as guided in the terminal. sudo /usr/sbin/alternatives –config…
Read more
Open “Terminal” app from Applications or Command + Space and then type “Terminal” press Enter/Return Key Install Homebrew (Copy / Paste the following command in the Terminal window and press enter) ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)” < /dev/null 2> /dev/null Now install Kafka(Copy / Paste the following command in the Terminal window and press enter)…
Read more
Get daily articles in your inbox for free.
Step:2 - Subscribe Softchief (Guru) Here and Press Bell icon then select All.
Step:3 - Subscribe Softchief (Xtreme) Here and Press Bell icon then select All.
Step:4 - subscribe SmarTechie channel here and Press Bell icon then select All.
After the above 4 steps. Give your email id below and get the answers document in your inbox.
Drop your contact information and our Experts will call you in les than a minute to discuss about your requirement.
