Folks transitioning from java or any object-oriented language to python might struggling or spending their substantial time when assigning the method argument variables to their class level variable every-time they write reusable classes and functions definitions. For example, Imagine the above function, what if I have to long list of arguments and in multiple functions…
Read more
As we all know, the Amazon S3 is an amazing storage to deal with persisting the hot and cold data in this big-data era. It has 99.99% uptime which has been claimed by amazon. You can follow the documentation from amazon for more details. When we all agreeing upon the S3 storage is easiest, resilient…
Read more
Objective The blog is to copy a file from the project path to build path after the project build completes Example Use-case : In my case, I have a Dockerfile in the project root directory. I wanted to copy the file to the build directory once project build completes. Project structure : smartechie-pro | —– src…
Read more
Error : java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty This is the problem I have faced when running build or test command on any java project. The following is the error I got, + make test ./gradlew test Downloading https://services.gradle.org/distributions/gradle-3.3-all.zip Exception in thread “main” javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty…
Read more
Install / upgrade Postgresql smartechie-macos :~ $ brew postgresql Get installation details smartechie-macos :~ $ brew info postgres Result postgresql: stable 10.4 (bottled), HEAD Object-relational database system https://www.postgresql.org/ Conflicts with: postgres-xc (because postgresql and postgres-xc install the same binaries.) /usr/local/Cellar/postgresql/9.6.3 (3,260 files, 36.6MB) Poured from bottle on 2017-06-05 at 20:47:39 /usr/local/Cellar/postgresql/10.4 (3,389 files, 39.2MB) *…
Read more
Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[172.27.10.191:50010,DS-51d68378-35de-4c70-b27e-7a98a53919cc,DISK], DatanodeInfoWithStorage[172.27.10.223:50010,DS-f172c682-713d-4a8f-b8af-69198ddc6756,DISK]], original=[DatanodeInfoWithStorage[172.27.10.191:50010,DS-51d68378-35de-4c70-b27e-7a98a53919cc,DISK], DatanodeInfoWithStorage[172.27.10.223:50010,DS-f172c682-713d-4a8f-b8af-69198ddc6756,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via ‘dfs.client.block.write.replace-datanode-on-failure.policy’ in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:925) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:988) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1156) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454)…
Read more
Problem : smartechies.mac $ hbase shell ArgumentError: wrong number of arguments (0 for 1) method_added at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10 method_added at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129 Pattern at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2 (root) at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1 require at org/jruby/RubyKernel.java:1062 (root) at file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42 (root) at /usr/local/Cellar/hbase/1.2.6/libexec/bin/../bin/hirb.rb:38 Solution : Point the JAVA_HOME variable to correct value in conf/hbase-env.sh Example : smartechies.mac $ cat ~/.profile | grep JAVA_HOME export…
Read more
[ec2-user@ip-123-45-67-890 ~]$ ls -ltr /usr/lib/hue/build/env/bin/hue -rwxr-xr-x 1 root root 523 Sep 22 22:09 /usr/lib/hue/build/env/bin/hue [ec2-user@ip-123-45-67-890 ~]$ sudo /usr/lib/hue/build/env/bin/hue createsuperuser Username (leave blank to use ‘root’): sudhir Email address: mail2sudhir.online@gmail.com Password: Password (again): Superuser created successfully. [ec2-user@ip-123-45-67-890 ~]$
Error : “/usr/bin/env: node: No such file or directory Solution : ln -s /usr/bin/nodejs /usr/bin/node Followup issue : Error: EACCES: permission denied, open ‘/home/ubuntu/.config/configstore/bower-github.json’ Solved (click here for solution)
Error : 21:04:36.791 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /home/ubuntu/.gradle/native 21:04:36.817 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. 21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 21:04:36.819 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: 21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine java version from ‘9.0.4’. 21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 21:04:36.822 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is: 21:04:36.823 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] java.lang.IllegalArgumentException:…
Read more
ERROR : 18/03/02 01:42:17 INFO RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping for 10000ms. com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: Directory ‘bucket/folder/_temporary’ present in the metadata but not s3 at com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.getFileStatus(ConsistencyCheckerS3FileSystem.java:506) Root cause : Mostly the consistent problem comes due to Manual deletion of files and directory from S3 console retry logic in spark and hadoop…
Read more
Connecting to a Database on a DB Instance Running the MySQL Database Engine Once Amazon RDS provisions your DB instance, you can use any standard SQL client application to connect to a database on the DB instance. In this example, you connect to a database on a MySQL DB instance using MySQL monitor commands. One…
Read more
Welcome to WordPress. This is your first post. Edit or delete it, then start writing!
Ulysses, Ulysses – Soaring through all the galaxies. In search of Earth, flying in to the night. Ulysses, Ulysses – Fighting evil and tyranny, with all his power, and with all of his might. Ulysses – no-one else can do the things you do. Ulysses – like a bolt of thunder from the blue. Ulysses…
Read more
80 days around the world, we’ll find a pot of gold just sitting where the rainbow’s ending. Time – we’ll fight against the time, and we’ll fly on the white wings of the wind. 80 days around the world, no we won’t say a word before the ship is really back. Round, round, all around…
Read more
Create table : hive> CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2)); OK Time taken: 1.084 seconds List tables : hive> show tables; OK students values__tmp__table__1 Time taken: 0.023 seconds, Fetched: 2 row(s) Describe table : hive> describe students; OK name varchar(64) age …
Read more
Login into master node : [ec2-user@ip-123-45-67-89 ~]$ sudo hbase shell HBase Shell; enter ‘help<RETURN>’ for list of supported commands. Type “exit<RETURN>” to leave the HBase Shell Version 1.3.1, rUnknown, Fri Sep 22 21:28:57 UTC 2017 hbase(main):001:0> list tables Create table : Command stntax create ‘<table_name>’, ‘<column_family>’ Example hbase(main):004:0> create ’employee_hbase’, ‘cf1’ Insert data into above…
Read more
The emrfs-site.xml is being create if the EMRFS is enabled when creating the EMR in AWS. You can manage other related configurations by logging into the master node and in the following location, [ec2-user@ip-123-45-67-89 ~]$ ls -ltr /usr/share/aws/emr/emrfs/conf/emrfs-site.xml -rw-r–r– 1 root root 609 Feb 6 21:59 /usr/share/aws/emr/emrfs/conf/emrfs-site.xml [ec2-user@ip-123-45-67-89 ~]$ [ec2-user@ip-123-45-67-89 ~]$ cat /usr/share/aws/emr/emrfs/conf/emrfs-site.xml <?xml version=”1.0″?>…
Read more
Problem FAILED: SemanticException Cannot find class ‘parquet.hive.DeprecatedParquetInputFormat’ Solution [hadoop@ip-123-45-67-890 extjars]$mkdir extjars [hadoop@ip-123-45-67-890 extjars]$cd extjars/ Now Download required jars: [hadoop@ip-123-45-67-890 extjars]$for f in parquet-avro parquet-cascading parquet-column parquet-common parquet-encoding parquet-generator parquet-hadoop parquet-hive parquet-pig parquet-scrooge parquet-test-hadoop2 parquet-thrift do curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/${f}/1.2.4/${f}-1.2.4.jar done curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/parquet-format/1.0.0/parquet-format-1.0.0.jar [hadoop@ip-123-45-67-890 extjars]$ ls -ltr total 5472 -rw-rw-r– 1 hadoop hadoop 891821 Dec…
Read more
Exception: smartechie:~ sudhir.pradhan$ hive hiveSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.8.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/hive-common-2.3.1.jar!/hive-log4j2.properties Async: trueException in thread “main” java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize(Path.java:254) at org.apache.hadoop.fs.Path.<init>(Path.java:212) at…
Read more
Problem : [hadoop@spradhan]$ unzip *.zip Archive: a.csv.zip caution: filename not matched: b.csv.zip caution: filename not matched: c.csv.zip caution: filename not matched: d.csv.zip caution: filename not matched: e.csv.zip Solution : [hadoop@spradhan]$ unzip ‘*.zip’ If you run in background, [hadoop@spradhan]$ nohup unzip ‘*.zip’ &
Install aws cli in the ec2 instance [if not installed] $ sudo yum install aws-cli Configure aws cli $ aws configure AWS Access Key ID [None]: <your_access_key> AWS Secret Access Key [None]:<your_secret_key> Default region name [None]: Default output format [None]: Execute sync command in ec2 instance aws s3 sync s3://<path_to_file> <ec2_local_path>
Get daily articles in your inbox for free.
Step:2 - Subscribe Softchief (Guru) Here and Press Bell icon then select All.
Step:3 - Subscribe Softchief (Xtreme) Here and Press Bell icon then select All.
Step:4 - subscribe SmarTechie channel here and Press Bell icon then select All.
After the above 4 steps. Give your email id below and get the answers document in your inbox.
Drop your contact information and our Experts will call you in les than a minute to discuss about your requirement.
