April 27, 2024 | Softchief Learn

Exception when creating hive table from hdfs parquet file

January 31, 2018

Problem FAILED: SemanticException Cannot find class ‘parquet.hive.DeprecatedParquetInputFormat’ Solution [hadoop@ip-123-45-67-890 extjars]$mkdir extjars [hadoop@ip-123-45-67-890 extjars]$cd extjars/ Now Download required jars: [hadoop@ip-123-45-67-890 extjars]$for f in parquet-avro parquet-cascading parquet-column parquet-common parquet-encoding parquet-generator parquet-hadoop parquet-hive parquet-pig parquet-scrooge parquet-test-hadoop2 parquet-thrift do curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/${f}/1.2.4/${f}-1.2.4.jar done curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/parquet-format/1.0.0/parquet-format-1.0.0.jar [hadoop@ip-123-45-67-890 extjars]$ ls -ltr total 5472 -rw-rw-r– 1 hadoop hadoop 891821 Dec…
Read more

Unzip Multiple Files from Linux Command Line

January 27, 2018

Problem : [hadoop@spradhan]$ unzip *.zip Archive: a.csv.zip caution: filename not matched: b.csv.zip caution: filename not matched: c.csv.zip caution: filename not matched: d.csv.zip caution: filename not matched: e.csv.zip Solution : [hadoop@spradhan]$ unzip ‘*.zip’ If you run in background, [hadoop@spradhan]$ nohup unzip ‘*.zip’ &

Copy file or folder from amazon S3 to EC2

January 26, 2018

Install aws cli in the ec2 instance [if not installed] $ sudo yum install aws-cli Configure aws cli $ aws configure AWS Access Key ID [None]: <your_access_key> AWS Secret Access Key [None]:<your_secret_key> Default region name [None]: Default output format [None]: Execute sync command in ec2 instance aws s3 sync s3://<path_to_file> <ec2_local_path>

Hive : Unable to start metastore issue

January 4, 2018

Exception : smartechie:confluent-3.3.0 sudhir.pradhan$ hive SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.1.0/libexec/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.8.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/usr/local/Cellar/hive/2.1.0/libexec/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: trueException in thread “main” java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)…
Read more

Hive: Unable to instantiate Metastore

January 4, 2018

Exception : 2017-12-28T15:05:52,943 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore – 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore2017-12-28T15:05:52,943 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore – 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStoreMetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:83) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6883) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6878) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:7136) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:7063) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at…
Read more

Most Used FetchXML Queries in Dynamics 365 CRM

January 4, 2018

FetchXML is a XML based query language used in Microsoft Dynamics 365 CRM to fetch data. FetchXML is capable of doing many things as explained below. Can only be used to retrieve data not able to perform CUD (Create/Update/Delete) operation. Can be used in JavaScript to retrieve data in client side also used in server…
Read more

7 Popular Articles of 2017 in Dynamics 365

January 1, 2018

Wish you all a very special Happy New Year 2018. Here is a list of Top 10 articles for you which are very popular in 2017 year. Please go through the below articles. 1 101 most used Dynamics 365 CRM code snippets. 2 13 important Technologies for CRM Developers. 3 11 Top websites for Dynamics…
Read more