EMR | Softchief Learn

Solved : Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try

July 9, 2018

Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[172.27.10.191:50010,DS-51d68378-35de-4c70-b27e-7a98a53919cc,DISK], DatanodeInfoWithStorage[172.27.10.223:50010,DS-f172c682-713d-4a8f-b8af-69198ddc6756,DISK]], original=[DatanodeInfoWithStorage[172.27.10.191:50010,DS-51d68378-35de-4c70-b27e-7a98a53919cc,DISK], DatanodeInfoWithStorage[172.27.10.223:50010,DS-f172c682-713d-4a8f-b8af-69198ddc6756,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via ‘dfs.client.block.write.replace-datanode-on-failure.policy’ in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:925) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:988) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1156) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454)…
Read more

Create and Insert to HBase table example

February 8, 2018

Login into master node : [ec2-user@ip-123-45-67-89 ~]$ sudo hbase shell HBase Shell; enter ‘help<RETURN>’ for list of supported commands. Type “exit<RETURN>” to leave the HBase Shell Version 1.3.1, rUnknown, Fri Sep 22 21:28:57 UTC 2017 hbase(main):001:0> list tables Create table : Command stntax create ‘<table_name>’, ‘<column_family>’ Example hbase(main):004:0> create ’employee_hbase’, ‘cf1’ Insert data into above…
Read more

Where is emrfs-site.xml ?

February 7, 2018

The emrfs-site.xml is being create if the EMRFS is enabled when creating the EMR in AWS. You can manage other related configurations by logging into the master node and in the following location, [ec2-user@ip-123-45-67-89 ~]$ ls -ltr /usr/share/aws/emr/emrfs/conf/emrfs-site.xml -rw-r–r– 1 root root 609 Feb 6 21:59 /usr/share/aws/emr/emrfs/conf/emrfs-site.xml [ec2-user@ip-123-45-67-89 ~]$ [ec2-user@ip-123-45-67-89 ~]$ cat /usr/share/aws/emr/emrfs/conf/emrfs-site.xml <?xml version=”1.0″?>…
Read more