How to use Hadoop FS API inside Storm Bolt in java -

-

I would like to store in HDFC data implies that storm I have added Hup FS API code Bolt Class , But it is throwing the compile error with the storm.

The following is the hurricane bolt class: 

  package bolts; Import java.io * *; Import java.util. *; Import Java.net *; Import org.apache.hadoop.fs. *; Import org.apache.hadoop.conf *; Import org.apache.hadoop.io *; Import org.apache.hadoop.mapred *; Import org.apache.hadoop.util *; Import backtip. Storm.pholology Basic Optical Collector; Import backtipes Storm. Topology Outputfield Declair; Import BackType.storm.pholology.base Basebescholt; Import backtype.storm.tuple.Fields; Import backtype.storm.tuple.Tuple; Import backtype.storm.tuple.Values; Public class datormormizer basebesciblet (public executed null) (Tupal Input, BasicOutput Collector Collector) {String sentence = input.getString (0); String [] process = sentence split (""); Int n = 1; String REC = ""; Try {string filepet = "/root/data/top_output.csv"; String dest = "hdfs: // localhost: 9000 / user / root / infinite / top_output / top_output_1.csv"; Configuration conf = new configuration (); File system filesystem = FileSystem.get (conf); Println (file system); Path srcPath = new path (source); Path dstPath = new path (destination); String file name = source.substring (source.lastIndexOf ('/') + 1, source.length ()); Try {if (FileSystem.exists (dstPath))} {FSDataOutputStream out = fileSystem.create (dstPath, true); InputStream = New Buffered InputStream (New FileInputStream (New File (Source)); Byte [] B = new byte [1024]; Int numBytes = 0; While ((numbytes = in.read (b)) gt; 0) {out.write (b, 0, numbytes); } In.close (); Out.close (); } Other {fileSystem.copyFromLocalFile (srcPath, dstPath); }} Hold (exception e) {System.err.println ("the exception was caught:" + E); System.exit (1); } Finally {fileSystem.close (); }} Hold (IOException e) {e.printStackTrace (); }}   
}  
 I have included the Hadoop jar in CLASSPATH. Value of classpath after:  
  $ STORM_HOME /storm-0k8klkjar:$jawa_hom/lib/:$hdup_hom/hdup-kor-lk0k4kjar:$hdup_hom/lib/:$storm_hom/lib/   
 In addition, Hoop libraries have been copied:  Thaup-cor-1.0.4.jar, Commerce-collection-3.2.1.jar  and  commons-cli-1.2.jar  In the  Storm / lib  directory.  
 When I am creating this project, it is throwing the following error:  
  3006 [thread-16] error backtext.storm. Daemon.executor - java.lang.NoClassDefFoundError :. Org / Apache / Commons / Configuration / org.apache.hadoop.metrics2.lib.DefaultMetricsSystem is configurable to & lt; Init & gt; (DefaultMetricsSystem.java37) at org.apache.hadoop. Metrics2.lib.DefaultMetricsSystem & Lt; Clinit & gt; On org.apache.hadoop.security.UgiInstrumentation.create (UgiInstrumentation.java:51) (DefaultMetricsSystem.java:34) org.apache tialize (UserGroupInformation.java:216) on org.apache.hadoop.security.UserGroupInformation.ini. At org.apache at hasoop.security.UserGroupInformation.ensureInitialized (UserGroupInformation.java:184) org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled (UserGroupInformation.java:236) .hadoop.security.UserGroupInformation.getLoginUser org.apache. on hadoop.security.UserGroupInformation.getCurrentUser (UserGroupInformation.java:452) (UserGroupInformation.java:466) org.apache.hadoop.fs.FileSystem $ cache $ key. & Lt; Init & gt; org.apache.hadoop.fs.FileSystem $ Cache.get (FileSystem.java:1395) on org.apache.hadoop.fs.FileSystem.get (FileSystem.java:1494) (FileSystem.java:254) backtype.storm on .topology.BasicBoltExecutor.execute on org.apache.hadoop.fs.FileSystem.get (FileSystem.java:123) bolts.DataNormalizer.execute (DataNormalizer.java:67) to (BasicBoltExecutor.java:32). .....................    
 
 
 The error message tells you that the Apache Commons configuration is missing. You have to add it to the classpath.  
 More generally, you should add all the Hydrop dependence to your classpath. You can look for them by using a dependency manager (Maven, Ivy, Gradle, etc.) or look for a code on Hadoop installed on  / usr / lib / Hadoop / lib .

Comments

Post a Comment

Popular posts from this blog

excel vba - How to delete Solver(SOLVER.XLAM) code -

-
After opening several examples of macro / code written, I suddenly get stuck with a code that seems safe. Solver (SOLVER.XLAM) is doing this notation and I would like to remove it from my system. I do not think this is a relativant code. Can anyone tell me what should I do? I have already installed something to remove the password but Solver is still asking for password and I can not get rid of it! itemprop = "text"> Solver Excel is add-in If you want to delete it, then Excel Application & gt; Options & gt; Add-ins & gt; Search for 'Go to' button and gt; Uncheck the solver on the list like press something (I can not give you the exact path because I have the Polish version of Excel). Trying to open Solver's VBA code is not good because it There is allegedly copyright. Besides, I do not think Solver creates any disadvantages while working with Excel or VBA.
Read more

github - Teamcity & Git - PR merge builds - anyway to get HEAD commit hash? -

-
I have a team city build project with Gitbab VCS route, I have been triggered for both / head on PR and / Merge the referee. The annoying thing is that you can not do anything useful with the merged hash - it is not present in Gitobb. I want to exclude newsgate packages with head in gutub comet situations in the version number along with Hewash (Newsaging 7) for the music build, but against head hash. However, I can not see a way to get the head hashes while constructing that merge. I just want to head because it's useless - we only care about merge. (PR - Pull Request) This command works reliably while running One on / merge is git log --no-merge -1 ---- = '% h'
Read more

ios - Replace text in UITextView run slowly -

-
I have a UITextView and I change a word like the code at the end of UITextView.text: - (void) Replace TexttechWitWord: (NSString *) word {NSString * currentTextViewText = self.textView.text; CurrentTextViewText = [currentTextViewText stringByReplacingCharactersInRange: NSMakeRange ([self.textView.text length] - [word length], [word length]) withString: word; Self.textView.text = currentTextViewText; } Replacement work is fine, but it runs very slowly and very slowly when the textView now lengthens Is there any better way? I try the replacement range: with text for a week but still stuck. Not to know how to use it to replace it in your case. Please guide! Thanks in advance! This is an important job. As you are using an irreversible string and you are creating a very long string and before Assigned to NSString created from. NSString will be used instead of NSMutableString . - (UITableViewCell *) TableView: (UITableView *) Table View CellForOutPath: (NSIndex...
Read more