How can I see the underlying Hadoop file system from Spark -


i have started spark this:

spark-shell --master local[10]

i'm trying see files on underlying hadoop installation.

i want this:

hdfs ls

how can it?

if understand question correctly want execute hdfs commands shell. in opinion running spark job may not help.

you need start hdfs instance first. below commands documentation. once hdfs started can run shell commands.

to start hadoop cluster need start both hdfs , yarn cluster.

the first time bring hdfs, must formatted. format new distributed filesystem hdfs:

[hdfs]$ $hadoop_prefix/bin/hdfs namenode -format start hdfs namenode following command on designated node hdfs:

[hdfs]$ $hadoop_prefix/sbin/hadoop-daemon.sh --config $hadoop_conf_dir --script hdfs start namenode start hdfs datanode following command on each designated node hdfs:

[hdfs]$ $hadoop_prefix/sbin/hadoop-daemons.sh --config $hadoop_conf_dir --script hdfs start datanode if etc/hadoop/slaves , ssh trusted access configured (see single node setup), of hdfs processes can started utility script. hdfs:

[hdfs]$ $hadoop_prefix/sbin/start-dfs.sh start yarn following command, run on designated resourcemanager yarn:

[yarn]$ $hadoop_yarn_home/sbin/yarn-daemon.sh --config $hadoop_conf_dir start resourcemanager run script start nodemanager on each designated host yarn:

[yarn]$ $hadoop_yarn_home/sbin/yarn-daemons.sh --config $hadoop_conf_dir start nodemanager start standalone webappproxy server. run on webappproxy server yarn. if multiple servers used load balancing should run on each of them:

[yarn]$ $hadoop_yarn_home/sbin/yarn-daemon.sh --config $hadoop_conf_dir start proxyserver if etc/hadoop/slaves , ssh trusted access configured (see single node setup), of yarn processes can started utility script. yarn:

[yarn]$ $hadoop_prefix/sbin/start-yarn.sh start mapreduce jobhistory server following command, run on designated server mapred:

[mapred]$ $hadoop_prefix/sbin/mr-jobhistory-daemon.sh --config $hadoop_conf_dir start historyserver

second option programmatic way. can use filesystem class hadoop (it java implementation.) , hdfs operations.

below link javadoc.

https://hadoop.apache.org/docs/r2.7.3/api/org/apache/hadoop/fs/filesystem.html


Comments

Post a Comment

Popular posts from this blog

asynchronous - C# WinSCP .NET assembly: How to upload multiple files asynchronously -

i using winscp .net assembly uploading files. want upload multiple files asynchronous way. have created method works single upload. public class uploadingdata { private sessionoptions _sessionoptions; private session _session; //connection etc private void onconnected() { foreach (var fileinfo in localfileslist) { var task = task.factory.startnew(() => uploadfilesasync(fileinfo)); } } private async task uploadfilesasync(string file) { string remotefilepath = _session.translatelocalpathtoremote(file, @"/", "/test_data_upload"); var uploading = _session.putfiles(file, remotefilepath, false); //when done await task.run(() => thread.sleep(1000)); } } please suggest me correct way. thanks you can use backgroundworker class this: string[] images = new string[50]; foreach (var path in images) { backgroundworker
Read more

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

i getting "__type": "com.amazon.coral.service#serializationexception" as reply in postman & in test console in api gateway trying post record directly dynamodb using api proxy services.. referring aws article - https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/ here's mapping { "tablename": "tablenamegoeshere", "item": { "id" : "$context.requestid" "eventname" : "$input.path('$.eventname')", "timestamp" : $input.path('$.timestamp'), "answers": "$util.parsejson($input.path('$.answers'))" } } update: did asked ... , worked if try add array of json objects gives me above same error - here's trying now. please - couldnt find on google well #set($inputroot = $input.path('$')) { "tablename": "answer", "item": {
Read more

asp.net - Problems sending emails from forum -

i have been redesigning site has message board, has been written guy looked after site. it has few problems , written in .asp , uses .mdb files database. you can see whole thing here . feel free post if helps, test thing have set using original files , new database. the basic problem when posts message, supposed email sent them, have posted it. when replies, originator supposed message, replier has been posted. i’d have thought @ point in process there should script/file in postmessage.asp? the postmessage.asp page below. <% option explicit %> <!-- #include virtual="common/adovbs.inc" --> <html> <head> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> <meta name="generator" content="microsoft frontpage 4.0"> <meta name="progid" content="frontpage.editor.document"> <title>new page 2</title> <base target="_self"
Read more