apache spark - How to open a file which is stored in HDFS in pySpark using with open -


how open file stored in hdfs - here input file hdfs - if give file bellow , wont able open , show file not found

from pyspark import sparkconf,sparkcontext conf = sparkconf () sc = sparkcontext(conf = conf) def getmoviename():     movienames = {}     open ("/user/sachinkerala6174/indata/moviestat") f:         line in f:             fields = line.split("|")             mid = fields[0]             mname = fields[1]             movienames[int(fields[0])] = fields[1]             return movienames namedict = sc.broadcast(getmoviename())

my assumption use 

with open (sc.textfile("/user/sachinkerala6174/indata/moviestat")) f:

but didnt work

to read textfile rdd:

rdd_name = sc.textfile("/user/sachinkerala6174/indata/moviestat")

you can use "collect()" in order use in pure python (not recommended - use on small data), or use spark rdd methods in order manipulate using pyspark methods (the recommended way)

for more info: http://spark.apache.org/docs/2.0.1/api/python/pyspark.html

textfile(name, minpartitions=none, use_unicode=true)

read text file hdfs, local file system (available on nodes), or hadoop-supported file system uri, , return rdd of strings.

if use_unicode false, strings kept str (encoding utf-8), faster , smaller unicode. (added in spark 1.2)

>>> path = os.path.join(tempdir, "sample-text.txt") >>> open(path, "w") testfile: ...    _ = testfile.write("hello world!") >>> textfile = sc.textfile(path) >>> textfile.collect() [u'hello world!']

Comments

Post a Comment

Popular posts from this blog

php - trouble displaying mysqli database results in correct order -

i have table in database stores events , date expire. want display events on website display current events first expired events after? i have tried following sql $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; but returns expired results first due expirydate being ordered asc in query. my tbl_notice structure looks this noticeid => int(4) notice => varchar(100) noticedesc => text noticedate => timestamp noticeimg => varchar(40) expirydate => datetime expired => int(1) 0 current or 1 expired my php code is $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; $result = mysqli_query($conn,$sql); if ($result->num_rows > 0) { // output data of each row while($row = $result->fetch_assoc()) { ...
Read more

depending on nth recurrence of job in control M -

we have need in control m follows. have 2 daily jobs in control m job , job b. job has recurrence count of 5 per day , job b needs depend on 3rd occurrence of job a. how can achieved in control m configuration? thanks.. use runcount variable below. do below job a do not add out condition in out condition tab in steps tab: on(stmt=*, code=runcount=3) addcondtion specify condition name ex: a-ok for job b do below. specify in condition below. a-ok let me know if u need complete structure
Read more

sql server - Cannot query correctly (MSSQL - PHP - JSON) -

i have little problem on project. , turns out query result returns nothing when value of first name or last name has 'ñ' in it. here's code config.php: <?php ini_set('mssql.charset', 'utf-8'); header('content-type: text/html; charset=utf-8'); $servername = "192.168.1.21"; /* ip add of db server */ $connection = array("database" => "db", "uid" => "?", "pwd" => "***"); $conn = sqlsrv_connect($servername, $connection); if(!$conn){ echo "connection not established."; die(print_r(sqlsrv_errors(), true)); } ?> myquery: $idnumber = $_post["idnum"]; $response = array(); $query2 = "select clname, cfname, cmname, cqualifier student cidno = '$idnumber'"; try{ $stmt2 = sqlsrv_query($conn, $query2); } catch(pdoexeption $ex){ $response["success"] = 0; $response["message"] = "d...
Read more