apache spark - error: not found: value sqlContext on EMR -


i on emr using spark 2. when ssh master node , run spark-shell can't see have access sqlcontext. there i'm missing? 

[hadoop@ip-172-31-13-180 ~]$ spark-shell setting default log level "warn". adjust logging level use sc.setloglevel(newlevel). 16/11/10 21:07:05 warn client: neither spark.yarn.jars nor spark.yarn.archive set, falling uploading libraries under spark_home. 16/11/10 21:07:14 warn sparkcontext: use existing sparkcontext, configuration may not take effect. spark context web ui available @ http://172.31.13.180:4040 spark context available 'sc' (master = yarn, app id = application_1478720853870_0003). spark session available 'spark'. welcome       ____              __      / __/__  ___ _____/ /__     _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 2.0.1       /_/  using scala version 2.11.8 (openjdk 64-bit server vm, java 1.8.0_111) type in expressions have them evaluated. type :help more information.  scala> import org.apache.spark.sql.sqlcontext import org.apache.spark.sql.sqlcontext  scala> sqlcontext <console>:25: error: not found: value sqlcontext        sqlcontext        ^

since i'm getting same error on local computer i've tried following no avail:

exported spark_local_ip

➜  play grep "spark_local_ip" ~/.zshrc export spark_local_ip=127.0.0.1 ➜  play source ~/.zshrc ➜  play spark-shell using spark's default log4j profile: org/apache/spark/log4j-defaults.properties setting default log level "warn". adjust logging level use sc.setloglevel(newlevel). 16/11/10 16:12:18 warn nativecodeloader: unable load native-hadoop library platform... using builtin-java classes applicable 16/11/10 16:12:19 warn sparkcontext: use existing sparkcontext, configuration may not take effect. spark context web ui available @ http://127.0.0.1:4040 spark context available 'sc' (master = local[*], app id = local-1478812339020). spark session available 'spark'. welcome       ____              __      / __/__  ___ _____/ /__     _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 2.0.1       /_/  using scala version 2.11.8 (java hotspot(tm) 64-bit server vm, java 1.7.0_79) type in expressions have them evaluated. type :help more information.  scala> sqlcontext <console>:24: error: not found: value sqlcontext        sqlcontext        ^  scala>

my /etc/hosts contains following

127.0.0.1       localhost 255.255.255.255 broadcasthost ::1             localhost

spark 2.0 doesn't use sqlcontext anymore:

  • use sparksession (initialized in spark-shell spark).

  • for legacy application can:

    val sqlcontext = spark.sqlcontext

Comments

Post a Comment

Popular posts from this blog

sql server - Cannot query correctly (MSSQL - PHP - JSON) -

i have little problem on project. , turns out query result returns nothing when value of first name or last name has 'ñ' in it. here's code config.php: <?php ini_set('mssql.charset', 'utf-8'); header('content-type: text/html; charset=utf-8'); $servername = "192.168.1.21"; /* ip add of db server */ $connection = array("database" => "db", "uid" => "?", "pwd" => "***"); $conn = sqlsrv_connect($servername, $connection); if(!$conn){ echo "connection not established."; die(print_r(sqlsrv_errors(), true)); } ?> myquery: $idnumber = $_post["idnum"]; $response = array(); $query2 = "select clname, cfname, cmname, cqualifier student cidno = '$idnumber'"; try{ $stmt2 = sqlsrv_query($conn, $query2); } catch(pdoexeption $ex){ $response["success"] = 0; $response["message"] = "d...
Read more

php - trouble displaying mysqli database results in correct order -

i have table in database stores events , date expire. want display events on website display current events first expired events after? i have tried following sql $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; but returns expired results first due expirydate being ordered asc in query. my tbl_notice structure looks this noticeid => int(4) notice => varchar(100) noticedesc => text noticedate => timestamp noticeimg => varchar(40) expirydate => datetime expired => int(1) 0 current or 1 expired my php code is $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; $result = mysqli_query($conn,$sql); if ($result->num_rows > 0) { // output data of each row while($row = $result->fetch_assoc()) { ...
Read more

C++ Linked List -

i trying create linked list download txt file , use linked list handle file line line. when handling downloaded linked list operations performed on such text editor would. encountering problems however. seems "node(string value)" section of code has wrong though original node() declaration no arguments passes. unable figure out quite is. node.h class node { public: node(); node(string value); void setnext(node *nextnode); // allows user set "next" pointer of node points friend class linkedlist; private: string data; // data box node* next; // pointer box }; node.cpp # include <string> # include "node.h" using namespace std; node::node() { data = ""; next = null; } node::node(string value) { data = value; next = null; } void node::setnext(node *nextnode) // allows user set "next" pointer of node points { this->next = nextnode; } your #include <string> ...
Read more