hadoop - Calculating yarn.nodemanager.resource.cpu-vcores for a yarn cluster with multiple spark clients -


if have 3 spark applications using same yarn cluster, how should set

yarn.nodemanager.resource.cpu-vcores

in each of 3 yarn-site.xml?

(each spark application required have it's own yarn-site.xml on classpath)

does value matter in client yarn-site.xml's ?

if does:

let's cluster has 16 cores.

should value in each yarn-site.xml 5 (for total of 15 leave 1 core system processes) ? or should set each 1 15 ?

(note: cloudera indicates 1 core should left system processes here: http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ however, not go details of using multiple clients against same cluster)

assume spark running yarn master, , running in cluster mode.

are talking server-side configuration each yarn node manager? if so, typically configured little less number of cpu cores (or virtual cores if have hyperthreading) on each node in cluster. if have 4 nodes 4 cores each, dedicate example 3 per node yarn node manager , cluster have total of 12 virtual cpus.

then request desired resources when submitting spark job (see http://spark.apache.org/docs/latest/submitting-applications.html example) cluster , yarn attempt fulfill request. if can't fulfilled, spark job (or application) queued or there timeout.

you can configure different resource pools in yarn guarantee specific amount of memory/cpu resources such pool, that's little bit more advanced.

if submit spark application in cluster mode, have consider spark driver run on cluster node , not local machine (that 1 submitted it). therefore require @ least 1 virtual cpu more.

hope clarifies things little you.


Comments

Post a Comment

Popular posts from this blog

sql server - Cannot query correctly (MSSQL - PHP - JSON) -

i have little problem on project. , turns out query result returns nothing when value of first name or last name has 'ñ' in it. here's code config.php: <?php ini_set('mssql.charset', 'utf-8'); header('content-type: text/html; charset=utf-8'); $servername = "192.168.1.21"; /* ip add of db server */ $connection = array("database" => "db", "uid" => "?", "pwd" => "***"); $conn = sqlsrv_connect($servername, $connection); if(!$conn){ echo "connection not established."; die(print_r(sqlsrv_errors(), true)); } ?> myquery: $idnumber = $_post["idnum"]; $response = array(); $query2 = "select clname, cfname, cmname, cqualifier student cidno = '$idnumber'"; try{ $stmt2 = sqlsrv_query($conn, $query2); } catch(pdoexeption $ex){ $response["success"] = 0; $response["message"] = "d...
Read more

php - trouble displaying mysqli database results in correct order -

i have table in database stores events , date expire. want display events on website display current events first expired events after? i have tried following sql $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; but returns expired results first due expirydate being ordered asc in query. my tbl_notice structure looks this noticeid => int(4) notice => varchar(100) noticedesc => text noticedate => timestamp noticeimg => varchar(40) expirydate => datetime expired => int(1) 0 current or 1 expired my php code is $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; $result = mysqli_query($conn,$sql); if ($result->num_rows > 0) { // output data of each row while($row = $result->fetch_assoc()) { ...
Read more

C++ Linked List -

i trying create linked list download txt file , use linked list handle file line line. when handling downloaded linked list operations performed on such text editor would. encountering problems however. seems "node(string value)" section of code has wrong though original node() declaration no arguments passes. unable figure out quite is. node.h class node { public: node(); node(string value); void setnext(node *nextnode); // allows user set "next" pointer of node points friend class linkedlist; private: string data; // data box node* next; // pointer box }; node.cpp # include <string> # include "node.h" using namespace std; node::node() { data = ""; next = null; } node::node(string value) { data = value; next = null; } void node::setnext(node *nextnode) // allows user set "next" pointer of node points { this->next = nextnode; } your #include <string> ...
Read more