parallel processing - How to do parallelization k-means in R? -


i have large dataset (5000*100) , want use kmeans function find clusters. however, not know how use clusterapply function.

set.seed(88) mydata=rnorm(5000*100) mydata=matrix(data=mydata,nrow = 5000,ncol = 100)  parallel.a=function(i) { kmeans(mydata,3,nstart = i,iter.max = 1000) }   library(parallel) cl.cores <- detectcores()-1 cl <- makecluster(cl.cores) clustersetrngstream(cl,iseed=1234) fit.km = clusterapply(cl,x,fun=parallel.a(500)) stopcluster(cl)

the clusterapply requires 'x' value not know how set. also, difference between clusterapply, parsapply , parlapply? lot.

here's way use clusterapply perform parallel kmeans parallelizing on nstart argument (assuming greater one):

library(parallel) nw <- detectcores() cl <- makecluster(nw) clustersetrngstream(cl, iseed=1234) set.seed(88) mydata <- matrix(rnorm(5000 * 100), nrow=5000, ncol=100)  # parallelize on "nstart" argument nstart <- 100 # create vector of length "nw" sum(nstartv) == nstart nstartv <- rep(ceiling(nstart / nw), nw) results <- clusterapply(cl, nstartv,         function(n, x) kmeans(x, 3, nstart=n, iter.max=1000),         mydata) # pick best result <- sapply(results, function(result) result$tot.withinss) result <- results[[which.min(i)]] print(result$tot.withinss)

people typically export mydata workers, example passes additional argument clusterapply. makes sense (since number of tasks equal number of workers), more efficient (since combines export computation), , avoids creating global variable on cluster workers (which bit more tidy). (of course, exporting makes more sense if plan perform more computations on workers data set.)

note can use detectcores()-1 workers if like, benchmarking on machine shows performs faster detectcores() workers. suggest benchmark on machine see works better you.

as difference between different parallel functions, clusterapply parallel version of lapply processes each value of x in separate task. parlapply parallel version of lapply splits x such sends 1 task per cluster worker (which can more efficient). parsapply calls parlapply simplifies result in same way sapply simplifies result of calling lapply.

clusterapply makes sense parallel kmeans since manually splitting nstart such sends 1 task per cluster worker, making parlapply unnecessary.


Comments

Post a Comment

Popular posts from this blog

sql server - Cannot query correctly (MSSQL - PHP - JSON) -

i have little problem on project. , turns out query result returns nothing when value of first name or last name has 'ñ' in it. here's code config.php: <?php ini_set('mssql.charset', 'utf-8'); header('content-type: text/html; charset=utf-8'); $servername = "192.168.1.21"; /* ip add of db server */ $connection = array("database" => "db", "uid" => "?", "pwd" => "***"); $conn = sqlsrv_connect($servername, $connection); if(!$conn){ echo "connection not established."; die(print_r(sqlsrv_errors(), true)); } ?> myquery: $idnumber = $_post["idnum"]; $response = array(); $query2 = "select clname, cfname, cmname, cqualifier student cidno = '$idnumber'"; try{ $stmt2 = sqlsrv_query($conn, $query2); } catch(pdoexeption $ex){ $response["success"] = 0; $response["message"] = "d...
Read more

php - trouble displaying mysqli database results in correct order -

i have table in database stores events , date expire. want display events on website display current events first expired events after? i have tried following sql $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; but returns expired results first due expirydate being ordered asc in query. my tbl_notice structure looks this noticeid => int(4) notice => varchar(100) noticedesc => text noticedate => timestamp noticeimg => varchar(40) expirydate => datetime expired => int(1) 0 current or 1 expired my php code is $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; $result = mysqli_query($conn,$sql); if ($result->num_rows > 0) { // output data of each row while($row = $result->fetch_assoc()) { ...
Read more

C++ Linked List -

i trying create linked list download txt file , use linked list handle file line line. when handling downloaded linked list operations performed on such text editor would. encountering problems however. seems "node(string value)" section of code has wrong though original node() declaration no arguments passes. unable figure out quite is. node.h class node { public: node(); node(string value); void setnext(node *nextnode); // allows user set "next" pointer of node points friend class linkedlist; private: string data; // data box node* next; // pointer box }; node.cpp # include <string> # include "node.h" using namespace std; node::node() { data = ""; next = null; } node::node(string value) { data = value; next = null; } void node::setnext(node *nextnode) // allows user set "next" pointer of node points { this->next = nextnode; } your #include <string> ...
Read more