python - Save data from Dataproc to Datastore -


i have implemented recommendation engine using python2.7 in google dataproc/ spark, , need store output records in datastore, subsequent use app engine apis. however, there doesn't seem way directly.

there no python datastore connector dataproc far can see. python dataflow sdk doesn't support writing datastore (although java 1 does). mapreduce doesn't have output writer datastore.

that doesn't appear leave many options. @ moment think have write records google cloud storage , have separate task running in app engine harvest them , store in datastore. not ideal- aligning 2 processes has own difficulties.

is there better way data dataproc datastore?

i succeeded in saving datastore records dataproc. involved installing additional components on master vm (ssh console)

the appengine sdk installed , initialised using

sudo apt-get install google-cloud-sdk-app-engine-python sudo gcloud init

this places new google directory under /usr/lib/google-cloud-sdk/platform/google_appengine/.

the datastore library installed via 

sudo apt-get install python-dev sudo apt-get install python-pip sudo pip install -t /usr/lib/google-cloud-sdk/platform/google_appengine/ google-cloud-datastore

for reasons have yet understand, installed @ 1 level lower, i.e. in /usr/lib/google-cloud-sdk/platform/google_appengine/google/google, purposes necessary manually move components 1 level in path.

to enable interpreter find code had add /usr/lib/google-cloud-sdk/platform/google_appengine/ path. usual bash tricks weren't being sustained, ended doing @ start of recommendation engine.

because of large amount of data stored, spent lot of time attempting save via mapreduce. came conclusion many of required services missing on dataproc. instead using multiprocessing pool, achieving acceptable performance


Comments

Post a Comment

Popular posts from this blog

asynchronous - C# WinSCP .NET assembly: How to upload multiple files asynchronously -

i using winscp .net assembly uploading files. want upload multiple files asynchronous way. have created method works single upload. public class uploadingdata { private sessionoptions _sessionoptions; private session _session; //connection etc private void onconnected() { foreach (var fileinfo in localfileslist) { var task = task.factory.startnew(() => uploadfilesasync(fileinfo)); } } private async task uploadfilesasync(string file) { string remotefilepath = _session.translatelocalpathtoremote(file, @"/", "/test_data_upload"); var uploading = _session.putfiles(file, remotefilepath, false); //when done await task.run(() => thread.sleep(1000)); } } please suggest me correct way. thanks you can use backgroundworker class this: string[] images = new string[50]; foreach (var path in images) { backgroundworker
Read more

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

i getting "__type": "com.amazon.coral.service#serializationexception" as reply in postman & in test console in api gateway trying post record directly dynamodb using api proxy services.. referring aws article - https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/ here's mapping { "tablename": "tablenamegoeshere", "item": { "id" : "$context.requestid" "eventname" : "$input.path('$.eventname')", "timestamp" : $input.path('$.timestamp'), "answers": "$util.parsejson($input.path('$.answers'))" } } update: did asked ... , worked if try add array of json objects gives me above same error - here's trying now. please - couldnt find on google well #set($inputroot = $input.path('$')) { "tablename": "answer", "item": {
Read more

asp.net - Problems sending emails from forum -

i have been redesigning site has message board, has been written guy looked after site. it has few problems , written in .asp , uses .mdb files database. you can see whole thing here . feel free post if helps, test thing have set using original files , new database. the basic problem when posts message, supposed email sent them, have posted it. when replies, originator supposed message, replier has been posted. i’d have thought @ point in process there should script/file in postmessage.asp? the postmessage.asp page below. <% option explicit %> <!-- #include virtual="common/adovbs.inc" --> <html> <head> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> <meta name="generator" content="microsoft frontpage 4.0"> <meta name="progid" content="frontpage.editor.document"> <title>new page 2</title> <base target="_self"
Read more