amazon s3 - Is DynamoDB suitable as an S3 Metadata index? -


i store , query large quantity of raw event data. architecture use 'data lake' architecture s3 holds actual event data, , dynamodb used index , provide metadata. architecture talked , recommended in many places:

however, struggling understand how use dynamodb purposes of querying event data in s3. in link aws blog above, use example of storing customer events produced multiple different servers:

s3 path format: [4-digit hash]/[server id]/[year]-[month]-[day]-[hour]-[minute]/[customer id]-[epoch timestamp].data

eg: a5b2/i-31cc02/2015-07-05-00-25/87423-1436055953839.data

and schema record event in dynamodb looks like:

customer id (partition key), timestamp-server (sort key), s3-key, size 87423, 1436055953839-i-31cc02, a5b2/i-31cc02/2015-07-05-00-25/87423-1436055953839.data, 1234

i perform query such as: "get me customer events produced servers in last 24 hours" far understand, it's impossible efficiently query dynamodb without using partition key. cannot specify partition key kind of query.

given requirement, should use database other dynamodb record events in s3? or need use different type of dynamodb schema?

the architecture looks fine , feasible using dynamodb database. dynamodbmapper class (present in aws sdk java) can used create model has useful methods data s3.

dynamodbmapper

gets3clientcache() returns underlying s3clientcache accessing s3.

dynamodb database can't queried without partition key. have scan whole dynamodb database if partition key not available. however, can create global secondary index (gsi) on date/time field , query data use case.

in simple terms, gsi similar index present in rdbms. difference can directly query gsi rather main table. normally, gsi required if query dynamodb use case when partition key not available. there options available include all (or) selective fields present in main table in gsi.

global secondary index (gsi)

difference between scan , query in dynamodb

yes, in use case, looks gsi can't use case requires a range query on partition key. dynamodb supports equality operator. dynamodb supports range queries on sort keys or other non-key attributes if partition key available. may have scan dynamodb fulfill use case costly operation.

either have think alternate data model can query partition key or use other database.


Comments

Post a Comment

Popular posts from this blog

asynchronous - C# WinSCP .NET assembly: How to upload multiple files asynchronously -

i using winscp .net assembly uploading files. want upload multiple files asynchronous way. have created method works single upload. public class uploadingdata { private sessionoptions _sessionoptions; private session _session; //connection etc private void onconnected() { foreach (var fileinfo in localfileslist) { var task = task.factory.startnew(() => uploadfilesasync(fileinfo)); } } private async task uploadfilesasync(string file) { string remotefilepath = _session.translatelocalpathtoremote(file, @"/", "/test_data_upload"); var uploading = _session.putfiles(file, remotefilepath, false); //when done await task.run(() => thread.sleep(1000)); } } please suggest me correct way. thanks you can use backgroundworker class this: string[] images = new string[50]; foreach (var path in images) { backgroundworker
Read more

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

i getting "__type": "com.amazon.coral.service#serializationexception" as reply in postman & in test console in api gateway trying post record directly dynamodb using api proxy services.. referring aws article - https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/ here's mapping { "tablename": "tablenamegoeshere", "item": { "id" : "$context.requestid" "eventname" : "$input.path('$.eventname')", "timestamp" : $input.path('$.timestamp'), "answers": "$util.parsejson($input.path('$.answers'))" } } update: did asked ... , worked if try add array of json objects gives me above same error - here's trying now. please - couldnt find on google well #set($inputroot = $input.path('$')) { "tablename": "answer", "item": {
Read more

asp.net - Problems sending emails from forum -

i have been redesigning site has message board, has been written guy looked after site. it has few problems , written in .asp , uses .mdb files database. you can see whole thing here . feel free post if helps, test thing have set using original files , new database. the basic problem when posts message, supposed email sent them, have posted it. when replies, originator supposed message, replier has been posted. i’d have thought @ point in process there should script/file in postmessage.asp? the postmessage.asp page below. <% option explicit %> <!-- #include virtual="common/adovbs.inc" --> <html> <head> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> <meta name="generator" content="microsoft frontpage 4.0"> <meta name="progid" content="frontpage.editor.document"> <title>new page 2</title> <base target="_self"
Read more