hadoop - flume taking time to copy data into hdfs when rolling based on file size -


i have usecase want copy remote file hdfs using flume. want copied files should align hdfs block size (128mb/256mb).total size of remote data 33gb.

i using avro source , sink copy remote data hdfs. sink side doing file size rolling(128,256).but copying file remote machine , storing hdfs(file size 128/256 mb) flume taking avg of 2 min.

flume configuration: avro source(remote machine)

### agent1 - spooling directory source , file channel, avro sink  ### # name components on agent agent1.sources = spooldir-source   agent1.channels = file-channel agent1.sinks = avro-sink  # describe/configure source agent1.sources.spooldir-source.type = spooldir agent1.sources.spooldir-source.spooldir =/home/benchmarking_simulation/test   # describe sink agent1.sinks.avro-sink.type = avro agent1.sinks.avro-sink.hostname = xx.xx.xx.xx   #ip address destination machine agent1.sinks.avro-sink.port = 50000  #use channel buffers events in file agent1.channels.file-channel.type = file agent1.channels.file-channel.checkpointdir = /home/flume_checkpoint_dir/ agent1.channels.file-channel.datadirs = /home/flume_data_dir/ agent1.channels.file-channel.capacity = 10000000 agent1.channels.file-channel.transactioncapacity=50000  # bind source , sink channel agent1.sources.spooldir-source.channels = file-channel agent1.sinks.avro-sink.channel = file-channel

avro sink(machine hdfs running)

### agent1 - avro source , file channel, avro sink  ### # name components on agent agent1.sources = avro-source1   agent1.channels = file-channel1 agent1.sinks = hdfs-sink1  # describe/configure source agent1.sources.avro-source1.type = avro agent1.sources.avro-source1.bind = xx.xx.xx.xx agent1.sources.avro-source1.port = 50000  # describe sink agent1.sinks.hdfs-sink1.type = hdfs agent1.sinks.hdfs-sink1.hdfs.path =/user/benchmarking_data/multiple_agent_parallel_1 agent1.sinks.hdfs-sink1.hdfs.rollinterval = 0 agent1.sinks.hdfs-sink1.hdfs.rollsize = 130023424 agent1.sinks.hdfs-sink1.hdfs.rollcount = 0 agent1.sinks.hdfs-sink1.hdfs.filetype = datastream agent1.sinks.hdfs-sink1.hdfs.batchsize = 50000 agent1.sinks.hdfs-sink1.hdfs.txneventmax = 40000 agent1.sinks.hdfs-sink1.hdfs.threadspoolsize=1000 agent1.sinks.hdfs-sink1.hdfs.appendtimeout = 10000 agent1.sinks.hdfs-sink1.hdfs.calltimeout = 200000   #use channel buffers events in file agent1.channels.file-channel1.type = file agent1.channels.file-channel1.checkpointdir = /home/flume_check_point_dir agent1.channels.file-channel1.datadirs = /home/flume_data_dir agent1.channels.file-channel1.capacity = 100000000 agent1.channels.file-channel1.transactioncapacity=100000   # bind source , sink channel agent1.sources.avro-source1.channels = file-channel1 agent1.sinks.hdfs-sink1.channel = file-channel1

network connectivity between both machine 686 mbps.

can please me identify whether wrong in configuration or alternate configuration copying doesn't take of time.

both agents use file channel. before writing hdfs, data has been written disk twice. can try use memory channel each agent see if performance improved.


Comments

Post a Comment

Popular posts from this blog

asynchronous - C# WinSCP .NET assembly: How to upload multiple files asynchronously -

i using winscp .net assembly uploading files. want upload multiple files asynchronous way. have created method works single upload. public class uploadingdata { private sessionoptions _sessionoptions; private session _session; //connection etc private void onconnected() { foreach (var fileinfo in localfileslist) { var task = task.factory.startnew(() => uploadfilesasync(fileinfo)); } } private async task uploadfilesasync(string file) { string remotefilepath = _session.translatelocalpathtoremote(file, @"/", "/test_data_upload"); var uploading = _session.putfiles(file, remotefilepath, false); //when done await task.run(() => thread.sleep(1000)); } } please suggest me correct way. thanks you can use backgroundworker class this: string[] images = new string[50]; foreach (var path in images) { backgroundworker
Read more

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

i getting "__type": "com.amazon.coral.service#serializationexception" as reply in postman & in test console in api gateway trying post record directly dynamodb using api proxy services.. referring aws article - https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/ here's mapping { "tablename": "tablenamegoeshere", "item": { "id" : "$context.requestid" "eventname" : "$input.path('$.eventname')", "timestamp" : $input.path('$.timestamp'), "answers": "$util.parsejson($input.path('$.answers'))" } } update: did asked ... , worked if try add array of json objects gives me above same error - here's trying now. please - couldnt find on google well #set($inputroot = $input.path('$')) { "tablename": "answer", "item": {
Read more

asp.net - Problems sending emails from forum -

i have been redesigning site has message board, has been written guy looked after site. it has few problems , written in .asp , uses .mdb files database. you can see whole thing here . feel free post if helps, test thing have set using original files , new database. the basic problem when posts message, supposed email sent them, have posted it. when replies, originator supposed message, replier has been posted. i’d have thought @ point in process there should script/file in postmessage.asp? the postmessage.asp page below. <% option explicit %> <!-- #include virtual="common/adovbs.inc" --> <html> <head> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> <meta name="generator" content="microsoft frontpage 4.0"> <meta name="progid" content="frontpage.editor.document"> <title>new page 2</title> <base target="_self"
Read more