python - Amending a script to turn all the text files in a folder into CSV files -


i have large directory full of thousands of text files. have script can run each file @ once , turn csv file. amending script automatically runs through every text file , turns csv file. script have: 

def split_ln(fname):     print 'processing\t',fname     #imort 2 required modules     import re     import csv     outname=fname.replace(fname.split('.')[-1],'csv') #replace   extension "csv"     #setup output file. maybe give option seperate text files, if desired.     outfile=open(outname,'wb')     writer = csv.writer(outfile)      lnraw=open(fname).read() #read file       workfile=re.sub('                copyright .*?\\r\\n','endofile',lnraw)  #silly hack find end of documents     workfile=workfile.replace('\xef\xbb\xbf\r\n','') #clean crud @ beginning of file     workfile=workfile.split('endofile') #split file list of documents.     workfile=[f f in workfile if len(f.split('\r\n\r\n'))>2] #remove blank rows      #figure out special meta data being reported     meta_list=list(set(re.findall('\\n([a-z][a-z-]*?):',lnraw))) #find them     meta_list=[m m in meta_list if float(lnraw.count(m))/len(workfile)>.20] #keep commonly occuring ones     meta_tuple=('search_row','publication','date','title','edition')     item in meta_list:         meta_tuple=meta_tuple+(item,)     writer.writerow(meta_tuple+('text',))        #begin loop on each file     f in workfile:          #split lines, , clean hard returns @ end of each line. removes blank lines occasional copyright lines           filessplit=[row.replace('\r\n',' ') row in f.split('\r\n\r\n') if len(row)>0 , 'all rights reserved' not in row]         #the id number (from search) first text in first item of list         docid=filessplit[0].lstrip().split(' ')[0]         dateedition=filessplit[2].lstrip()         date=dateedition.split(' ')[0]+' '+dateedition.split(' ')[1]+' '+dateedition.split(' ')[2].replace(',','')         edition= dateedition.replace(date,'').split('                         ')[-1].lstrip()         if 'gmt' in edition or ('day' in edition):             edition=''         title= filessplit[3]         publication=filessplit[1].lstrip()         #extra text , other information         text=''         meta_dict={k : '' k in meta_list}         line in filessplit:             if len(line)>0 , line[:2]!='  ' , line!=line.upper() , len(re.findall('^[a-z][a-z-]*?:',line))==0 , title not in line:                 text=text.lstrip()+' '+line.replace('","','" , "')             else:                 metacheck=re.findall('^([a-z][a-z-]*?):',line)                 if len(metacheck)>0:                     if metacheck[0] in meta_list:                        meta_dict[metacheck[0]]=line.replace(metacheck[0]+': ','')            #output results csv file         meta_tuple=(docid,publication,date,title,edition)         item in meta_list:             meta_tuple=meta_tuple+(meta_dict[item],)         writer.writerow(meta_tuple+(text,))                 #output.write(docid+'\t'+title+'\t'+text+'\n')        print 'wrote\t\t',outname   if __name__ == "__main__":     import sys     try:          flist=sys.argv[1:]     except:         print 'only 1 argument please. can use things *.txt'     else:         fname in flist:             split_ln(fname)         print 'done'


Comments

Post a Comment

Popular posts from this blog

asynchronous - C# WinSCP .NET assembly: How to upload multiple files asynchronously -

i using winscp .net assembly uploading files. want upload multiple files asynchronous way. have created method works single upload. public class uploadingdata { private sessionoptions _sessionoptions; private session _session; //connection etc private void onconnected() { foreach (var fileinfo in localfileslist) { var task = task.factory.startnew(() => uploadfilesasync(fileinfo)); } } private async task uploadfilesasync(string file) { string remotefilepath = _session.translatelocalpathtoremote(file, @"/", "/test_data_upload"); var uploading = _session.putfiles(file, remotefilepath, false); //when done await task.run(() => thread.sleep(1000)); } } please suggest me correct way. thanks you can use backgroundworker class this: string[] images = new string[50]; foreach (var path in images) { backgroundworker
Read more

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

i getting "__type": "com.amazon.coral.service#serializationexception" as reply in postman & in test console in api gateway trying post record directly dynamodb using api proxy services.. referring aws article - https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/ here's mapping { "tablename": "tablenamegoeshere", "item": { "id" : "$context.requestid" "eventname" : "$input.path('$.eventname')", "timestamp" : $input.path('$.timestamp'), "answers": "$util.parsejson($input.path('$.answers'))" } } update: did asked ... , worked if try add array of json objects gives me above same error - here's trying now. please - couldnt find on google well #set($inputroot = $input.path('$')) { "tablename": "answer", "item": {
Read more

asp.net - Problems sending emails from forum -

i have been redesigning site has message board, has been written guy looked after site. it has few problems , written in .asp , uses .mdb files database. you can see whole thing here . feel free post if helps, test thing have set using original files , new database. the basic problem when posts message, supposed email sent them, have posted it. when replies, originator supposed message, replier has been posted. i’d have thought @ point in process there should script/file in postmessage.asp? the postmessage.asp page below. <% option explicit %> <!-- #include virtual="common/adovbs.inc" --> <html> <head> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> <meta name="generator" content="microsoft frontpage 4.0"> <meta name="progid" content="frontpage.editor.document"> <title>new page 2</title> <base target="_self"
Read more