awk - Find duplicate records with only text case difference -


i have log file 8m entries/records urls. i'd find duplicate urls (same urls) difference being type / text case.

example:

origin-www.example.com/this/is/hard.html origin-www.example.com/this/is/hard.html origin-www.example.com/this/is/hard.html

in case, there 3 duplicates case sensitivity.

output should count -c , new file duplicates.

use typical awk '!seen[$0]++' file trick combined tolower() or toupper() make lines in same case:

$ awk '!seen[tolower($0)]++' file origin-www.example.com/this/is/hard.html

for different output , counters whatsoever, provide valid desired output.


Comments

Post a Comment

Popular posts from this blog

php - trouble displaying mysqli database results in correct order -

i have table in database stores events , date expire. want display events on website display current events first expired events after? i have tried following sql $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; but returns expired results first due expirydate being ordered asc in query. my tbl_notice structure looks this noticeid => int(4) notice => varchar(100) noticedesc => text noticedate => timestamp noticeimg => varchar(40) expirydate => datetime expired => int(1) 0 current or 1 expired my php code is $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; $result = mysqli_query($conn,$sql); if ($result->num_rows > 0) { // output data of each row while($row = $result->fetch_assoc()) { ...
Read more

sql server - Cannot query correctly (MSSQL - PHP - JSON) -

i have little problem on project. , turns out query result returns nothing when value of first name or last name has 'ñ' in it. here's code config.php: <?php ini_set('mssql.charset', 'utf-8'); header('content-type: text/html; charset=utf-8'); $servername = "192.168.1.21"; /* ip add of db server */ $connection = array("database" => "db", "uid" => "?", "pwd" => "***"); $conn = sqlsrv_connect($servername, $connection); if(!$conn){ echo "connection not established."; die(print_r(sqlsrv_errors(), true)); } ?> myquery: $idnumber = $_post["idnum"]; $response = array(); $query2 = "select clname, cfname, cmname, cqualifier student cidno = '$idnumber'"; try{ $stmt2 = sqlsrv_query($conn, $query2); } catch(pdoexeption $ex){ $response["success"] = 0; $response["message"] = "d...
Read more

depending on nth recurrence of job in control M -

we have need in control m follows. have 2 daily jobs in control m job , job b. job has recurrence count of 5 per day , job b needs depend on 3rd occurrence of job a. how can achieved in control m configuration? thanks.. use runcount variable below. do below job a do not add out condition in out condition tab in steps tab: on(stmt=*, code=runcount=3) addcondtion specify condition name ex: a-ok for job b do below. specify in condition below. a-ok let me know if u need complete structure
Read more