.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "LR_ANONYMIZE.IN 1" .TH LR_ANONYMIZE.IN 1 "2020-07-07" "Lire 2.1.1" "LogReport's Lire Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" lr_anonymize \- anonymize a logfile .SH "SYNOPSIS" .IX Header "SYNOPSIS" \&\fBlr_anonymize\fR \fIdumpfilestem\fR .SH "DESCRIPTION" .IX Header "DESCRIPTION" \&\fBlr_anonymize\fR is typically used when sending logs to a responder. See the section on \*(L"Sending Anonymized Log Files To A Responder\*(R" in the chapter on \&\*(L"Using A Responder\*(R" in the Lire User Manual for usage examples. .PP \&\fBlr_anonymize\fR reads a file containing emailaddresses, ipnumbers, and hostnames (typically a logfile from an internet service) from stdin, and prints an \*(L"anonymized\*(R" version of this file to stdout. It dumps the information to \&\*(L"deanonymize\*(R" the file, using \fBlr_deanonymize\fR\|(1), to 5 Berkeley \s-1DB\s0 databases; the names of the files holding these databases is contructed by concatenating the strings 'inaddr', 'maildomain', 'email', 'ip' and 'domain' to \&\fIdumpfilestem\fR. \fIdumpfilestem\fR can be e.g. /tmp/dump, ../../var/dump or dump. The db files created silently overwrite stale ones, if found. .PP The script builds temporary files. By default, these are created in /tmp. One can override this by setting the \s-1TMPDIR\s0 environment variable. .SH "CONNECTION WITH lr_deanonymize" .IX Header "CONNECTION WITH lr_deanonymize" When running .PP .Vb 2 \& $ lr_anonymize dump < log > log.anon \& $ lr_deanonymize dump < log.anon > log.new .Ve .PP then log and log.new have the same content (except for case, check it with \&\fBdiff \-i\fR). .SH "NOTES" .IX Header "NOTES" We tried to optimize this script for memoryusage. This has the drawback the script will run for quite some time when anonymizing a big logfile. .PP We've run the script on a 25 M sendmail logfile. .PP Typical values in such a case are 2500 K for total amount of physical memory used, and 15m real, 8m user and 22s systime spent on a 64 \s-1MB\s0 system with a 300 MHz Pentium \s-1II\s0 processor. de_anonymizing this file took 11m real, 8m user and 17s system time. .PP We store maildomains in the dumpfile. These are used by \fBlr_deanonymize\fR\|(1), in case email addresses in our input file reoccur in the file read by \fBlr_deanonymize\fR\|(1) in split form, i.e. joe.user@example.com occurs as both joe.user@example.com and example.com in the to be deanonymized file. .PP All dumped objects are casted to lowercase. .SH "BACKGROUND" .IX Header "BACKGROUND" For your convenience, we quote a bit of rfc822: .PP .Vb 10 \& SPACE = ; ( 40, 32.) \& CTL = ; ( 177, 127.) \& specials = "(" / ")" / "<" / ">" / "@" ; Must be in quoted\- \& / "," / ";" / ":" / "\e" / <"> ; string, to use \& / "." / "[" / "]" ; within a word. \& atom = 1* \& quoted\-string = <"> *(qtext/quoted\-pair) <">; Regular qtext or \& ; quoted chars. \& word = atom / quoted\-string \& domain\-ref = atom ; symbolic reference \& domain\-literal = "[" *(dtext / quoted\-pair) "]" \& sub\-domain = domain\-ref / domain\-literal \& domain = sub\-domain *("." sub\-domain) \& local\-part = word *("." word) ; uninterpreted \& ; case\-preserved \& addr\-spec = local\-part "@" domain ; global address .Ve .PP and of rfc 2181 .PP .Vb 5 \& The DNS itself places only one restriction on the particular labels \& that can be used to identify resource records. That one restriction \& relates to the length of the label and the full name. The length of \& any one label is limited to between 1 and 63 octets. A full domain \& name is limited to 255 octets (including the separators). .Ve .PP rfc1123 .PP .Vb 3 \& However, a valid host name can never \& have the dotted\-decimal form #.#.#.#, since at least the \& highest\-level component label will be alphabetic. .Ve .PP rfc819 .PP .Vb 9 \& ::= | "." \& ::= |
\& ::= \& ::= | \& ::= | \& ::= | | "\-" \& ::= any one of the 52 alphabetic characters A through Z in upper \& case and a through z in lower case \& ::= any one of the ten digits 0 through 9 .Ve .SH "EXAMPLE" .IX Header "EXAMPLE" A 'logfile' like e.g. .PP .Vb 8 \& blaat fkrf 1.2.3.4.in\-addr.arpa] pietje@bigcompany.com bla 1 2 3 lj;agas;gag \& blaat 1.2.3.4 fkrf 3.2.3.4.in\-addr.arpa] bla 1 www.hotsex.com 2 3 lj;agas;gag \& jan@blaat.frut.com agagag \& blaat fkrf 4.2.3.4.in\-addr.arpa] bla pietje@bigcompany.com www.hotsex.com \& 234.34.2.0 jan@blaat.frut.com 4.2.3.4.in\-addr.arpa1 2 3 lj;agas;gag \& blaat fkrf tweede 3.2.3.4.in\-addr.arpa] bla 1.2.3.4 1 blablabla.com \& 2 mdcc.cx \& 3 lj;agas;gag .Ve .PP wil get anonymized to .PP .Vb 8 \& blaat fkrf 1.0.0.10.in\-addr.arpa] john.doe.1@example.com bla 1 2 3 lj;agas;gag \& blaat 10.0.0.1 fkrf 2.0.0.10.in\-addr.arpa] bla 1 1.example.com 2 3 lj;agas;gag \& john.doe.2@example.com agagag \& blaat fkrf 3.0.0.10.in\-addr.arpa] bla john.doe.1@example.com 1.example.com \& 10.0.0.2 john.doe.2@example.com 3.0.0.10.in\-addr.arpa1 2 3 lj;agas;gag \& blaat fkrf tweede 2.0.0.10.in\-addr.arpa] bla 10.0.0.1 1 2.example.com \& 2 3.example.com \& 3 lj;agas;gag .Ve .PP The dumps will represent something like .PP .Vb 10 \& ip 234.34.2.0 10.0.0.2 \& ip 1.2.3.4 10.0.0.1 \& inaddr 3.2.3.4.in\-addr.arpa 2.0.0.10.in\-addr.arpa \& inaddr 1.2.3.4.in\-addr.arpa 1.0.0.10.in\-addr.arpa \& inaddr 4.2.3.4.in\-addr.arpa 3.0.0.10.in\-addr.arpa \& domain mdcc.cx 3.example.com \& domain blablabla.com 2.example.com \& domain www.hotsex.com 1.example.com \& email jan@blaat.frut.com john.doe.2@example.com \& email pietje@bigcompany.com john.doe.1@example.com .Ve .SH "BUGS" .IX Header "BUGS" We can't handle files containing hostnames or email addresses in the example.com domain, usernames of the form john.doe. or ipnumbers in the rfc 1918 private network 10.0.0.0 \- 10.255.255.255 (10/8 prefix). .PP We don't handle quoted-string's in email addresses. We don't handle domain-literals in email addresses' domain. .PP We regard 999.999.999.999 as an \s-1IP\s0 address: we don't mind the 255 limit. .PP We don't treat network ipaddresses like 100.10.3 as ipaddresses. These will \fInot\fR get anonymized. .SH "SEE ALSO" .IX Header "SEE ALSO" \&\fBlr_deanonymize\fR\|(1), the \s-1README\s0.lire\-client file in the Lire distribution. .SH "VERSION" .IX Header "VERSION" \&\f(CW$Id:\fR lr_anonymize.in,v 1.5 2006/07/23 13:16:32 vanbaal Exp $ .SH "COPYRIGHT" .IX Header "COPYRIGHT" Copyright (C) 2000\-2001 Stichting LogReport Foundation LogReport@LogReport.org .PP This program is free software; you can redistribute it and/or modify it under the terms of the \s-1GNU\s0 General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. .PP This program is distributed in the hope that it will be useful, but \s-1WITHOUT ANY WARRANTY\s0; without even the implied warranty of \&\s-1MERCHANTABILITY\s0 or \s-1FITNESS FOR A PARTICULAR PURPOSE.\s0 See the \&\s-1GNU\s0 General Public License for more details. .PP You should have received a copy of the \s-1GNU\s0 General Public License along with this program (see \s-1COPYING\s0); if not, check with http://www.gnu.org/copyleft/gpl.html. .SH "AUTHOR" .IX Header "AUTHOR" Joost van Baal