日志统计

通过访问日志来统计服务的被访问情况
 1）httpd/nginx/lighthttp/IIS
 2）FTP
 3）MAIL

1、通过awstats（perl）软件来分析httpd访问日志
# yum -y install httpd

配置两个虚拟主机,每个虚拟主机的日志使用日志格式：combined
# vim /etc/hosts
192.168.0.1             www.abc.com    www.bcd.com

# vim /etc/httpd/conf/httpd.conf
NameVirtualHost *:80
<VirtualHost *:80>
    DocumentRoot /www/www.abc.com
    ServerName www.abc.com
    ErrorLog logs/www.abc.com-error_log
    CustomLog logs/www.abc.com-access_log combined
</VirtualHost>
<VirtualHost *:80>
    DocumentRoot /www/www.bcd.com
    ServerName www.bcd.com
    ErrorLog logs/www.bcd.com-error_log
    CustomLog logs/www.bcd.com-access_log combined
</VirtualHost>

# mkdir -p /www/www.abc.com
# mkdir -p /www/www.bcd.com
# echo www.abc.com > /www/www.abc.com/index.html
# echo www.bcd.com > /www/www.bcd.com/index.html

# elinks -dump 1 http://www.abc.com
   www.abc.com
# elinks -dump 1 http://www.bcd.com
   www.bcd.com

 

2、部署awstats软件来分析访问日志
# wget ftp://192.168.0.254:/notes/softwares/project/awstats-6.9.tar.gz -P /root
# tar xf /root/awstats-6.9.tar.gz -C /usr/local/
# cd /usr/local/
# mv awstats-6.9/ awstats

配置awstat分析apache的访问日志
# cd /usr/local/awstats/tools/
# ./awstats_configure.pl

----- AWStats awstats_configure 1.0 (build 1.8) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix

-----> Check for web server install

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> /etc/httpd/conf/httpd.conf  －－输入你的apache服务器的主配置文件

-----> Check and complete web server config file '/etc/httpd/conf/httpd.conf'
  Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'
  Add 'Alias /awstatscss "/usr/local/awstats/wwwroot/css/"'
  Add 'Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"'
  Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'
  Add '<Directory>' directive
  AWStats directives added to Apache config file.

-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
  File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.abc.com

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
>    －－回车，把配置文件存放至/etc/awstats

-----> Create config file '/etc/awstats/awstats.www.abc.com.conf'
 Config file /etc/awstats/awstats.www.abc.com.conf created.

-----> Restart Web server with '/sbin/service httpd restart'
httpd: apr_sockaddr_info_get() failed for nis.uplooking.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.abc.com
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue...

A SIMPLE config file has been created: /etc/awstats/awstats.www.abc.com.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'www.abc.com' with command:
> perl awstats.pl -update -config=www.abc.com   －－更新网站统计数据
You can also read your statistics for 'www.abc.com' with URL:
> http://localhost/awstats/awstats.pl?config=www.abc.com －－访问统计结果的URL

Press ENTER to finish...

了解和更改awstats生成配置文件
# vim /etc/awstats/awstats.www.abc.com.conf
  51 LogFile="/var/log/httpd/www.abc.com-access_log"  －－告诉awstats，www.abc.com对应访问日志在何处

# mkdir -p /var/lib/awstats  －－存放统计结果

产生对www.abc.com网站的统计结果
# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.abc.com

访问统计结果：
http://192.168.0.1/awstats/awstats.pl?config=www.abc.com

更新对www.abc.com网站的统计结果
# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.abc.com
Create/Update database for config "/etc/awstats/awstats.www.abc.com.conf" by AWStats version 6.9 (build 1.925)
From data in log file "/var/log/httpd/www.abc.com-access_log"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 2)
Jumped lines in file: 2
 Found 2 already parsed records.
Parsed lines in file: 535
 Found 0 dropped records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 535 new qualified records.

再次访问统计结果

结合crontab自动刷新网站的统计结果
# crontab -e
00 08 * * *  /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.abc.com &> /dev
/null

 

二、统计第二个网站www.bcd.com

 

三、结合帮助文档统计vsftpd的上传和下载的日志（统计前需要自己先上传和下载过，这样才有日志）
/var/log/xferlog

1.
分析VSFTPD的上传下载日志:
# yum -y install vsftpd
# vim /etc/vsftpd/vsfptd.conf
xferlog_enable=YES
xferlog_std_format=YES
xferlog_file=/var/log/vsftpd.log

# service vsftpd restart
# cp -r /etc /var/ftp
# lftp ftp.abc.com

 

2.定义awstats:
# vim /etc/awstats/awstats.ftp.abc.com.conf
LogFile="/var/log/vsftpd.log"
LogType=F
LogFormat="%time3 %other %host %bytesd %url %other %other %method %other %logname %other %code %other %other"
LogSeparator="\s"
NotPageList=""
LevelForBrowsersDetection=0
LevelForOSDetection=0
LevelForRefererAnalyze=0
LevelForRobotsDetection=0
LevelForWormsDetection=0
LevelForSearchEnginesDetection=0
ShowLinksOnUrl=0
ShowMenu=1
ShowSummary=UVHB
ShowMonthStats=UVHB
ShowDaysOfMonthStats=HB
ShowDaysOfWeekStats=HB
ShowHoursStats=HB
ShowDomainsStats=HB
ShowHostsStats=HBL
ShowAuthenticatedUsers=HBL
ShowRobotsStats=0
ShowEMailSenders=0
ShowEMailReceivers=0
ShowSessionsStats=1
ShowPagesStats=PBEX
ShowFileTypesStats=HB
ShowFileSizesStats=0
ShowBrowsersStats=0
ShowOSStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=0
SiteDomain="ftp.abc.com"

更新访问日志
# usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=ftp.abc.com

访问结果:
# firefox http://www.abc.com/awstats/awstats.pl?config=ftp.abc.com

设置每天自动更新:
# crontab -e
00 08 * * * usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=ftp.abc.com &> /dev/null