Webalizer
Encyclopedia
The Webalizer is a GPL
application that generates web pages of analysis
, from access and usage logs, i.e. it is web log analysis software
. It is one of the most commonly used web server administration tools. It was initiated by Bradford L. Barrett in 1997. Statistics commonly reported by Webalizer include: hits; visits; referrers; the visitors' countries; and the amount of data downloaded. These statistics can be viewed graphically and presented by different time frames, such as per day, hour, or month.
In order to illustrate the difference between hits, pages and files, let's consider a user requesting an HTML file referring to five images, one of which is missing. In this case the web server will log six hits (i.e. one successful for the HTML file itself and four for successfully retrieved images and one for the missing image), five files (i.e. five successful HTML requests) and one page (i.e. the HTML file).
Web servers log HTTP traffic using different file formats. Most popular file formats are CLF
, the Apache Custom Log Format and W3C Extended Log File Format. CLF is the less informative log format and should be used only when there are no other options. An example of a CLF log line is shown below.
Apache Custom Log Format can be customized to log most HTTP parameters, including request processing time and the size of the request itself. The format of a custom log is controlled by the format line. A typical Apache log format configuration is shown below.
Microsoft's Internet Information Services (IIS) web server logs HTTP traffic in W3C Extended Log File Format. Similarly to Apache Custom Log format, IIS logs may be configured to capture such extended parameters as request processing time. W3C extended logs may be recognized by the presence of one or more format lines, such as the one shown below.
The original version of The Webalizer can process CLF log files, as well as HTTP proxy log files produced by Squid servers. Other log file formats are usually converted to CLF in order to be analyzed. Some of the forks listed in the External Links section below are capable of processing IIS and Apache log files without having to convert them to CLF first.
shell prompt. A typical command is shown below.
This command instructs The Webalizer to analyze the log file access_log, run in the incremental mode (-p), interpret the log as a CLF log file (-F), use the domain name en.wikipedia.org for report links (-n) and produce the output subdirectory of the current directory.
Use the -h option to see the complete list of command line options.
For example, if the webmaster would like to ignore all requests made from a particular group of hosts, he or she can use the IgnoreSite parameter to discard all log records with the IP address matching the specified pattern:
There are over one hundred available configuration parameters, which make The Webalizer a highly-configurable web traffic analysis application. For a complete list of configuration parameters please refer to the README file shipped with every source or binary distribution.
The yearly summary report provides such information as the number of hits, file and page requests, hosts and visits, as well as daily averages of these counters for each month. The report is accompanied by a yearly summary graph.
Each of the monthly reports is generated as a single HTML page containing a monthly summary report (listing the overall number of hits, file and page requests, visits, hosts, etc.), a daily report (grouping these counters for each of the days of the month), an aggregated hourly report (grouping counters for the same hour of each day together), a URL report (grouping collected information by URL), a host report (by IP address), website entry and exit URL reports (showing most common first and last visit URLs), a referrer report (grouping the referring third-party URLs leading to the analyzed website), a search string report (grouping items by search terms used in such search engines as Google), a user agent report (grouping by the browser type) and a country report (grouping by the host's country of origin).
Each of the standard HTML reports described above lists only top entries for each item (e.g. top 20 URLs). The actual number of lines for each of the reports is controlled by configuration. The Webalizer may also be configured to produce a separate report for each of the items, which will list every single item, such as all website visitors, all requested URLs, etc.
In addition to HTML reports, The Webalizer may be configured to produce comma-delimited dump files, which list all of the report data in a plain-text file. Dump files may be imported to spreadsheet applications or databases for further analysis.
To generate reports in an alternate language requires a separate webalizer binary compiled specifically for that language.
GNU General Public License
The GNU General Public License is the most widely used free software license, originally written by Richard Stallman for the GNU Project....
application that generates web pages of analysis
Web analytics
Web analytics is the measurement, collection, analysis and reporting of internet data for purposes of understanding and optimizing web usage....
, from access and usage logs, i.e. it is web log analysis software
Web log analysis software
Web log analysis software is a simple kind of Web analytics software that parses a log file from a web server, and based on the values contained in the log file, derives indicators about who, when, and how a web server is visited...
. It is one of the most commonly used web server administration tools. It was initiated by Bradford L. Barrett in 1997. Statistics commonly reported by Webalizer include: hits; visits; referrers; the visitors' countries; and the amount of data downloaded. These statistics can be viewed graphically and presented by different time frames, such as per day, hour, or month.
Overview
Website traffic analysis is produced by grouping and aggregating various data items captured by the web server in the form of log files while the website visitor is browsing the website. Some of the most commonly-used website traffic analysis terms are listed below:- URL
A Uniform Resource Locator (URLUniform Resource LocatorIn computing, a uniform resource locator or universal resource locator is a specific character string that constitutes a reference to an Internet resource....
) uniquely identifies the resource requested by the user's browser.
- Hit
Each HTTP request submitted by the browser is counted as one hit. Note that HTTP requests may be submitted for non-existent content, in which case they still will be counted. For example, if one of the five image files referred by the example page mentioned above is missing, the web server will still count six HTTP requests, but in this case, five will be marked as successful (one HTMLHTMLHyperText Markup Language is the predominant markup language for web pages. HTML elements are the basic building-blocks of webpages....
file and four images) and one as a failed request (the missing image)
- Page
A page is a successful HTTP request for a resource that constitutes primary website's content. Pages are usually identified by a file extension (e.g. .html, .php, .asp, etc.) or by a missing extension, in which case the subject of the HTTP request is considered a directory and the default page for this directory is served.
- File
Each successful HTTP request is counted as a file.
- Visitor
A visitor is the actual person browsing the website. A typical website serves content to anonymous visitors and cannot associate visitors with the actual person browsing the website. Visitor identification may be based on their IPInternet ProtocolThe Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
address or an HTTP cookieHTTP cookieA cookie, also known as an HTTP cookie, web cookie, or browser cookie, is used for an origin website to send state information to a user's browser and for the browser to return the state information to the origin site...
. The former approach is simple to implement, but results in all visitors browsing the same website from behind a firewall counted as a single visitor. The latter approach requires special configuration of the web server (i.e. to log HTTP cookies) and is more expensive to implement. Note that neither of the approaches identifies the actual person browsing the website and neither provides 100% accuracy in determining that the same visitor has visited the website again.
- Visit
A visit is a series of HTTP requests submitted by a visitor with the maximum time between requests not exceeding a certain amount configured by the webmaster, which is typically set at 30 minutes. For example, if a visitor requested page A, then in 10 minutes page B and then in 40 minutes page C, then this visitor has generated two visits, one when pages A and B were requested and another when the page C was requested.
- Host
In general, a host is the visitor's machine running the browser. Hosts are often identified by IP addresses or domain names. Those web traffic analysis tools that use IP addresses to identify visitors use the words hosts, domain names and IP addresses interchangeably.
- User Agent
User agent is a synonym for a web browser.
In order to illustrate the difference between hits, pages and files, let's consider a user requesting an HTML file referring to five images, one of which is missing. In this case the web server will log six hits (i.e. one successful for the HTML file itself and four for successfully retrieved images and one for the missing image), five files (i.e. five successful HTML requests) and one page (i.e. the HTML file).
Log file types
The Webalizer analyzes web server log files, extracting such items as client's IP addresses, URL paths, processing times, user agents, referrers, etc. and grouping them in order to produce HTML reports.Web servers log HTTP traffic using different file formats. Most popular file formats are CLF
Common Log Format
The Common Log Format is a standardised text file format used by web servers when generating log files. Because the format is standardised, the files may be analysed by a variety of analysis programs....
, the Apache Custom Log Format and W3C Extended Log File Format. CLF is the less informative log format and should be used only when there are no other options. An example of a CLF log line is shown below.
192.168.1.20 - - [26/Dec/2006:03:09:16 -0500] "GET HTTP/ 1.1" 200 1774
Apache Custom Log Format can be customized to log most HTTP parameters, including request processing time and the size of the request itself. The format of a custom log is controlled by the format line. A typical Apache log format configuration is shown below.
LogFormat "%a %l \"%u\" %t %m \"%U\" \"%q\" %p %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" my_custom_log
CustomLog logs/access_log my_custom_logMicrosoft's Internet Information Services (IIS) web server logs HTTP traffic in W3C Extended Log File Format. Similarly to Apache Custom Log format, IIS logs may be configured to capture such extended parameters as request processing time. W3C extended logs may be recognized by the presence of one or more format lines, such as the one shown below.
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-bytes cs-bytes time-taken
The original version of The Webalizer can process CLF log files, as well as HTTP proxy log files produced by Squid servers. Other log file formats are usually converted to CLF in order to be analyzed. Some of the forks listed in the External Links section below are capable of processing IIS and Apache log files without having to convert them to CLF first.
Command line
The Webalizer is a command line application and is launched from the OSOperating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
shell prompt. A typical command is shown below.
webalizer -p -F clf -n en.wikipedia.org -o reports logfiles/access_log
This command instructs The Webalizer to analyze the log file access_log, run in the incremental mode (-p), interpret the log as a CLF log file (-F), use the domain name en.wikipedia.org for report links (-n) and produce the output subdirectory of the current directory.
Use the -h option to see the complete list of command line options.
Configuration
Besides the command line options, The Webalizer may be configured through parameters of a configuration file. By default, The Webalizer reads the file webalizer.conf and interprets each line as a processing instruction. Alternatively, a user-specified file may be provided using the -c option.For example, if the webmaster would like to ignore all requests made from a particular group of hosts, he or she can use the IgnoreSite parameter to discard all log records with the IP address matching the specified pattern:
IgnoreSite 192.168.0.*
There are over one hundred available configuration parameters, which make The Webalizer a highly-configurable web traffic analysis application. For a complete list of configuration parameters please refer to the README file shipped with every source or binary distribution.
Reports
By default, The Webalizer produces two kinds of reports - a yearly summary report and a detailed monthly report, one for each analyzed month.The yearly summary report provides such information as the number of hits, file and page requests, hosts and visits, as well as daily averages of these counters for each month. The report is accompanied by a yearly summary graph.
Each of the monthly reports is generated as a single HTML page containing a monthly summary report (listing the overall number of hits, file and page requests, visits, hosts, etc.), a daily report (grouping these counters for each of the days of the month), an aggregated hourly report (grouping counters for the same hour of each day together), a URL report (grouping collected information by URL), a host report (by IP address), website entry and exit URL reports (showing most common first and last visit URLs), a referrer report (grouping the referring third-party URLs leading to the analyzed website), a search string report (grouping items by search terms used in such search engines as Google), a user agent report (grouping by the browser type) and a country report (grouping by the host's country of origin).
Each of the standard HTML reports described above lists only top entries for each item (e.g. top 20 URLs). The actual number of lines for each of the reports is controlled by configuration. The Webalizer may also be configured to produce a separate report for each of the items, which will list every single item, such as all website visitors, all requested URLs, etc.
In addition to HTML reports, The Webalizer may be configured to produce comma-delimited dump files, which list all of the report data in a plain-text file. Dump files may be imported to spreadsheet applications or databases for further analysis.
Internationalization
HTML reports may be produced reports in over 30 languages, including Catalan, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, German, Greek, Hungarian, Icelandic, Indonesian, Italian, Japanese, Korean, Latvian, Malay, Norwegian, Polish, Portuguese, Portuguese (Brazil), Romanian, Russian, Serbian, Simplified Chinese, Slovak, Slovene, Spanish, Swedish, Turkish, Ukrainian.To generate reports in an alternate language requires a separate webalizer binary compiled specifically for that language.
Criticism
- Generated statistics do not differentiate between human visitors and robots. As a result all reported metrics are higher than those due to people alone. Many webmasters claim that webalizer produces highly unrealistic figures of visits, which are sometimes 200 to 900% higher than the data produced by javascript based web statistics such as Google AnalyticsGoogle AnalyticsGoogle Analytics is a free service offered by Google that generates detailed statistics about the visitors to a website. The product is aimed at marketers as opposed to webmasters and technologists from which the industry of web analytics originally grew. It is the most widely used website...
or StatCounterStatCounterStatCounter is a web traffic analysis tool. Access to basic services is free and advanced services can cost between $9 and $119 a month. The company is based in Dublin. The statistics from StatCounter are used to compute web usage share for example....
. - Reported hits are too high for download managers with segmented downloads, each 206 "Partial Content" is reported as one hit
- No query string analysis. Dynamic generated websites can not be listed separately (e.g. PHP pages with arguments)
External links
- Webalizer
- Webalizer Quick Help (Provides definitions for the statistics.)
- Simpletons Guide to Web Server Analysis ("This guide is intended to provide the necessary background and insight to how web server analysis works")
- Webalizer User Guide (A non-technical guide aimed at marketers that explains how to interpret webalizer reports.)
- IPligence Community Edition (Free ip-location database for Webalizer.)
- Forks
- AWFFull (Segmenting, Greater than 12 months display, CSS Customisable, Auto-identify log formats)
- Webalizer Xtended (with additional features as e.g. HTTP-404-error statistics, user definable colors for all statistics and several security bugfixes)
- Stone Steps Webalizer (dynamic languages, more log formats and reports)
- Bennington College Webalizer Classic Webalizer that adds detailed hit pages to several of the reports.
- Geolizer (MaxMind GeoIP-based geographic statistics, country flag pictures, human-readable transfer size display and MinGWMinGWMinGW , formerly mingw32, is a native software port of the GNU Compiler Collection and GNU Binutils for use in the development of native Microsoft Windows applications; MinGW can function either as a cross compiler targeting Windows or as a native toolchain run on Windows itself...
/MSYS WindowsMicrosoft WindowsMicrosoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
port) - Webdruid (The Webalizer brother.)
- Webalizer Streaming Streaming media version of Webalizer