A web server log is a text document that contains a record of all activity related to a specific web server over a defined period of time. The web server gathers data automatically and constantly to provide administrators with insight into how and when a server is used, as well as the users that correspond with that activity.
While server data is available immediately within the server log itself, in most cases the log file is also stored in a database and can be used to produce customized reports on demand. This information can be used by administrators to better understand and accommodate web traffic patterns, better allocate IT resources, and adapt sales and marketing activity.
Standard log file format
Most web servers generate a log file in the Common Log Format (CLF) for every HTTP request. Each log entry contains detailed information about the request, including: the IP address of the device; the date and time of the request; the name and location of the requested file; and the file size.
Since CLF files are raw log files, they tend to be of limited use to administrators or webmasters in their existing form. Further, given the volume of data captured, most organizations automatically delete CLF files after a given time.
Server Log Content and Values
Each line within the server log file contains significant information, including:
- The device's IP address
- Request method
- Date and time of the request
- Status of the request
- Referrer method
- User-Agent
- Requested file information, including file name, size and network location
The server log data can also be used to produce the following insights:
- Number of users who accessed the server
- Number of unique or authenticated visitors
- Location of visitors
- Peak traffic periods
- Visit duration
- Page views per visit
- Top-viewed pages or content
- HTTP referrers
- HTTP status
- HTTP errors
- Search terms or phrases used to find the site
- Whether or not they are using a mobile device
Why do you need server logs?
Web Server logs provide an overview of all activity associated with the web server. For most organizations these logs are the only way to understand how and when the server is used and by whom.
What can you do with a server log
Information within the web server log can be leveraged throughout the business to enhance performance and optimize operations. For example, this information can be used to:
- Optimize limited IT resources, including staff
- Establish dedicated logging levels and prioritize activity based on impact to the business or severity of the issue
- Address and debug HTTP errors
- Identify and fix broken links from external sources
- Streamline the user journey based on typical navigation patterns
- Adapt other business activity, such as sales, marketing or partner outreach
- Identify security risks and issues, including the presence of bots, malicious code or spam
Additional web server log functions
Most organizations use web server logs to produce other file logs that serve a distinct purpose. These may include:
- Error logs
- Access logs
- Referrer logs
Error log: One of the most common server logs is an error log. As the name implies, an error log tracks all failed requests for the server. Web administrators can use this information to review standard error information and determine what aspects of their website need to be updated or changed.
Access log: An access log gathers data related to the files requested from the server. This log will reveal the number of users who accessed the server, how they were directed to the site, and their activity on the site. An access log helps organizations understand how the site is being used and the most popular or useful aspects of the site, which can in turn be used to improve or evolve the user journey, site navigation, or content.
Referrer log: A referrer log collects information about the URLs that direct users to your site. A referrer log is a vital aspect of most modern sales and marketing initiatives, in that most businesses want to enhance natural traffic. The referrer log is one way to determine which affiliate links or partners are responsible for drawing people into the site.
How can you monitor your web server logs?
While there are seemingly infinite insights to be gained from server logs, there are a few core challenges that prevent organizations from unlocking the value within log data.
Challenge #1: Volume
Server logs produce a significant amount of data that must be collected, stored and analyzed in order to produce timely and relevant insights.
Challenge #2: Integration
Web Server log data can be very useful to the organization, but additional value can be gleaned when integrated with other file logs, such as event logs, application logs, system logs, availability logs, and resource logs.
Challenge #3: Standardization
Unfortunately, not all log files follow a uniform format. Depending on the type of log, the data may be structured, semi-structured or unstructured. In order to perform effective log file analysis, the data requires a level of normalization to make it easily parsable.
Challenge #4: High IT Burden
When done manually, log management is incredibly time consuming and expensive. Digital log management tools help to automate some of these activities and alleviate the strain on IT professionals.
Discover the world’s leading AI-native platform for next-gen SIEM and log management
Elevate your cybersecurity with the CrowdStrike Falcon® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.
Arfan Sharif is a product marketing lead for the Observability portfolio at CrowdStrike. He has over 15 years experience driving Log Management, ITOps, Observability, Security and CX solutions for companies such as Splunk, Genesys and Quest Software. Arfan graduated in Computer Science at Bucks and Chilterns University and has a career spanning across Product Marketing and Sales Engineering.
