October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsPC HealthRecommendedCrashes, freezes, slowdowns? Check your PC nowSpot repairable issues before they interrupt work.Check PCOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
VGSources
Blog

How to Collect NGINX and Apache Logs With Logstash

A format-aware guide to collecting NGINX and Apache logs with Logstash, including paths, Grok parsing, ECS fields, shipping choices, rotation, and failure diagnosis.
Length10 min Posted Quest giverVGSources Team
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The reliable pattern is to collect access and error logs as separate streams, identify the real paths on each server, parse each deployed format with the appropriate Grok pattern, normalize timestamps, and only then send events to Elasticsearch or another supported destination. Use Logstash’s file input when it can read local files; use Filebeat or syslog when another component already ships the logs.

NGINX and Apache do not produce interchangeable records. Apache’s standard combined access format can usually be parsed with %{COMBINEDAPACHELOG}, while NGINX access fields depend on the active log_format. Error logs from either server need their own parser. The configuration below keeps the original message, tags failures, handles file state deliberately, and leaves enough metadata to troubleshoot production pipelines.

As an Amazon Associate I earn from qualifying purchases.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Choose the collection design before writing filters

There are three practical ways to get web-server logs into Logstash. The right choice depends on where Logstash runs, how much buffering you need, and who owns parsing.

Method Use it when Advantages Important limitations
Logstash file input Logstash runs on the web host or has dependable read access to its files. Simple topology; Logstash reads the files directly and tracks its position. Requires filesystem access, careful permissions, and correct rotation handling on every host.
Filebeat to Logstash A lightweight shipper is already installed, or logs must leave the web host without running a full Logstash instance there. Separates collection from parsing, provides local buffering, and is suited to many hosts. You must coordinate Filebeat inputs, tags or fields, Logstash filters, and queue/back-pressure behavior.
Syslog input An existing syslog forwarder already sends the web-server records. Works with established TCP or UDP forwarding and centralizes network ingestion. Transport metadata can obscure the original format; UDP can lose records, and custom messages need a timestamp-bearing Grok pattern.

Whichever method you select, preserve the original line in message, attach source metadata, and keep access and error events distinguishable. This makes a malformed line diagnosable instead of silently discarded.

#1 Best Overall
Sale
Gaoye Blue Light Blocking Glasses
  • BLUE LIGHT BLOCKING GLASSES: Our blue light glasses for women boast UV400 protection, blocking harmful blue rays from computer screens. With these bluelight glasses women, you can enjoy gaming, reading, or watching TV with no worries of eye strain.
  • EYE RELIEF: Our blue light glasses are engineered to reduce blurry vision or strain caused by prolonged screen time. These womens blue light glasses aid in better sleep by preventing blue light from disrupting your natural sleep cycle, helping you fall asleep faster and enjoy a more restful night's sleep, making them the perfect christmas present.
  • COMFORTABLE: Crafted from durable material, our trendy blue light glasses for woman or blue light glasses men ensure long-lasting comfort. The high-transmittance lenses accessories, gamer gifts for men, help keep your eyes relaxed. Perfect stocking stuffers, gifts for gamers men, christmas gifts for men, halloween costumes.
  • LIGHTWEIGHT: These clear blue light glasses feature a lightweight frame, meaning they’ll stay comfy for long-term wear. Perfect as blue blockers glasses for women or mens blue light glasses, these non-prescription fake glasses for men can be worn throughout the day. The blue light blocking glasses women feature polycarbonate HD lenses, which are anti-reflective and restore true color.
  • WORRY-FREE PURCHASE: If you experience any issues, our support team is ready to assist you quickly. These are the perfect costume glasses for cosplay dressing up.

Find the actual NGINX and Apache log files

Do not assume a copied example path is correct. Package layouts, operating systems, containers, virtual hosts, and custom configurations change locations.

NGINX

Inspect the running configuration and each included file for access_log, error_log, and log_format. The access_log directive determines the access destination and format; error_log determines the error destination and severity threshold. A host can have several access files for different virtual servers and more than one error file.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Apache

Check the active virtual-host configuration for CustomLog and ErrorLog. Distribution packages commonly place files under a directory such as /var/log/apache2 or /var/log/httpd, but the directive in the running configuration is authoritative. Confirm that the Logstash service account can traverse the directory and read rotated files when required.

Build a path inventory

  1. Record every active access and error destination, including per-site files.
  2. Record the exact format name or literal format string for each access destination.
  3. Identify rotation behavior, compression, truncation, and retention rules.
  4. Choose narrow globs that include only intended logs. For example, /var/log/nginx/*.log and /var/log/apache2/*_log are safer starting points than a recursive glob covering unrelated application files.
  5. Copy one current access line and one current error line from every distinct format for testing.

Start with a local-file pipeline

The following skeleton demonstrates conditional routing. It uses valid field names, keeps a debugging output, and deliberately leaves NGINX parsing format-specific. Replace the Elasticsearch host with your own supported destination and add authentication according to that deployment.

input {
  file {
    path => ["/var/log/nginx/*.log", "/var/log/apache2/*_log"]
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/plugins/inputs/file/sincedb-web"
    tags => ["web-file"]
  }
}

filter {
  if [path] =~ /apache/ and [path] =~ /access/ {
    mutate {
      add_field => {
        "[service][name]" => "apache"
        "[log][type]" => "access"
      }
    }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      tag_on_failure => ["_apache_access_grok_failure"]
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      tag_on_failure => ["_apache_access_date_failure"]
    }
  } else if [path] =~ /nginx/ and [path] =~ /access/ {
    mutate {
      add_field => {
        "[service][name]" => "nginx"
        "[log][type]" => "access"
      }
    }
    # Add a Grok pattern matching this host's configured log_format.
  } else {
    mutate {
      add_field => { "[log][type]" => "error" }
    }
    # Apply an Apache- or NGINX-specific error pattern here.
  }
}

output {
  elasticsearch {
    hosts => ["https://your-cluster.example"]
  }
  stdout { codec => rubydebug }
}

start_position => "beginning" controls the initial read of content that has no recorded position. The file input stores offsets in its sincedb registry; it does not replay content whose position is already registered merely because Logstash restarts. Treat the registry as state: back it up when appropriate, give it a stable path, and do not delete it casually in production.

Rank #2
Windex Gaming Cleaning Wipes, Pre-Moistened Wipes to Instantly Clean all Gaming Gear and Electronics, Resealable Packaging for a Freshness Lock, 25 count (Pack of 1)
  • WINDEX GAMING WIPES: effortlessly clean all your gaming gear and gaming electronics in one easy step with our new Windex Gaming Wipes
  • AMMONIA FREE: our ammonia-free formula is safe to use on all of your favorite gaming equipment like consoles, headsets, controllers, screens, keyboards, and more
  • PRE-MOISTENED WIPES: each wipe instantly removes dust, dirt, and stubborn smudges and fingerprints leaving a streak-free shine
  • ANTI-STATIC: great for cleaning and wiping down your gaming gear and electronics leaving behind no visible residue
  • COMPACT RESEALABLE PACKAGE: store your pack of Windex Gaming Wipes anywhere in your home for whenever a quick and easy wipedown is needed

Parse Apache access logs with the combined pattern

A standard Apache combined access line contains the client address, identity and user fields, timestamp, request line, status, response byte count, referrer, and user agent. Elastic’s Grok examples use %{COMBINEDAPACHELOG} to split that line into fields. Apply the date filter after Grok has created timestamp.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
filter {
  if [service][name] == "apache" and [log][type] == "access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      tag_on_failure => ["_grokparsefailure_apache_access"]
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
      tag_on_failure => ["_dateparsefailure_apache_access"]
    }
  }
}

Keep the raw message even after extraction. It is useful when a module adds a field, a proxy changes quoting, or a line must be reconstructed for an incident review. If a site uses a format other than combined access, use a pattern that reflects that format instead of forcing the composite pattern to match.

Parse NGINX access logs from the deployed log_format

NGINX does not have one universal access-line grammar. Operators often add upstream response time, forwarded addresses, request IDs, TLS details, or custom variables. A pattern copied from another server can therefore fail or, worse, assign values to the wrong fields.

Turn the format declaration into a contract

  1. Read the active log_format declaration, including inherited configuration and included files.
  2. List each variable in order, noting literal separators and whether values are quoted.
  3. Write a Grok pattern for that exact order and save it with a version or configuration identifier.
  4. Test it against lines containing empty values, spaces in quoted fields, IPv6 addresses, and long request URLs.
  5. Tag failures and retain the original line instead of dropping it.

For example, a pattern for a format containing an address, timestamp, request, status, byte count, referrer, user agent, and upstream timing must capture those fields in precisely that sequence. The names should be stable across hosts so dashboards do not need one query per server. If two NGINX configurations differ, route them using path, host metadata, or an explicit format tag rather than one permissive pattern.

Normalize the NGINX timestamp

Once Grok captures the timestamp, use a date filter with the exact layout emitted by that format. Include the timezone when the line supplies one. If the format emits a local time without an offset, configure the date filter’s timezone deliberately and document the assumption; otherwise events can appear shifted when hosts use different zones.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #3
Sale
LISEN Cell Phone Stand Phone Holder for Desk, Office Decor Gadgets for Men
  • Ergonomic & Adjustable Design: Fight "text neck" and improve your posture. Designed for comfort, it offers an adjustable height (7.1in-8.5in) (1.4in adjustable range) and angle (5°-85°) Please note: cell phone stand is only 9 inches tall—the length of a standard pencil
  • Without blocking subtitles Perfect for office desk Tiktok Face Time Zoom video—this is the ultimate kitchen desk essential for phones! Perfect viewing experience while reducing neck back strain, A must-have for gamers streamers desk, and anyone who values both style and support!
  • Sleek & Stable Design fits Otterbox Case: Say goodbye to bent cables, tangles, and tip-overs! LISEN’s stylish adjustable desk phone holder features a built-in charging hole and smart cable management, keeping your phone secure and your workspace clutter-free cool gadgets for men
  • ANTI-SLIP DESIGN free hands: The under 10.00 items easy gadgets back and the bottom of this kitchen essentials for women are fully covered by anti-skid silicone, which LISEN stand up desk decor can provide maximum protection device from any scratches and slides
  • Your Perfect Viewing Companion:If your product arrives damaged or breaks within 365 days, reach out to LISEN for a quick and hassle-free solution. Struggling with assembly, like tightening the screw? We’ve got your back! Your experience matters to us. (Product including: phone Holder stand*1, screw*1, hexagon screwdriver*1, specification*1)

Handle NGINX and Apache error logs separately

Error lines are not access records. They describe severity and a diagnostic message rather than an HTTP request and response. Sending them through an access parser creates false failures and misleading fields.

NGINX error logs

NGINX error records commonly include a timestamp, severity, process or connection context, and message text. The configured error_log location and severity determine what is written. Build a pattern for the exact version and deployment style, capture severity into a consistent field such as log.level, and leave the remainder in a message field. Preserve request IDs or upstream details when present.

Apache error logs

Apache error formats vary by version, module, and logging configuration. Use a separate conditional and a pattern that matches the installed format. Capture the timestamp and level when available, then apply a date filter. Do not apply COMBINEDAPACHELOG to an error line; the grammar is fundamentally different.

Keep unmatched errors visible

Add a source-specific failure tag such as _grokparsefailure_nginx_error or _grokparsefailure_apache_error. Route tagged events to a review index or alert stream while still retaining the raw line. A parse failure should be an observable condition, not silent data loss.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Choose ECS compatibility and field names deliberately

Grok composite patterns can emit different field names when ecs_compatibility is enabled. Decide this before creating index templates, dashboards, alerts, and saved queries. Changing modes later can leave old and new documents with incompatible mappings.

Rank #4
Ledagic IC+RGBW USB Powered Led Light Bar with App & Remote Control, 2 Pack
  • Easy Installation: Our gaming lights have been designed to enhance your entertainment experience. Simply position the led light bar next to your TV or game console to create a game room environment with ease. They come with USB power and a 60 inch cord for direct mounting and mobility
  • Audio Syncing & Extensive Mode: The built-in high-sensitivity music sensor allows you to sync the TV led lights to your favorite music. Lights react and move to the beat of music, film and game audio, creating an incredible immersive sound and light experience – and all without the need for an additional hub. You can choose from 64+ patterns, including strobe, illusion, jump, energy, gradient, running, chasing, fluttering mode etc. Each mode can be applied to different scenes
  • 64+ Chasing IC+RGB & W Effect: The independent control design of IC+RGB and white all-in-one multifunctional rgbw led bar lights monitor backlight allows you to easily switch the led lights for desk between IC+RGB and white to enjoy a variety of lighting effects, you can choose white lighting mode with high brightness for better illumination when working, etc. Choose IC+RGB lighting mode when you need to create an entertaining atmosphere for the light show, 16 million colours of Color changing rgb light bar let you dance to the beat and create a dynamic party atmosphere
  • Dynamic and Dimmable Lights: 1%-full brightness free brightness adjustment gives you the flexibility to adjust dimmable led light bar for room decorations or a variety of occasions
  • USB Powered: USB light bars for room can be powered by powernank, usb charger, TV, Monitor, laptop PC mainframe, making behind TV led lighting bar easy to install in any location.TV led lights are powered by the TV usb interface / PC mainframe, can be turned on/off synchronously with TV / PC
  • Use one documented ECS mode for the pipeline and its shared patterns.
  • Keep service identity under consistent fields such as [service][name].
  • Use [log][type] for the access/error distinction, or another documented namespace used consistently by your index templates.
  • Store source-specific values such as upstream timing under stable, clearly named fields.
  • Keep @timestamp as the event time after parsing the web server’s timestamp.

Validate a representative Apache access event, NGINX access event, and each error format against the destination’s mappings before enabling broad ingestion.

File input, Filebeat, or syslog: a practical decision

Use the file input when Logstash is close to the files

This is appropriate for a small number of hosts or a colocated Logstash process. Verify read permissions, monitor the sincedb location, and test rotation. The input follows files by path and position; careless registry deletion can cause duplicates, while an incorrect path or inaccessible directory causes gaps.

Use Filebeat when collection and parsing should be separate

Elastic’s advanced web-log tutorial demonstrates Filebeat sending Apache logs to Logstash, where the events are parsed into named fields before indexing. In this design, Filebeat owns discovery, harvesting, and local buffering; Logstash owns conditional routing, Grok, date conversion, enrichment, and output. Attach fields or tags in Filebeat so Logstash can distinguish NGINX from Apache and access from error streams without relying only on filename conventions.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use syslog when forwarding already exists

Logstash’s syslog input can listen on TCP and UDP and accept a custom grok_pattern for non-standard messages. The pattern must expose a timestamp that Logstash can parse. Confirm whether the sender prepends priority, hostname, or program fields, and test both transport paths separately. Prefer TCP where delivery guarantees matter; if UDP is required, account for possible loss and size limits.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Rotation, truncation, and state: prevent gaps and duplicates

Test the lifecycle, not just a static sample.

  • Rename-and-create rotation: verify that Logstash continues reading the old file to its end and begins the new file.
  • Copy-truncate rotation: verify behavior when a file is truncated in place; offsets and inode tracking can produce duplicates or missed lines depending on timing.
  • Compression: ensure the input does not attempt to parse compressed archives unless your chosen shipper explicitly supports that workflow.
  • Restart recovery: stop and start Logstash, then confirm that the registry resumes from the expected offsets.
  • Permissions: test after rotation, because a newly created file can have different ownership or mode bits.
  • Multiline records: web access lines are normally single-line; if an error format can span lines, configure multiline handling at the collector rather than letting fragments reach Grok independently.

Measure event counts around a controlled rotation and compare the raw files with indexed documents. Do not “fix” duplicates by deleting state until you understand which offsets were recorded.

Best Value
Sale
HUANUO FlowLift™ Dual Monitor Stand, Fully Adjustable Gaming Monitor Desk Mount for 13–32″ Computer Screens, Full Motion VESA 75x75/100x100 with C-Clamp & Grommet Base, Each Arm Holds 4.4 to 19.8 lbs
  • Compatible with Wide Screens - To ensure compatibility with the dual monitor mount, your each monitor must meet three conditions at the same time: First, computer screens size range: 13 to 32 inches. Second, screen weight range: 4.4 to 19.8 lbs. Third, the back of the monitor screen must have VESA mounting holes with a pitch of 75x75mm or 100x100mm.
  • Regarding the compatibility with desks - Your desk must meet three conditions at the same time: First, desk material: Only wooden desks are recommended, plastic or glass desks cannot be used. Second, desk thickness range: 0.59" - 3.54". Third, the bottom of the desk should not have any cross beams or panels, as this will interfere with installation. We recommend carefully checking that your desk and monitors meets all above conditions before purchasing.
  • Dual C-Clamp Hold - Worried your dual monitors might wobble or slip? Our upgraded base uses a larger platform plus a dual C-clamp structure to lock the dual monitor arm firmly to your desk. Each arm safely keeps your screens steady while you type, click and game—no shaking, no sliding, just a clean and secure setup you can trust every day. It also provides Grommet Mounting installation choice, both options ensure stable and secure fixation for your 0.59" - 3.54" desk.
  • Full-Motion Adjustment For Comfortable View - Pull the screen closer when you’re deep in a spreadsheet, push it back to watch videos, or rotate to portrait for coding — moving everything smoothly with just one hand. The monitor stand offers +85°/-50° tilt, ±90° swivel and 360° rotation. Raise your monitor up to 15.75″ to support a healthy sitting posture. Whether you’re working from home, gaming through the night, or switching between video calls and documents, getting the screens to your natural line of sight helps relieve neck, shoulder and back strain so you can stay focused longer with less fatigue.
  • Keep Your Desk Organized: By lifting both screens off the desktop, this dual monitor stand opens up valuable space for your keyboard, notebook, docking station or a simple, clutter-free work area. Built-in cable management guides wires along the arms, keeping cords out of sight and out of the way. Enjoy a tidy, modern workstation that looks as good as it feels to use.

Validate the pipeline before indexing everything

  1. Run Logstash with stdout { codec => rubydebug } alongside the production output.
  2. Feed one known Apache combined access line, one non-combined Apache line, one NGINX access line for every deployed format, and one error line from each server.
  3. Confirm that @timestamp reflects the log event time, not ingestion time.
  4. Check that status, request, bytes, client address, severity, and source metadata have the expected values.
  5. Intentionally submit malformed lines and verify that they retain message and receive the intended failure tag.
  6. Query the destination for failure tags and inspect them after deployment, especially following a web-server configuration change.

Troubleshoot common failures

No events arrive

Check the path against the running configuration, directory traversal permissions, SELinux or other mandatory access controls, and the sincedb position. A file already registered at its end will not be replayed by start_position => "beginning"; use a controlled test file or a separate registry for testing.

Apache fields are empty

Confirm that the line is actually combined format and that a proxy or module has not added fields. Inspect the raw message, then select a matching pattern or write a custom one.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

NGINX parsing fails after a configuration change

Compare the active log_format with the Grok pattern. Added variables, changed quoting, and optional upstream values are common causes. Version the pattern with the deployment and add a test line to the pipeline’s validation set.

Timestamps are wrong

Inspect the captured timestamp, date pattern, and timezone. Apply the date filter only after Grok has populated the field, and ensure the format token matches the server’s month, offset, and sub-second precision.

Events are duplicated after rotation

Review the rotation method, inode behavior, and registry state. Avoid deleting sincedb as a first response; doing so can replay all retained content. Correct the rotation and state strategy, then test it under the same timing used in production.

Production checklist

  • Confirm every NGINX and Apache path from the active configuration.
  • Separate access and error inputs and parsers.
  • Use COMBINEDAPACHELOG only for compatible Apache access lines.
  • Maintain custom, versioned Grok patterns for each NGINX log_format.
  • Parse source timestamps and preserve the original message.
  • Choose ECS compatibility before templates and dashboards.
  • Tag and monitor parse failures instead of dropping unmatched events.
  • Test rotation, truncation, compression, permissions, multiline behavior, and registry recovery.
  • Keep rubydebug output during development, then remove or restrict it when production logging is established.
  • Validate destination mappings with representative events from every deployed format.

The Bottom Line

Collect local files with Logstash’s file input when it has reliable access; otherwise ship with Filebeat or an existing syslog path. The durable solution is format-aware: separate access from error logs, parse Apache combined access lines with COMBINEDAPACHELOG, derive NGINX patterns from the active log_format, normalize timestamps, preserve raw messages, and test file state through rotation before trusting the indexed data.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More quests from Patch Notes

  1. How to Set Up a RedM (RDR2) Server in 2026: License Key, txAdmin and server.cfg, Step by StepBlog11min
  2. Best RedM (RDR2) Server Hosting in 2026: Comparing Four Hosts on Slots, Memory and PriceBlog10min
  3. How to Host a Mindustry Server in 2026: server-release.jar, Port 6567 and the Commands That MatterBlog7min
Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Outdated Drivers Are Slowing You DownFree scan - exact matches

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.