To make Filebeat read old files from the beginning, stop it, remove the registry state for that installation, and start it again with the archived path enabled. To replay archived logs into Elasticsearch, stage the files on a reachable host, parse them with Filebeat or Logstash, and write the result to an isolated destination while preserving each event’s original timestamp and timezone. Filebeat is usually enough for straightforward harvesting; choose Logstash when parsing, enrichment, routing, or multiple outputs are involved.
Antihost offers Linux VPS plans with full root access, unlimited traffic on a 1 Gbps port and a 14-day free trial.
The safe method is a controlled replay, not a blind restart: test one file, inspect the indexed documents, then expand in bounded batches while watching failures, mappings, latency, and disk watermarks.
As an Amazon Associate I earn from qualifying purchases.
Free tools Windows power users keep installed
One-click scans. No signup required.
What “reload old logs” means in ELK
Historical log files are new input data. They are not automatically recovered by Elasticsearch’s reindex API, because reindex operates on documents that already exist in an Elasticsearch index. Loose files must be harvested or converted into documents and then indexed.
Elastic describes ingestion as getting data into Elasticsearch before it can be searched, visualized, or used for insights. A historical replay therefore has four separate concerns:
#1 Best Overall
- Reading the archived bytes, including rotated and compressed files.
- Turning lines into events, including multiline records such as stack traces.
- Assigning the event’s original time and timezone rather than the replay time.
- Writing to a destination where you can validate, roll back, and detect duplicates.
Those concerns apply whether the files contain application, infrastructure, or game-server events. The filenames, date range, rotation scheme, and timestamp format determine the configuration.
Choose the ingestion route
| Source or requirement | Approach | When it fits |
|---|---|---|
| Archived files with simple line-oriented records | Filebeat to Elasticsearch | Lightweight harvesting and forwarding with minimal transformation |
| Archived files needing grok parsing, enrichment, routing, or multiple outputs | Filebeat to Logstash to Elasticsearch | Heavier processing before indexing |
| One file for a non-production test | Kibana file uploader | Quick single-file indexing experiment |
| Already structured records | Elasticsearch document or bulk APIs | You can generate valid JSON documents directly |
| Supported time-series source with an integration | Elastic Agent and an integration | The integration provides the current collection and mapping workflow |
| Logs already indexed in another Elasticsearch cluster | Remote reindex, snapshot and restore, or Logstash migration | The source is an existing cluster rather than loose files |
Do not use a file replay when the authoritative data already exists as indexed documents in another cluster unless you have a reason to reconstruct it from files. Elastic recommends indexing from the original source when available. Remote reindex rebuilds the destination index and can consume more resources than snapshot restore.
Stage and inventory the archive
Copy or mount the files on a host that can reach the Elasticsearch deployment, or the Logstash endpoint you will use. Keep the originals unchanged. Work from a copy when possible so that a failed parser or accidental state reset does not alter your evidence.
Record the properties that affect parsing
- Date range: note the earliest and latest event dates, not just the filesystem modification dates.
- Timezone: record the timezone used by the writer. A timestamp without an offset needs an explicit timezone in the parser.
- Rotation naming: document suffixes, date patterns, and whether a file is an active file, a rotated file, or an archive.
- Compression: identify whether files are plain text or compressed and confirm that your chosen reader supports the format or that you have decompressed a working copy.
- Encoding and delimiters: check line endings, character encoding, and whether a message can contain embedded newlines.
- Source identity: preserve the host, service, environment, and file path as fields or metadata so events from several archives remain distinguishable.
Do not put an entire archive into a live destination immediately. Select one representative file containing ordinary events, a boundary between dates, an error, and any multiline record. That sample exposes timestamp and mapping errors before they multiply.
Decide between Filebeat and Logstash
Use Filebeat when harvesting is the main job
Filebeat is a lightweight file harvester that forwards events to Elasticsearch or Logstash. It is a good fit when each event is already close to the shape you want and you mainly need path selection, multiline handling, timestamp extraction, and forwarding.
Use Logstash for transformation and routing
Choose Logstash when the archive needs complex grok patterns, conditional parsing, enrichment, field conversion, routing to different destinations, or more than one output. A common pattern is Filebeat reading files, Logstash receiving Beats events, a Grok filter creating named fields, and the Elasticsearch output indexing the result.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Filebeat and Logstash are not mutually exclusive. Filebeat can provide reliable file-state tracking while Logstash performs the expensive or variable transformations. For a small, structured archive, adding Logstash may create unnecessary operational work; for inconsistent historical formats, it can prevent a large amount of manual cleanup.
Make Filebeat reread files from the beginning
Filebeat records which files and offsets it has read in its registry. If the registry says an archived file is complete, merely restarting Filebeat will not replay it. Elastic’s tutorial gives the direct procedure: stop Filebeat, delete the registry file, and restart it.
Controlled replay procedure
- Stop Filebeat cleanly. Use the service manager appropriate to your operating system, or run it in the foreground and press
Ctrl+C. Confirm that the process has stopped before changing state. - Preserve the current state. Back up or otherwise record the registry location and contents. This gives you a recovery point and documents which live files had already been read.
- Remove only the intended registry state. Delete the registry file or state directory used by this Filebeat installation, not unrelated agent data. The exact path depends on the installation and configuration.
- Check the input paths. Ensure the archived directories are included and that the process has permission to read them. Keep live production paths out of this replay if you do not intend to reread them.
- Start Filebeat with the replay configuration. Confirm that the output, index or data stream, credentials, TLS settings, and processors point to the isolated historical destination.
- Watch the first events. Verify that the oldest sample events appear, that their timestamps are historical, and that multiline messages remain a single event.
- Stop or separate the replay when complete. Do not leave a reset registry attached to a live harvester unless rereading and possible duplication are deliberate.
Deleting registry state is a replay operation. Every file still visible to the configured inputs may be read again, so duplicates are expected unless you isolate the destination or implement a deterministic deduplication strategy. Never delete state on a live production harvester merely to make an old file appear.
Parse events before indexing
Multiline records
Stack traces and wrapped messages often occupy several physical lines but represent one event. Configure multiline rules that identify where a new event starts and attach continuation lines to it. Test both a normal single-line event and a multiline exception; an incorrect rule can merge unrelated records or split one stack trace into many documents.
Event timestamps and timezone
Extract the timestamp written by the application and map it to the field used for time-based searches and visualizations. Apply the timezone in which the source generated the timestamp. If you use ingestion time instead, a five-year-old event will appear in the replay window rather than in its original period, making retention checks and incident timelines misleading.
Field names and data types
Map fields consistently across the sample and the full archive. A field that is text in one file and numeric in another can cause mapping conflicts. Decide how to represent status codes, durations, IDs, host names, and message text before loading multiple formats into one index or data stream.
Parsing with Logstash
A representative pipeline shape for Apache-style records is:
input { beats { port => 5044 } }
filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } }
output { elasticsearch { hosts => ["https://your-elastic-endpoint:9200"] } }The port, endpoint, authentication, TLS, pipeline, data-stream, and index settings in this example are illustrative. Use the exact settings required by your installed Elastic version and deployment. Treat the sample as a shape: Beats input, Grok parsing, and Elasticsearch output. Validate the resulting fields before sending the complete archive.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Isolate the historical destination
Write the replay to a dedicated historical index or data stream, or add an explicit field such as a replay source identifier. Separate storage makes it possible to compare counts, discard a bad run, and identify duplicates without contaminating live search data.
Choose a naming convention that records the source and replay purpose without changing the event’s original time. Keep index templates and mappings aligned with the fields your parser emits. If the archive spans formats that cannot share one mapping, use separate destinations and normalize them later.
Do not treat a separate destination as permission to ignore security. Apply the same authentication, TLS, role, and data-retention controls required for the target deployment. Historical logs may contain credentials, player identifiers, IP addresses, or other sensitive values even when they are no longer current.
Run the replay in bounded batches
Begin with one file or a narrow date range. A bounded first run limits the cost of a bad timestamp parser, a malformed multiline rule, or a mapping conflict.
Recommended Free Tools
Rank #4
Increase scope only after these checks
- The oldest and newest events in the sample have the expected event times.
- Representative messages and stack traces are complete documents.
- Fields have the intended names and data types.
- The destination index or data stream receives documents without ingest failures.
- Document counts are explainable from the source sample.
- The cluster remains below its disk watermarks and indexing latency is acceptable.
- Rejected documents and authentication or TLS errors are absent or understood.
Expand by file, day, or another boundary that you can repeat. Keep a run log containing the source file set, configuration revision, start and end time, destination, and observed document count. This record is useful when two replays overlap or a duplicate must be removed.
Verify results in Elasticsearch and Kibana
Verification should use the original event time and a distinctive field, not only the ingestion timestamp. Query the historical destination for the earliest date in the archive and a recognizable message, host, service, or game-server identifier. Compare the returned document to the source line or record.
Check the destination structure
- Confirm the expected index name or data-stream backing index.
- Inspect mappings for timestamps, keyword fields, numeric values, and message text.
- Compare source and indexed counts for the batch, allowing for deliberately rejected records that you have investigated.
- Open several documents in Kibana to verify parsed fields, timezone conversion, and multiline assembly.
- Search across the date range to ensure events are distributed according to their original times rather than the replay window.
A successful connection does not prove a successful replay. A pipeline can accept input while dropping fields, assigning the wrong date, or routing documents to an unexpected destination. Inspect actual documents before increasing volume.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Alternatives for special cases
Kibana file uploader
Kibana’s file uploader is intended for indexing a single file in a non-production test. It can answer whether a sample is readable, but it is not a substitute for a repeatable archive pipeline with registry control, batching, and operational monitoring.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Elasticsearch document and bulk APIs
If you already have structured records, generate valid JSON documents or bulk requests and send them directly to Elasticsearch. This avoids line parsing, but you must supply the timestamp, field types, destination, authentication, and error handling yourself. Keep request batches bounded and inspect rejected items rather than assuming a successful HTTP response means every record was indexed.
Elastic Agent and integrations
For a supported time-series source, an Elastic Agent integration is often the simplest current collection path. Confirm that the integration can read the archived format and that its processors preserve the historical timestamp before committing the full replay.
Best Value
Another Elasticsearch cluster
When the old logs are already indexed elsewhere, consider remote reindex, snapshot and restore, or a Logstash migration instead of exporting and reparsing files. Remote reindex rebuilds the destination index and may be more resource-intensive than restoring a snapshot. The right choice depends on access to the source cluster, version compatibility, required transformations, and whether you need to change mappings during migration.
Troubleshoot the failure mode
Only new files are read
The registry still contains offsets for the archive, or the archived path is not included in the input. Stop Filebeat, confirm the path, and perform the controlled registry reset described above. Keep the destination isolated so the reread cannot silently duplicate live data.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallEvents show the replay date
The parser did not extract the source timestamp, or the timezone was not configured. Inspect a raw message and the indexed date field. Correct the timestamp processor and timezone, then replay the affected batch into a clean destination.
One stack trace becomes many documents
The multiline start pattern does not match the archive’s actual first-line format. Test the rule against both normal and exception records, then rerun a small sample before expanding.
Documents are rejected or fields conflict
Inspect ingest errors and mappings. Typical causes include a field changing type between files, a malformed timestamp, or a parser emitting a value that the destination template does not accept. Correct the mapping or normalize the field in the parser; do not repeatedly resend the same failing batch without changing the cause.
Counts are higher than expected
A registry reset, overlapping file selection, or repeated batch likely created duplicates. Compare the run log with the configured paths and use the isolated destination to remove the bad run. Add a stable source-file, offset, or event identifier when the data provides one so future replays can be detected.
The cluster slows or reaches a disk watermark
Pause expansion, reduce the batch scope, and investigate indexing latency, rejected documents, and available disk. A historical load competes with live workloads; resume only after the destination and cluster have headroom.
A repeatable runbook
- Inventory files, dates, timezone, rotation, compression, encoding, and source identity.
- Select Filebeat alone or Filebeat plus Logstash according to parsing and routing complexity.
- Build an isolated destination and define mappings, timestamp handling, multiline rules, and security settings.
- Test one representative file and inspect real documents.
- For a reread, stop Filebeat, preserve its registry, remove only the intended state, and restart with archive-only inputs.
- Replay bounded batches while recording source files, configuration, counts, failures, and destination.
- Verify timestamps, mappings, multiline records, counts, and Kibana searches.
- Expand only after the sample passes, and clean up or retain the historical destination according to your retention policy.
The durable pattern is simple: preserve the files, reset Filebeat deliberately when necessary, parse before indexing, isolate the destination, and verify the original event time. Use cluster-to-cluster migration tools when the source is already indexed data rather than a directory of files.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
