The fastest command-line check is a recursive wget spider crawl that writes to a log, followed by a search for HTTP and connection failures:
Antihost offers Linux VPS plans with full root access, unlimited traffic on a 1 Gbps port and a 14-day free trial.
wget --spider -r -nd -nv -w 2 -o crawl.log https://yoursite.example
grep -E "broken link|failed|404|403" crawl.logRun it against a site you control, keep the crawl scoped to the intended host, set a deliberate recursion depth, and verify suspicious 403 or HEAD-related results with a normal GET request before changing a link.
As an Amazon Associate I earn from qualifying purchases.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhat a wget broken-link crawl actually does
wget is a command-line network downloader. In spider mode, it checks discovered URLs without saving the page bodies to disk. Adding recursive mode makes wget parse pages and follow links it finds, allowing one command to inspect a site rather than a single landing page.
The crawl is a reachability test, not a visual audit. It can identify responses such as 404 Not Found, refused requests, DNS failures, connection errors and redirect loops when those conditions appear in wget’s output. It does not decide whether a page’s wording, layout or destination is useful, and it cannot reliably discover links that exist only after browser-side JavaScript runs.
#1 Best Overall
- BLUE LIGHT BLOCKING GLASSES: Our blue light glasses for women boast UV400 protection, blocking harmful blue rays from computer screens. With these bluelight glasses women, you can enjoy gaming, reading, or watching TV with no worries of eye strain.
- EYE RELIEF: Our blue light glasses are engineered to reduce blurry vision or strain caused by prolonged screen time. These womens blue light glasses aid in better sleep by preventing blue light from disrupting your natural sleep cycle, helping you fall asleep faster and enjoy a more restful night's sleep, making them the perfect christmas present.
- COMFORTABLE: Crafted from durable material, our trendy blue light glasses for woman or blue light glasses men ensure long-lasting comfort. The high-transmittance lenses accessories, gamer gifts for men, help keep your eyes relaxed. Perfect stocking stuffers, gifts for gamers men, christmas gifts for men, halloween costumes.
- LIGHTWEIGHT: These clear blue light glasses feature a lightweight frame, meaning they’ll stay comfy for long-term wear. Perfect as blue blockers glasses for women or mens blue light glasses, these non-prescription fake glasses for men can be worn throughout the day. The blue light blocking glasses women feature polycarbonate HD lenses, which are anti-reflective and restore true color.
- WORRY-FREE PURCHASE: If you experience any issues, our support team is ready to assist you quickly. These are the perfect costume glasses for cosplay dressing up.
The command above writes diagnostic output to crawl.log. Searching that file after the crawl is safer than trying to interpret a fast-moving terminal stream. Treat each match as a lead to verify, not automatically as proof that a link is permanently broken.
Before you start
Install and identify the wget build
Use the wget package supplied for your operating system, then check the executable and its supported switches:
Recommended Free Tools
wget --version
wget --helpThe GNU Wget manual indexed for this subject is labelled version 1.25.0, but distributions can ship another release or apply packaging changes. The commonly published command uses --spider, -r, -nd, -nv, -w and -o; confirm their definitions on the machine that will run the audit rather than assuming every build has identical defaults.
Get permission and choose a safe starting URL
- Crawl a site you own or are explicitly authorised to test. A recursive request can generate substantial traffic.
- Start with the canonical scheme and host, such as
https://yoursite.example. Decide whether thewwwand non-wwwnames are separate properties for your purposes. - Run from a network that can resolve the domain and reach the same services your visitors use. A corporate proxy, VPN or firewall can create failures that public visitors do not see.
- Pick a writable log location and enough time for the crawl to finish. Do not close the shell or delete the log while wget is running.
Run the basic internal-link crawl
- Open a terminal. Change to a directory where you want the log stored, or provide an absolute path to it.
- Run the spider command.
wget --spider -r -nd -nv -w 2 -o crawl.log https://yoursite.example--spiderperforms checks without saving retrieved pages.-renables recursive retrieval so links found in pages are followed.-ndmeans no directories, avoiding a downloaded directory tree.-nvreduces routine progress output.-w 2asks wget to wait two seconds between requests, and-o crawl.logwrites output to that file. - Wait for completion. A large site can take a long time. The process must finish before the log is a complete inventory.
- Search the log for likely failures.
grep -E "broken link|failed|404|403" crawl.logOn systems without grep, open the log in a text editor and search for the same terms. Wget’s exact wording can differ by version, so also inspect nearby lines and search for terms such as
not found,unable to resolve,connection refusedandtoo many redirects. - Record each URL and its referring page. Copy the full URL, response or error text, and the page on your site that linked to it. That information lets an editor or developer fix the source link instead of merely noting that a request failed.
Command options and what they change
| Option | Purpose | Operational consequence |
|---|---|---|
--spider |
Check URLs without saving page content. | Useful for link auditing; it is still a network crawl and can trigger access controls. |
-r or --recursive |
Follow links discovered in retrieved pages. | Turns a one-page check into a site crawl. |
-nd or --no-directories |
Do not create a directory hierarchy. | Keeps a spider run from building a local copy structure. |
-nv |
Use less verbose output. | Makes logs easier to scan, but leaves fewer routine details than verbose mode. |
-w 2 |
Wait two seconds between requests. | Reduces request rate; change it only after considering server capacity and your site’s policies. |
-o crawl.log |
Write output to the named log file. | Creates a durable record for filtering and later verification. |
-l or --level |
Set maximum recursion depth. | Prevents an unbounded-feeling crawl from travelling farther than intended. |
-H or --span-hosts |
Allow recursion onto other hosts. | Expands the crawl to external domains; use only with deliberate limits. |
-D or --domains |
Restrict permitted domains when spanning hosts. | Provides a whitelist so an external crawl does not follow every linked domain. |
The table describes the long-standing behaviour represented in the GNU Wget documentation and common usage. Because this article is not tied to a particular operating-system package, run wget --help locally when a switch is critical to an automated job.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #2
- WINDEX GAMING WIPES: effortlessly clean all your gaming gear and gaming electronics in one easy step with our new Windex Gaming Wipes
- AMMONIA FREE: our ammonia-free formula is safe to use on all of your favorite gaming equipment like consoles, headsets, controllers, screens, keyboards, and more
- PRE-MOISTENED WIPES: each wipe instantly removes dust, dirt, and stubborn smudges and fingerprints leaving a streak-free shine
- ANTI-STATIC: great for cleaning and wiping down your gaming gear and electronics leaving behind no visible residue
- COMPACT RESEALABLE PACKAGE: store your pack of Windex Gaming Wipes anywhere in your home for whenever a quick and easy wipedown is needed
Keep the first crawl on your own host
By default, recursive retrieval refuses to visit hosts other than the one supplied on the command line. That is the safest starting point: it checks links within your site without turning every social, payment, documentation or image-service URL into a second crawl target.
Do not add --span-hosts merely because a page contains an external link. Use it when you intentionally want wget to request other hosts, and combine it with a depth limit or an explicit domain allow-list:
wget --spider -r -l 2 -H -D yoursite.example,docs.example -nd -nv -w 2 -o external.log https://yoursite.exampleThe example permits recursion to the two named domains and stops after level 2. Replace the domains with ones you administer or have permission to test. The GNU manual warns that foreign hosts commonly link to still more hosts; without a depth or domain boundary, a small external-link check can widen rapidly.
Remember that a URL can be external even when it appears in an internal page. Decide in advance whether “broken links” means only links whose target host is your site or also links to services outside it. Report those categories separately so a third-party outage is not confused with a defect in your own server.
Choose recursion depth deliberately
The default maximum recursion depth is 5. Depth is counted through links discovered from the starting document, so a shallow setting can miss pages several clicks into a navigation tree while a deeper setting increases requests and log size.
Use a bounded crawl for routine checks
wget --spider -r --level=3 -nd -nv -w 2 -o crawl-level-3.log https://yoursite.exampleA level of 3 is an example, not a universal best value. Match it to your information architecture: a compact brochure site may be covered at a low level, while a game guide, knowledge base or archive may require more. Keep the chosen value in your audit notes so two runs can be compared meaningfully.
Rank #3
- Ergonomic & Adjustable Design: Fight "text neck" and improve your posture. Designed for comfort, it offers an adjustable height (7.1in-8.5in) (1.4in adjustable range) and angle (5°-85°) Please note: cell phone stand is only 9 inches tall—the length of a standard pencil
- Without blocking subtitles Perfect for office desk Tiktok Face Time Zoom video—this is the ultimate kitchen desk essential for phones! Perfect viewing experience while reducing neck back strain, A must-have for gamers streamers desk, and anyone who values both style and support!
- Sleek & Stable Design fits Otterbox Case: Say goodbye to bent cables, tangles, and tip-overs! LISEN’s stylish adjustable desk phone holder features a built-in charging hole and smart cable management, keeping your phone secure and your workspace clutter-free cool gadgets for men
- ANTI-SLIP DESIGN free hands: The under 10.00 items easy gadgets back and the bottom of this kitchen essentials for women are fully covered by anti-skid silicone, which LISEN stand up desk decor can provide maximum protection device from any scratches and slides
- Your Perfect Viewing Companion:If your product arrives damaged or breaks within 365 days, reach out to LISEN for a quick and hassle-free solution. Struggling with assembly, like tightening the screw? We’ve got your back! Your experience matters to us. (Product including: phone Holder stand*1, screw*1, hexagon screwdriver*1, specification*1)
Do not assume an unverified special value
Some examples in circulation use --level=0 for an internal crawl, but the meaning of that value was not confirmed for this guidance. Do not build a production audit around it without checking the manual and wget --help for your installed version. An explicit finite level is easier to reason about and safer for a first run.
Interpret the failure signatures
| Log signature | What it usually indicates | What to do next |
|---|---|---|
404 Not Found |
The server says the requested resource does not exist at that URL. | Open the referring page, check for a changed slug or typo, then update the link or restore the intended page. |
403 Forbidden |
The server refuses the request. | Do not label it a missing page automatically. Check authentication, firewall rules, rate limits and request-method differences. |
Failed: Name or service not known |
DNS resolution or a related connection lookup failed. | Retry from another network or resolver and inspect the hostname’s DNS records before editing site content. |
Too many redirects |
The redirect chain loops or exceeds wget’s limit. | Follow the chain manually and correct the server, CMS or protocol redirects that point back into the loop. |
| Connection timeout, refused or TLS errors | The client could not establish or complete a connection. | Retry, compare with a browser or another network, and check uptime, certificates, firewall and proxy settings. |
Use the URL and referring page as the unit of work. One destination may be linked from dozens of pages, so fix the shared template or navigation component when appropriate rather than making identical edits one page at a time.
Free tools Windows power users keep installed
One-click scans. No signup required.
Verify before calling a link broken
Recheck 403 responses
A 403 means “refused,” not “missing.” A private page, a bot rule, an IP allow-list, a web-application firewall or an expired login can all produce it. Confirm the URL in a normal browser session and ask the site owner whether automated requests are permitted. If the page is intentionally protected, document it as an access-controlled destination instead of replacing a valid link.
Repeat suspicious results with a GET
Spider checks may use HEAD requests. Some servers handle HEAD differently from GET, so a URL that looks failed in spider mode can still return the expected page to a normal request. Recheck a suspect URL with a non-spider wget request, preferably without saving content to your public web directory:
wget -S -O /dev/null https://yoursite.example/path/to/check-S shows server response headers and -O /dev/null discards the body on systems that provide /dev/null. If your platform does not have that path, use a temporary output file and remove it afterward. Compare the GET result with the browser result before changing the source link.
Separate transient outages from content errors
Run a second check after a short interval for DNS, timeout and connection failures. A transient network event should not cause a permanent content edit. Capture the date, network location and response so recurring failures can be escalated with useful evidence.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteRank #4
- Easy Installation: Our gaming lights have been designed to enhance your entertainment experience. Simply position the led light bar next to your TV or game console to create a game room environment with ease. They come with USB power and a 60 inch cord for direct mounting and mobility
- Audio Syncing & Extensive Mode: The built-in high-sensitivity music sensor allows you to sync the TV led lights to your favorite music. Lights react and move to the beat of music, film and game audio, creating an incredible immersive sound and light experience – and all without the need for an additional hub. You can choose from 64+ patterns, including strobe, illusion, jump, energy, gradient, running, chasing, fluttering mode etc. Each mode can be applied to different scenes
- 64+ Chasing IC+RGB & W Effect: The independent control design of IC+RGB and white all-in-one multifunctional rgbw led bar lights monitor backlight allows you to easily switch the led lights for desk between IC+RGB and white to enjoy a variety of lighting effects, you can choose white lighting mode with high brightness for better illumination when working, etc. Choose IC+RGB lighting mode when you need to create an entertaining atmosphere for the light show, 16 million colours of Color changing rgb light bar let you dance to the beat and create a dynamic party atmosphere
- Dynamic and Dimmable Lights: 1%-full brightness free brightness adjustment gives you the flexibility to adjust dimmable led light bar for room decorations or a variety of occasions
- USB Powered: USB light bars for room can be powered by powernank, usb charger, TV, Monitor, laptop PC mainframe, making behind TV led lighting bar easy to install in any location.TV led lights are powered by the TV usb interface / PC mainframe, can be turned on/off synchronously with TV / PC
Robots rules, request rate and ownership
Some command examples disable robots handling with -e robots=off. That setting ignores robots.txt and should be used only on a site you own or are explicitly authorised to crawl. Ignoring a policy file on someone else’s host can violate their operating expectations and trigger defensive systems.
A two-second wait is a conservative starting point, not a guarantee that a server will tolerate the crawl. Keep the process during a low-traffic window when possible, avoid launching several recursive jobs at once, and stop if the server shows strain. If a host publishes a crawl policy or asks you to use a particular user agent or rate, follow that direction.
Because wget keeps track of discovered URLs while recursing, very large sites can require substantial memory. Use a narrower starting path, a lower depth, or separate crawls by section when a single run becomes unwieldy. Save each log with the scope and date in its filename.
Know what JavaScript-heavy pages hide
wget parses the HTML and other responses it receives; it does not render a page like a modern browser. Links inserted after load by client-side JavaScript, links behind an interaction, and destinations returned only by an API call may never appear in the crawl. A clean wget log therefore means “no failures among the links wget discovered,” not “every clickable destination in the application works.”
For an application with client-rendered navigation, inspect the server-rendered HTML as well as the browser experience, and maintain a separate test for routes created at runtime. Do not inflate the wget command with unrelated switches in the hope of making it a JavaScript browser; instead, record its coverage boundary in the audit report.
Practical command recipes
| Goal | Command | Use when |
|---|---|---|
| Default internal audit | wget --spider -r -nd -nv -w 2 -o crawl.log https://yoursite.example |
You want the commonly cited starting point with the default recursion limit. |
| Bound the crawl to three levels | wget --spider -r --level=3 -nd -nv -w 2 -o crawl-level-3.log https://yoursite.example |
Your site is deep and you want a predictable request budget. |
| Check selected external domains | wget --spider -r -l 2 -H -D yoursite.example,docs.example -nd -nv -w 2 -o external.log https://yoursite.example |
You have permission to test the listed hosts and want a controlled cross-host crawl. |
| Inspect one suspect URL with GET | wget -S -O /dev/null https://yoursite.example/path/to/check |
A spider result needs confirmation using a normal request. |
| Filter common errors | grep -E "broken link|failed|404|403|not found|too many redirects" crawl.log |
You need a first pass through a completed log; review surrounding lines manually. |
A repeatable maintenance workflow
- Define scope: record the starting host, whether external hosts are excluded, and the recursion level.
- Run politely: use a wait interval, avoid robot-policy bypasses unless authorised, and schedule heavy crawls responsibly.
- Preserve evidence: name the log with the date and scope, for example
crawl-2026-10-14-internal.log. - Extract candidates: search for status codes and failure phrases, but retain the complete surrounding log context.
- Verify: re-request 403, DNS, timeout and redirect findings; use GET when a HEAD-style check may be misleading.
- Fix at the source: correct the referring template, navigation item, content entry or redirect rule. If the destination moved, use the confirmed canonical URL.
- Rerun the affected scope: check the repaired page and its nearby section, then run the broader crawl on the cadence your site can support.
- Report coverage: state that results apply to URLs wget discovered at the chosen depth and host scope, excluding browser-only JavaScript destinations.
When wget is the right tool—and when it is not
wget is a good fit when you want a free, scriptable check from a shell, control over host scope and depth, and a plain-text log that can be searched or archived. It is especially useful for a site owner who can run the command on a server or scheduled job.
Best Value
- Compatible with Wide Screens - To ensure compatibility with the dual monitor mount, your each monitor must meet three conditions at the same time: First, computer screens size range: 13 to 32 inches. Second, screen weight range: 4.4 to 19.8 lbs. Third, the back of the monitor screen must have VESA mounting holes with a pitch of 75x75mm or 100x100mm.
- Regarding the compatibility with desks - Your desk must meet three conditions at the same time: First, desk material: Only wooden desks are recommended, plastic or glass desks cannot be used. Second, desk thickness range: 0.59" - 3.54". Third, the bottom of the desk should not have any cross beams or panels, as this will interfere with installation. We recommend carefully checking that your desk and monitors meets all above conditions before purchasing.
- Dual C-Clamp Hold - Worried your dual monitors might wobble or slip? Our upgraded base uses a larger platform plus a dual C-clamp structure to lock the dual monitor arm firmly to your desk. Each arm safely keeps your screens steady while you type, click and game—no shaking, no sliding, just a clean and secure setup you can trust every day. It also provides Grommet Mounting installation choice, both options ensure stable and secure fixation for your 0.59" - 3.54" desk.
- Full-Motion Adjustment For Comfortable View - Pull the screen closer when you’re deep in a spreadsheet, push it back to watch videos, or rotate to portrait for coding — moving everything smoothly with just one hand. The monitor stand offers +85°/-50° tilt, ±90° swivel and 360° rotation. Raise your monitor up to 15.75″ to support a healthy sitting posture. Whether you’re working from home, gaming through the night, or switching between video calls and documents, getting the screens to your natural line of sight helps relieve neck, shoulder and back strain so you can stay focused longer with less fatigue.
- Keep Your Desk Organized: By lifting both screens off the desktop, this dual monitor stand opens up valuable space for your keyboard, notebook, docking station or a simple, clutter-free work area. Built-in cable management guides wires along the arms, keeping cords out of sight and out of the way. Enjoy a tidy, modern workstation that looks as good as it feels to use.
It is less suitable as the only test for a JavaScript application, an authenticated area, or a site whose important links depend on user actions. A dedicated link-auditing system may add browser rendering, scheduling or richer reports, but those capabilities are outside the evidence established here. If you evaluate another tool, compare it on request method, JavaScript coverage, authentication handling, host-span controls, scheduling, export format and rate limiting rather than assuming that every “link checker” sees the same URLs as wget.
Troubleshooting common runs
The log is empty or unexpectedly short
Confirm that the command reached the intended URL, that crawl.log is writable, and that the process was allowed to finish. Run a single-URL request with visible output to distinguish a logging problem from a network problem.
Only the home page was checked
Verify that recursive mode was included and that the starting page contains crawlable HTML links. A page that builds navigation entirely in JavaScript will not expose those routes to wget. Also check whether your chosen depth is too low for the site’s structure.
Many pages show 403
Pause the crawl and determine whether a firewall, authentication layer, robots policy or rate limit is responding. Lower the request rate, obtain permission or use an authorised testing environment. Do not bulk-replace links solely because wget received 403.
External links are missing
That is expected under the default same-host rule. Add --span-hosts only for a deliberate, permitted external audit, and pair it with --level and --domains limits.
A browser works but wget reports failure
Compare a normal GET with the spider result. The server may treat HEAD differently, require a cookie or authentication token, enforce a bot rule, or behave differently from your browser’s network. Classify the result as an access or tooling difference until you have confirmed the destination’s real visitor experience.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Bottom line
Start with wget --spider -r, log the crawl, search for 404, 403, DNS and redirect errors, and keep the default same-host boundary unless you intentionally expand it. Set recursion depth, wait between requests, verify ambiguous findings with GET, and document what JavaScript-rendered or access-controlled content the crawl could not test. Used that way, wget provides a transparent, repeatable inventory of broken-link candidates without pretending to be a full browser.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
