The httpd conf file: A Complete Guide for 2026

A container starts clean in CI, then fails in staging after a config reload because the image copied one include file into the wrong path. Or a replay test floods Apache with realistic traffic and exposes a timeout, worker, or logging setting that looked harmless at normal volume. That is usually when a team stops treating the httpd conf file like boilerplate and starts treating it like production control logic.

That shift is justified.

Apache has been in production use since the mid-1990s, and it still shows up everywhere from internal admin tools to reverse proxies in front of older applications. W3Techs reports Apache remains one of the most widely used web servers on the public web. Its staying power comes from how much behavior you can define in configuration. You can shape request routing, module loading, process behavior, TLS handling, logging, and access rules without changing application code.

That flexibility cuts both ways. A small syntax error can prevent the server from starting. A valid but poorly chosen directive can pass basic checks, then fail only under container restarts, blue-green rollouts, or replay-heavy load tests that send traffic patterns closer to production.

Older tutorials rarely spend time on those failure modes. They explain a single VirtualHost, then stop. Real systems use layered includes, environment-specific overrides, immutable images, config validation in pipelines, and reload procedures that need to be safe under live traffic. That is the gap this guide addresses.

Introduction Why Your Server Depends on the httpd conf file

The httpd conf file is Apache’s rulebook. It tells the server what to listen on, what content to serve, which modules to load, how to apply access rules, and how to behave under load. If Apache is the engine, httpd.conf is the set of controls that decides whether that engine starts cleanly or stalls on boot.

In daily operations, this file matters because configuration errors are operational errors. A bad application release can usually be rolled back. A bad Apache config can block every request before your app even gets a chance to fail. That’s why experienced teams treat Apache config changes with the same caution they give database migrations.

Practical rule: Never edit Apache configuration without a validation step ready. The config file is declarative, but the failure mode is immediate.

The good news is that Apache’s configuration model is logical once you stop treating it like a giant text blob. The directives have a grammar. The file has scope and structure. The best setups are modular, versioned, and tested before reload.

That also makes Apache easier to run in places where older tutorials struggle, especially containers and replay-heavy test environments. The same principles still apply. You just need to use them deliberately.

What Is the httpd conf file and Where to Find It

The httpd conf file is Apache’s control surface. It defines what the server listens on, which content it serves, which modules are active, how requests are logged, and which security rules apply before a request ever reaches your application. Apache is explicit by design. If a behavior exists, some directive enabled it.

The file name and path depend heavily on the operating system and packaging choices. On Red Hat based systems, the primary file is usually /etc/httpd/conf/httpd.conf. On Debian and Ubuntu, the main entry point is commonly /etc/apache2/apache2.conf, with much of the site-specific work split into included files under directories such as sites-available, sites-enabled, mods-enabled, and conf-enabled.

That difference matters in practice. A developer SSHs into a host, searches for httpd.conf, finds nothing, and assumes Apache was installed incorrectly. In many cases, Apache is fine. The distribution just uses a different layout and a heavier include model.

Typical file locations

Operating System / Distribution	Typical Path
RHEL / CentOS	/etc/httpd/conf/httpd.conf
Debian / Ubuntu	/etc/apache2/apache2.conf
Oracle Solaris	/etc/apache2/2.4/httpd.conf

The useful question is not only “where is the file?” Ask how this Apache build composes configuration. Some environments rely on one large file. Others keep the main file thin and load most settings from smaller fragments. That split changes how you review diffs, stage changes, and debug bad includes.

What the file actually controls

At a minimum, this configuration governs several parts of server behavior:

• Networking such as which IPs and ports Apache binds to
• Content mapping such as DocumentRoot and URL-to-file handling
• Module loading for features like rewrite rules, TLS termination, reverse proxying, and status endpoints
• Access policy for directories, locations, and administrative paths
• Virtual hosts so one Apache instance can serve multiple domains and apps

In production, many outages that look like app failures start here instead. Common causes include a missing module after a package update, an include file that was not mounted, a redirect loop inside a virtual host, or directory permissions that block requests before the app can respond.

What changes in containers

Containers add a layer older Apache tutorials rarely account for. The familiar paths may still exist, but they are no longer the whole story. A Docker image might ship a base httpd.conf, mount virtual host fragments at runtime, and inject environment-specific values during startup. In Kubernetes, a ConfigMap or baked image layer often matters more than what you see by shelling into a running pod.

That shifts the troubleshooting approach. The job is not just finding httpd.conf. The job is identifying which config Apache loaded in this container, from which layer, and whether that state will survive a restart.

On a mutable VM, editing one file in place can be enough. In a container, the sustainable fix is usually updating the image, the mounted config, or the deployment manifest.

Teams that miss this spend time patching a live container, then lose the change on the next rollout. For load testing and high-volume environments, that mistake is expensive. You need config that is repeatable, versioned, and identical across rebuilds, or your test results and production behavior will drift for reasons that have nothing to do with the application.

Understanding the Configuration File Structure

Apache config feels dense because one file can hold global server settings, per-site rules, filesystem permissions, and request routing. The fastest way to read it is to stop treating it like a flat text file and start reading it by context.

A simple starting point looks like this:

ServerName app.internal
Listen 80

Each line has a directive and its arguments. Apache parses those lines based on where they appear. Some directives are valid at the top level. Others only work inside containers such as <Directory> or <VirtualHost>. If you put a valid directive in the wrong context, Apache will reject it or apply it differently than you expected.

That matters in production and even more in containers. A config that works on a long-lived VM often gets split across image layers, mounted fragments, and environment-specific includes. During a load test, you need to know which file set defined the running server, not just which file looked correct in Git.

Directives and arguments

Directive names are case-insensitive. Arguments are not always forgiving. The Apache configuration guide explains that distinction clearly at Apache 2.4 configuration guide.

For example:

DocumentRoot "/var/www/html"

DocumentRoot is the directive. "/var/www/html" is the argument. If the path is wrong, mounted read-only, or missing in the container image, Apache may start and still serve the wrong content or fail the config test.

I see more problems from bad arguments than bad directive names. Paths drift between environments. Hostnames get copied from staging. Include globs match nothing in one image and several files in another.

Context defines where rules apply

Apache has several configuration contexts, and they stack in a predictable way:

• Global context affects the server process as a whole
• Main server context applies to the base server definition
• Virtual host context applies to one hostname, port, or listener combination
• Directory, location, and file contexts narrow behavior to a filesystem path or URL target

That hierarchy is what keeps one setting from spilling into everything else.

Here is a small example:

<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

Those rules apply to /var/www/html. They do not automatically apply to every path Apache serves. This is one of the first places teams get tripped up during migrations. They move an app from /var/www/html to /srv/app/public, keep the old <Directory> block, and then spend an hour debugging 403 responses that were caused by config scope, not application code.

<VirtualHost> has the same pattern. A redirect inside one virtual host does not become global just because it sits in the same file. File order still matters, but context decides the boundary.

Includes keep configs maintainable

Once an Apache deployment serves multiple apps or environments, a single giant httpd.conf becomes hard to review and easy to break. Include and IncludeOptional let you split config into smaller files and load them from the main config.

Include conf.d/*.conf
IncludeOptional sites-enabled/*.conf

That structure helps in real operations:

1. Changes stay isolated. Updating one virtual host does not require editing unrelated module or logging settings.
2. Ownership is clearer. A platform team can keep the base image and security defaults, while application teams maintain site-specific fragments.
3. Diffs are easier to review. That matters when a rollout changes behavior under load and you need to find the exact config delta fast.

In containerized deployments, includes are also how teams inject environment-specific config without rebuilding the whole image. That is useful, but it creates a trade-off. More fragments make reuse easier. They also make startup order, mount paths, and missing files harder to reason about unless the include layout is disciplined and versioned.

Apache can parse very long lines, and .htaccess has tighter limits than main server config. The practical lesson is simpler than the parser detail. Do not rely on giant generated directives when a small included file is easier to audit.

Multi-line config and readability

Apache supports line continuation when a backslash is the last character on the line.

LogFormat "%h %l %u %t \"%r\" %>s %b" \
combined

Use that sparingly. A trailing space after the backslash can break the continuation and waste time during deploy validation.

Readable config wins. Keep the main file boring, keep include names obvious, and run apachectl -t or httpd -t against the exact config set your container or host will load. That habit catches a lot of bad assumptions before they turn into failed rollouts.

Essential Directives You Must Know

You don’t need to memorize every Apache directive. You do need to recognize the handful that shape server behavior quickly. These are the ones that come up constantly in reviews, incident calls, and environment migrations.

Network identity and listeners

Listen tells Apache which port to bind to. ServerName tells Apache how to identify itself for request handling and URL generation.

Listen 80
ServerName app.internal

Why it matters: if Listen is wrong, Apache may start but never accept traffic where you expect. If ServerName is missing or misleading, name-based virtual host behavior and redirects can get messy fast.

A common mistake is adding a virtual host and forgetting that Apache still needs to be listening on the corresponding port.

Content roots and directory rules

DocumentRoot maps the site to a filesystem path. <Directory> sets permissions and behavior for that path.

DocumentRoot "/var/www/app/current/public"

<Directory "/var/www/app/current/public">
    AllowOverride None
    Require all granted
</Directory>

Why it matters: DocumentRoot answers “where are the files?” The directory block answers “under what rules may Apache serve them?”

Many security and functionality bugs frequently arise from this configuration. Teams enable broad access in a rush, then forget to tighten it later. Or they deny too much and break static assets, uploads, or framework front controllers.

Loading modules

Apache is modular. LoadModule activates optional capabilities.

LoadModule rewrite_module modules/mod_rewrite.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

Why it matters: Apache won’t assume you want rewriting, TLS, or reverse proxy features. If the module isn’t loaded, related directives either fail validation or do not work the way you expect.

On many systems, distro packaging manages module loading in separate files. That’s one more reason to learn the layout your platform uses instead of assuming every change belongs in the main config.

Logging directives

Logs are where Apache tells you what it did.

ErrorLog logs/error_log
CustomLog logs/access_log combined

Why it matters: when a config change breaks traffic, the access log shows what requests arrived and the error log shows what Apache rejected, denied, or failed to load. If those logs aren’t where your team expects, troubleshooting slows down immediately.

In containerized deployments, it often makes more sense to send logs to stdout and stderr through the image’s runtime conventions rather than relying on local file rotation inside the container.

Directory index and default documents

DirectoryIndex controls which files Apache should serve when a client requests a directory.

DirectoryIndex index.html index.php

Why it matters: this directive sounds minor, but it changes the first page users see and can affect framework routing assumptions. Missing or incorrect index settings often produce confusing directory responses or unintended exposure of directory listings.

URL rewriting and redirect support

When mod_rewrite is enabled, RewriteEngine, RewriteCond, and RewriteRule become core tools for redirects and application routing.

RewriteEngine On
RewriteRule ^old-page$ /new-page [R=301,L]

Why it matters: rewrite logic is powerful and dangerous. It solves canonicalization, HTTPS enforcement, front-controller routing, and legacy URL migration. It also creates loops, hard-to-debug chains, and hidden complexity if you pile everything into one file.

Access control in modern Apache

In Apache 2.4 style config, Require is central.

<Directory "/var/www/private">
    Require all denied
</Directory>

Or:

<Location "/health">
    Require all granted
</Location>

Why it matters: this is how you make intent explicit. Public endpoints should be public on purpose. Internal endpoints should be restricted on purpose. Apache config is one place where “default open” often becomes technical debt.

Operational commands you should always use

Any time you change config, validate first.

• Syntax validation with apachectl -t
• Reload after success with your platform’s reload or graceful command
• Module verification with httpd -M or apache2ctl -M

Those commands matter more than any single directive because they shorten the path from edit to safe deployment.

The best Apache admins aren’t the ones who type config fastest. They’re the ones who validate every change before it touches live traffic.

Common Configuration Tasks and Real World Examples

Theory helps, but Apache starts to make sense when you see working blocks that solve common problems. These examples are intentionally practical. They’re the kinds of snippets teams keep in internal runbooks and reuse across environments.

Hosting more than one site with virtual hosts

A single Apache instance can serve multiple sites by using <VirtualHost> blocks.

<VirtualHost *:80>
    ServerName app.example.internal
    DocumentRoot "/var/www/app/public"

    <Directory "/var/www/app/public">
        AllowOverride None
        Require all granted
    </Directory>

    ErrorLog logs/app_error.log
    CustomLog logs/app_access.log combined
</VirtualHost>

<VirtualHost *:80>
    ServerName admin.example.internal
    DocumentRoot "/var/www/admin/public"

    <Directory "/var/www/admin/public">
        AllowOverride None
        Require all granted
    </Directory>

    ErrorLog logs/admin_error.log
    CustomLog logs/admin_access.log combined
</VirtualHost>

This is the cleanest way to separate sites, logs, and directory rules without running multiple Apache processes. It also makes ownership easier. One app can change its virtual host file without touching the other.

A mistake I see often is putting unrelated sites into one giant vhost with conditional rewrite rules. That works until it doesn’t. Separate virtual hosts are easier to review, easier to roll back, and easier to test.

Simple redirects without overcomplicating things

Not every redirect needs mod_rewrite. If you’re just moving one path, Redirect is easier to read.

Redirect 301 /docs /documentation

Use the simpler tool when it fits. The trade-off is straightforward. Redirect is easy to maintain. RewriteRule is more flexible when you need conditions, patterns, or scheme enforcement.

Enforcing HTTPS and a canonical host

For broader request normalization, rewriting is the right tool.

<VirtualHost *:80>
    ServerName example.internal
    RewriteEngine On

    RewriteCond %{HTTPS} !=on [OR]
    RewriteCond %{HTTP_HOST} !^www\.example\.internal$ [NC]
    RewriteRule ^ https://www.example.internal%{REQUEST_URI} [R=301,L]
</VirtualHost>

That block gives you one consistent entry point. It reduces duplicate URL variants, avoids mixed assumptions in applications, and keeps external behavior predictable.

The catch is that rewrite logic can become unreadable if you stack too many concerns together. Keep canonicalization rules near the top of the vhost and keep application-specific routing elsewhere.

Basic TLS virtual host

A minimal TLS-enabled host usually looks like this:

<VirtualHost *:443>
    ServerName www.example.internal
    DocumentRoot "/var/www/app/public"

    SSLEngine on
    SSLCertificateFile "/path/to/cert.pem"
    SSLCertificateKeyFile "/path/to/key.pem"

    <Directory "/var/www/app/public">
        AllowOverride None
        Require all granted
    </Directory>

    ErrorLog logs/ssl_app_error.log
    CustomLog logs/ssl_app_access.log combined
</VirtualHost>

This isn’t a full hardened TLS policy, but it shows the moving parts clearly. Apache needs the SSL module loaded, a listener for the secure vhost, and valid certificate paths.

One operational lesson matters here. Keep certificate paths and site logic separate if you can. Teams that mix TLS material, redirects, app routing, and proxy rules into one dense vhost create painful incident reviews later.

Reverse proxying to an app server

Apache still shows up in front of application servers and internal services. A basic reverse proxy is often enough:

<VirtualHost *:80>
    ServerName api.example.internal

    ProxyPreserveHost On
    ProxyPass / http://backend-app/
    ProxyPassReverse / http://backend-app/

    ErrorLog logs/api_proxy_error.log
    CustomLog logs/api_proxy_access.log combined
</VirtualHost>

This pattern is useful when Apache handles front-end concerns while the app runs elsewhere. It also gives you one layer where logging, access controls, and URL normalization can stay consistent.

If you’re validating replayed traffic or trying to mirror production request patterns in a test stack, start by understanding the requests you’re receiving. Capturing request behavior before tuning proxy config is usually more useful than guessing. The GoReplay guide to capture HTTP requests is a practical reference for that workflow.

A cleaner layout than one giant file

A maintainable Apache config often looks more like this:

• Main config file for global defaults, module loading, and include statements
• One file per site in a sites-enabled or conf.d directory
• One file per concern for logging, status endpoints, proxy settings, or TLS snippets

That separation helps when the site grows. It also reduces the blast radius of mistakes. A broken redirect rule in one vhost file is a smaller review and a faster rollback than a tangled edit in the middle of a giant httpd.conf.

A quick visual walk-through helps if you’re mapping these examples to your own server layout.

Rewrite rules need discipline

Apache gives you enough rope to build elegant routing or a knot you can’t untangle.

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^ index.php [L]

That front-controller pattern is common in PHP applications. It’s also where teams start adding exception after exception until no one can reason about the final behavior.

When rewrite sections start to feel like a miniature programming language, stop and refactor. Split concerns. Add comments. Keep the request flow legible.

If you can’t explain a rewrite chain aloud in under a minute, it’s too complex for an on-call engineer at 2 a.m.

Tuning Apache for Performance and Monitoring

A common failure pattern looks like this. The app passed local tests, the container starts cleanly, and then a replay run or traffic spike hits Apache harder than production ever did because the defaults were never revisited. httpd.conf is where that gets fixed.

MPM choice changes behavior

Apache’s Multi-Processing Module sets the server’s execution model. In practice, that choice affects memory use, connection reuse, and how well the server behaves under keep-alive traffic.

MPM	Best fit	Trade-off
prefork	Older module compatibility	Simple process model, higher memory cost per connection
worker	Threaded workloads	Better concurrency, requires thread-safe modules and application code
event	Keep-alive heavy traffic	Frees workers more efficiently, still needs testing with your proxy and app stack

On modern Linux deployments, event is often the first place to start. It handles idle keep-alive connections more efficiently than prefork, which matters when Apache sits behind a load balancer or reverse proxy and clients hold connections open. If you’re running a legacy PHP setup that still depends on older assumptions, compatibility may force a different choice.

The directives teams usually tune first

Three directives shape Apache’s behavior under load faster than almost anything else:

• KeepAlive decides whether clients can reuse connections.
• KeepAliveTimeout decides how long Apache waits before closing an idle keep-alive connection.
• MaxRequestWorkers caps how many requests Apache serves at the same time.

Those settings matter even more in replay and load-test environments than in day-to-day production. Axelerant’s analysis points out that Apache guidance often leaves teams to create their own operating rules for replay-heavy or high-volume scenarios, especially when they need to balance concurrency against session behavior and application realism.

That trade-off shows up quickly in containers. Raising MaxRequestWorkers inside a pod with a tight memory limit can turn a tuning win into an OOM kill. Lowering KeepAliveTimeout can free workers faster, but it can also make your test environment behave less like production if real clients reuse connections heavily.

A small starting point looks like this:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 2
MaxRequestWorkers 300

Do not copy those values blindly. Validate them against your MPM, available memory, upstream latency, and whether Apache is serving static files, proxying to an app tier, or both.

Monitoring with mod_status

mod_status gives you a live view of what Apache is doing right now. It is one of the fastest ways to confirm whether a tuning change helped or just moved the bottleneck.

A basic setup looks like this:

LoadModule status_module modules/mod_status.so
ExtendedStatus On

<Location /server-status>
    SetHandler server-status
    Require local
</Location>

If the module is loaded, httpd -M or apache2ctl -M should show status_module (shared).

Use it carefully in production. Expose /server-status only to localhost, a VPN, or a tightly controlled admin network. In Kubernetes or Docker, I usually keep it internal and scrape it through a sidecar, port-forward, or private ingress rule instead of publishing it broadly.

Use metrics to validate test fidelity

Performance tuning without observation is guesswork. During replay tests, watch whether Apache’s worker usage, request rate, and connection handling match the behavior you expect from production. The GoReplay monitoring guide for traffic replay observability is a useful reference if you’re building out that workflow.

A few signals are worth watching closely:

• Busy workers climb and stay pinned. Apache has hit a concurrency limit, or upstream services are responding too slowly.
• Idle workers stay high during a supposed stress test. The replay is not exercising the path you think it is.
• Request counts rise but latency still worsens. Apache may be accepting work faster than the application tier can finish it.

The goal is not to make Apache survive an artificial benchmark. The goal is to tune httpd.conf so the server behaves predictably under the traffic pattern you run.

Security Backups and Modern Deployments

At 2 a.m., nobody wants to debug an Apache outage by SSHing into a container, opening a half-edited httpd.conf, and guessing which change was live before the last restart. Good security practice reduces the chance of that situation. Good deployment practice makes recovery routine when it still happens.

For Apache, security is broader than access rules and TLS settings. It also includes whether your team can review config changes, restore a known-good version, and reproduce the same behavior across VMs, containers, and test environments. If the config on disk cannot be traced back to a commit and a deployment, you have configuration drift, and drift turns small mistakes into production incidents.

A practical baseline looks like this:

• Disable modules you do not use. Every loaded module adds code paths, maintenance overhead, and possible exposure.
• Make directory access explicit. Grant access to public paths on purpose. Deny sensitive paths on purpose.
• Prefer central config over scattered overrides when you need predictable behavior across environments.

.htaccess still has valid use cases, especially on shared hosting or in teams that cannot touch the main server config. In managed environments, though, it usually creates more audit work and more surprises. Centralized config is easier to review, test, and reproduce.

Treat config like code

If httpd.conf only exists on a running server, you do not really control it. Put Apache config in Git, review it through pull requests, and tag versions that map to known infrastructure states.

That gives you concrete operational benefits:

Practice	Why it helps
Version control	You can diff changes and restore a known-good config quickly
Peer review	Another engineer can catch bad directive scope, weak access rules, or risky includes
Environment templates	Staging and production stay aligned instead of drifting through manual edits

A backup is more than a copied file. It is recoverable history with context, ownership, and a deployment path your team can repeat under pressure.

The best backup for Apache config is a repository plus a repeatable deployment path, not a mystery zip file from six months ago.

Containers change how Apache config should be managed

A significant share of Apache deployments now run inside containers, but a lot of httpd.conf advice still assumes a long-lived host, manual edits under /etc/httpd, and a filesystem that survives forever. That gap matters in practice. In a containerized setup, the file may be baked into the image, mounted from a volume, or generated at startup, and each option changes how you secure, review, and roll back config.

A few rules hold up well:

• Do not patch running containers by hand unless you are debugging and you know the change is temporary.
• Mount config intentionally if your platform uses bind mounts, volumes, or Kubernetes ConfigMaps.
• Keep the base config small and load environment-specific fragments through Include or IncludeOptional.
• Be careful with entrypoint templating when you generate config from environment variables at startup.

I usually see two container patterns work well. One is an image with a stable Apache baseline, plus mounted virtual host or environment-specific snippets. The other is a fully mounted config tree managed by the orchestration layer. Both are defensible. The failure mode is mixing image-baked edits, shell scripts that patch files on boot, and ad hoc mounted overrides until nobody can tell which directive wins.

Kubernetes adds another layer of trade-offs. ConfigMaps are convenient and easy to version alongside manifests, but they also force you to think about rollout behavior and reload strategy. Some teams expect Apache to pick up changes automatically and then discover that nothing changed until the pod restarted or received an explicit reload. Docker Compose has a similar trap. Bind mounts are straightforward, but they can mask parts of the image filesystem and unintentionally obscure included config files you expected Apache to load.

Modern deployment habits and backup strategy intersect. For a VM, a backup might mean restoring /etc/httpd from source control and reloading the service. For a containerized service, recovery often means rebuilding or redeploying the workload from the last known-good image and config revision. Same goal. Different operating model. Teams that account for that difference usually recover faster and make fewer last-minute edits that create security holes.

Conclusion From Configuration File to Control Panel

A bad httpd.conf rarely fails at a convenient time. It fails during a deploy, under replayed traffic, or in the middle of an incident when nobody wants to guess which include file changed request handling.

The practical shift is to stop treating Apache configuration as a static server file and start treating it like an interface between application behavior, infrastructure, and operations. On a long-lived VM, that means clear ownership, version control, and predictable reloads. In containers, it also means deciding what belongs in the image, what gets mounted at runtime, and how config changes roll out under orchestration. The teams that do this well are not memorizing directives. They are building a change process that makes Apache predictable under load.

That matters even more when load testing is part of delivery. Synthetic checks can confirm that Apache starts. Replayed production traffic shows whether your keepalive settings, proxy timeouts, header rules, and worker limits behave the way you expect when requests arrive in the shape they do in production. That is the gap many basic httpd.conf tutorials miss.

Treat the config file like code, but also treat it like an operational contract.

If you’re testing Apache-backed applications with real traffic patterns instead of synthetic guesswork, GoReplay is worth a look. It captures live HTTP traffic and replays it into test environments so you can validate config changes, application releases, and performance behavior before they hit production.