January 2019
M T W T F S S
« Dec    
 123456
78910111213
14151617181920
21222324252627
28293031  

Categories

WordPress Quotes

People often say that this or that person has not yet found himself. But the self is not something one finds, it is something one creates.
Thomas S. Szasz

Recent Comments

January 2019
M T W T F S S
« Dec    
 123456
78910111213
14151617181920
21222324252627
28293031  

Short Cuts

2012 SERVER (64)
2016 windows (9)
AIX (13)
Amazon (34)
Ansibile (19)
Apache (133)
Asterisk (2)
cassandra (2)
Centos (209)
Centos RHEL 7 (261)
chef (3)
cloud (2)
cluster (3)
Coherence (1)
DB2 (5)
DISK (25)
DNS (9)
Docker (28)
Eassy (11)
ELKS (1)
EXCHANGE (3)
Fedora (6)
ftp (5)
GIT (3)
GOD (2)
Grub (1)
Hacking (10)
Hadoop (6)
horoscope (23)
Hyper-V (10)
IIS (15)
IPTABLES (15)
JAVA (7)
JBOSS (32)
jenkins (1)
Kubernetes (2)
Ldap (5)
Linux (189)
Linux Commands (167)
Load balancer (5)
mariadb (14)
Mongodb (4)
MQ Server (24)
MYSQL (84)
Nagios (5)
NaturalOil (13)
Nginx (31)
Ngix (1)
openldap (1)
Openstack (6)
Oracle (34)
Perl (3)
Postfix (19)
Postgresql (1)
PowerShell (2)
Python (3)
qmail (36)
Redis (12)
RHCE (28)
SCALEIO (1)
Security on Centos (29)
SFTP (1)
Shell (64)
Solaris (58)
Sql Server 2012 (4)
squid (3)
SSH (10)
SSL (14)
Storage (1)
swap (3)
TIPS on Linux (28)
tomcat (60)
Uncategorized (29)
Veritas (2)
vfabric (1)
VMware (28)
Weblogic (38)
Websphere (71)
Windows (19)
Windows Software (2)
wordpress (1)
ZIMBRA (17)

WP Cumulus Flash tag cloud by Roy Tanck requires Flash Player 9 or better.

Who's Online

12 visitors online now
1 guests, 11 bots, 0 members

Hit Counter provided by dental implants orange county

Mod Security

Mod Security

Mod security has a default configuration file, and comes with a core rule set. The configuration works with include files which work for the modsecurity part like this:

httpd.conf
|
|– default-server.conf . . . . . . . . . set up the default server that replies to non-virtual-host requests
| `–conf.d/mod_security2.conf . . . . enable mod-security default configuration
|
`–conf.d/modsecurity/*.conf . . . . . . add the core rule set

Since this include structure is not enabled by default (because the core rule set is not enabled by default) we have to include the core rule set manually.

Create the correct directories and copy the core rule set config files to this directory:

reverseproxy:/usr/share/doc/packages/apache2-mod_security2/rules # mkdir /etc/apache2/conf.d/modsecurity
reverseproxy:/usr/share/doc/packages/apache2-mod_security2/rules # cp *.conf /etc/apache2/conf.d/modsecurity
reverseproxy:/usr/share/doc/packages/apache2-mod_security2/rules # cd /etc/apache2/conf.d/modsecurity

reverseproxy:/etc/apache2/conf.d/modsecurity # ll
-rw-r–r– 1 root root 12325 Jan 31 14:03 modsecurity_crs_10_config.conf
-rw-r–r– 1 root root 5164 Jan 31 14:03 modsecurity_crs_20_protocol_violations.conf
-rw-r–r– 1 root root 3538 Jan 31 14:03 modsecurity_crs_21_protocol_anomalies.conf
-rw-r–r– 1 root root 2496 Jan 31 14:03 modsecurity_crs_23_request_limits.conf
-rw-r–r– 1 root root 6399 Jan 31 14:03 modsecurity_crs_30_http_policy.conf
-rw-r–r– 1 root root 2720 Jan 31 14:03 modsecurity_crs_35_bad_robots.conf
-rw-r–r– 1 root root 28726 Jan 31 14:03 modsecurity_crs_40_generic_attacks.conf
-rw-r–r– 1 root root 2463 Jan 31 14:03 modsecurity_crs_45_trojans.conf
-rw-r–r– 1 root root 8268 Jan 31 14:03 modsecurity_crs_50_outbound.conf

Add the include line for the core rule set in the httpd.conf:

# Include Mod Security Core Rule Set
Include /etc/apache2/conf.d/modsecurity/*.conf

Now we will configure the config files themselves to run modsecurity first in DetectionOnly
mode to prevent the risk for false positives. We also set the logfiles correctly:

vi /etc/apache2/conf.d/mod_security2.conf:
# Basic configuration options
#SecRuleEngine On
SecRuleEngine DetectionOnly

vi /etc/apache2/conf.d/modsecurity/modsecurity_crs_10_config.conf:
SecRuleEngine DetectionOnly
SecAuditLog /var/log/apache2/modsec_audit.log
SecDebugLog /var/log/apache2/modsec_debug.log
SecDebugLogLevel 3

Now restart apache:

reverseproxy:/var/log/apache2 # /etc/init.d/apache2 start
Starting httpd2 (prefork) [Mon Jan 31 14:30:35 2011] [warn] worker http://10.10.12.20/start already used by another worker
[Mon Jan 31 14:30:35 2011] [warn] worker http://10.10.12.20/start already used by another worker

Documentation Core Rule Set

Core Rule Set Structure & Usage
====================================

To activate the rules for your web server installation:

1) You may want to edit and customize modsecurity_crs_10_config.conf.
Additionally you may want to edit modsecurity_crs_30_http_policy.conf
which enforces an application specific HTTP protocol usage.

2) Add the following line to your httpd.conf (assuming
you’ve placed the rule files into conf/modsecurity/):

Include conf/modsecurity/*.conf

3) Restart web server.

4) Make sure your web sites are still running fine.

Core Rule Set Content
=========================

In order to provide generic web applications protection, the Core Rule Set
uses the following techniques:

1. HTTP protection – detecting violations of the HTTP protocol and a locally
defined usage policy.

2. Common Web Attacks Protection – detecting common web application security
attack.

3. Automation detection – Detecting bots, crawlers, scanners and other surface
malicious activity.

4. Trojan Protection – Detecting access to Trojans horses.

5. Errors Hiding – Disguising error messages sent by the server

In addition the rule set also hints at the power of ModSecurity beyond
providing security by reporting access from the major search engines to your
site.

HTTP Protection – This first line of protection ensures that all abnormal HTTP
requests are detected. This line of defense eliminates a large number of
automated and non targeted attacks as well as protects the web server itself.
Common Web Attacks Protection Rules on the second level address the common web
application security attack methods. These are the issues that can appear in
any web application. Some of the issues addressed are:

– SQL Injection
– Cross-Site Scripting (XSS)
– OS Command execution
– Remote code inclusion
– LDAP Injection
– SSI Injection
– Information leak
– Buffer overflows
– File disclosure

Automation Detection – Automated clients are both a security risk and a
commercial risk. Automated crawlers collect information from your site, consume
bandwidth and might also search for vulnerabilities on the web site. Automation
detection is especially useful for generic detection of comments spam.

Trojan Protection – ModSecurity Core Rule Set detects access to back doors
installed on a web server. This feature is very important in a hosting
environment when some of this backdoors may be uploaded in a legitimate way and
used maliciously. In addition the Core Rule Set includes a hook for adding
an Anti-Virus program such as ClamAV for checking file uploads.

Errors Hiding – If all fails, the Core Rule Set will detect errors sent by
the web server. Detecting and blocking errors prevents attackers from
collecting reconnaissance information about the web application and also server
as a last line of defense in case an attack was not detected eariler.

Few Word of Caution
——————-

As with every new technology, using the ModSecurity Core Rule Set requires some caution:

– Every Rule Set can have false positive in new environments and any new
installation should initially use the log only Rule Set version or if no such
version is available, set ModSecurity to Detection only using the SecRuleEngine
DetectionOnly command.

After running ModSecurity in a detection only mode for a while review the evens
generated and decide if any modification to the rule set should be made before
moving to protection mode.

From the mod security manual:

SecRuleEngine

Description: Configures the rules engine.
Syntax: SecRuleEngine On|Off|DetectionOnly
Example Usage: SecRuleEngine On
Processing Phase: Any
Scope: Any
Version: 2.0.0
Dependencies/Notes: This directive can also be controlled by the ctl action (ctl:ruleEngine=off) for per rule processing.
Possible values are:
* On – process rules.
* Off – do not process rules.
* DetectionOnly – process rules but never intercept transactions, even when rules are configured to do so.

Mod Security Handling False Positives
Mod security is now configured as detection only. For now, we keep it like this, closely monitoring the mod security logfiles for false positives. When we are sure there are no more false positives (or at least nothing our customers will notice) we can set the SecRuleEngine to On.

This blog also explains how to deal with false positives: Handling False Positives
Mod Security Troubleshooting

Starting httpd2 (prefork) [Mon Jan 31 14:20:51 2011] [warn] worker http://10.10.12.20/start already used by another worker
[Mon Jan 31 14:20:51 2011] [warn] worker http://10.10.12.20/start already used by another worker
Syntax error on line 53 of /etc/apache2/conf.d/modsecurity/modsecurity_crs_10_config.conf:
Invalid command ‘SecRuleEngine’, perhaps misspelled or defined by a module not included in the server configuration

The command line was:
/usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -DSSL

‘Solution:’ The module mod_security is not enabled. Check for the module with the command ‘httpd2 -M’. If the module is really not there, add the module in /etc/sysconfig/apache2.

reverseproxy:/var/log/apache2 # /etc/init.d/apache2 restart
[Mon Jan 31 14:29:23 2011] [warn] worker http://10.10.12.20/start already used by another worker
[Mon Jan 31 14:29:23 2011] [warn] worker http://10.10.12.20/start already used by another worker
Syntax error on line 191 of /etc/apache2/conf.d/modsecurity/modsecurity_crs_10_config.conf:
ModSecurity: Failed to open the audit log file: /srv/www/logs/modsec_audit.log

‘Solution:’ The directory specified for the logs does not exist. Create the directory with this command:

reverseproxy:/var/log/apache2 # mkdir -p /srv/www/logs/

or change the location to /var/log/apache2. Of course, the same message can be displayed for /srv/www/logs/modsec_debug.log.
Testing Mod Security
You can test if mod security is running correctly by going to the index file of your website by ip-address and adding ‘?file=/etc/passwd’ to the url:

https://10.10.10.20/start/index.html?file=/etc/passwd

This will be noticed, and displayed in the log (not stopped, remember, we’re running in DetectionOnly mode):

less modsec_debug.log

[31/Jan/2011:15:46:31 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98feb488][/start/0100_NavigationPublic.html][2] Warning. Pattern match “^[\d\.]+$” at REQUEST_HEADERS:Host. [
file “/etc/apache2/conf.d/modsecurity/modsecurity_crs_21_protocol_anomalies.conf”] [line “60”] [id “960017”] [msg “Host header is a numeric IP address”] [severity “CRITICAL”] [ta
g “PROTOCOL_VIOLATION/IP_HOST”]
[31/Jan/2011:15:46:42 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98fe2908][/start/index.html][2] Warning. Pattern match “^[\d\.]+$” at REQUEST_HEADERS:Host. [file “/etc/apach
e2/conf.d/modsecurity/modsecurity_crs_21_protocol_anomalies.conf”] [line “60”] [id “960017”] [msg “Host header is a numeric IP address”] [severity “CRITICAL”] [tag “PROTOCOL_VIOL
ATION/IP_HOST”]
[31/Jan/2011:15:46:42 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98fe2908][/start/index.html][2] Warning. Pattern match “(?:\b(?:\.(?:ht(?:access|passwd|group)|www_?acl)|glob
al\.asa|httpd\.conf|boot\.ini)\b|\/etc\/)” at ARGS:file. [file “/etc/apache2/conf.d/modsecurity/modsecurity_crs_40_generic_attacks.conf”] [line “114”] [id “950005”] [msg “Remote
File Access Attempt”] [data “/etc/”] [severity “CRITICAL”] [tag “WEB_ATTACK/FILE_INJECTION”]
[31/Jan/2011:15:46:42 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98fe2908][/start/index.html][2] Warning. Pattern match “(?:\b(?:(?:n(?:et(?:\b\W+?\blocalgroup|\.exe)|(?:map|
c)\.exe)|t(?:racer(?:oute|t)|elnet\.exe|clsh8?|ftp)|(?:w(?:guest|sh)|rcmd|ftp)\.exe|echo\b\W*?\by+)\b|c(?:md(?:(?:32)?\.exe\b|\b\W*?\/c)|d(?:\b\W*?[\\/]|\W*?\.\.)|hmod.{0,40}?\+.
{0,3}x))|[\;\|\`]\W*? …” at ARGS:file. [file “/etc/apache2/conf.d/modsecurity/modsecurity_crs_40_generic_attacks.conf”] [line “133”] [id “950006”] [msg “System Command Injectio
n”] [data “/passwd”] [severity “CRITICAL”] [tag “WEB_ATTACK/COMMAND_INJECTION”]

less modsec_audit.log:

Message: Warning. Pattern match “^[\d\.]+$” at REQUEST_HEADERS:Host. [file “/etc/apache2/conf.d/modsecurity/modsecurity_crs_21_protocol_anomalies.conf”] [line “60”] [id “960017”]
[msg “Host header is a numeric IP address”] [severity “CRITICAL”] [tag “PROTOCOL_VIOLATION/IP_HOST”]
Apache-Handler: proxy-server
Stopwatch: 1296487473036980 19376 (997 2882 -)
Producer: ModSecurity for Apache/2.5.6 (http://www.modsecurity.org/); core ruleset/1.6.1.
Server: Apache/2.2.10 (Linux/SUSE)

Mod Security Resources

http://www.modsecurity.org/
http://www.modsecurity.org/documentation/faq.html
http://www.modsecurity.org/documentation/modsecurity-apache/2.5.12/html-multipage/introduction.html
Install Modsecurity
Install core rule set

 

MY SET OF RULES TO DEFEND THE WEB SERVER
SecFilterEngine On

# Make sure that URL encoding is valid
SecFilterCheckURLEncoding On

# Unicode encoding check
SecFilterCheckUnicodeEncoding On

# Only allow bytes from this range
SecFilterForceByteRange 0 255

# Only log actionable requests
SecAuditEngine RelevantOnly

# The name of the audit log file
SecAuditLog /var/log/apache2/audit_log

# Debug level set to a minimum
SecFilterDebugLog /var/log/apache2/modsec_debug_log
SecFilterDebugLevel 2

# Should mod_security inspect POST payloads
SecFilterScanPOST On

# By default log and deny suspicious requests
# with HTTP status 500
SecFilterDefaultAction “deny,log,status:500”

# Add custom secfilter rules here

Apache troubleshooting commands

Apache troubleshooting commands

Commands

ps aux | grep httpd
pstree -p | grep httpd

strace -f -o trace.txt /etc/rc.d/init.d/httpd start

Sometime Apache process, keeps on execution (Seems like Hangs), so generally trying to get the exact PHP file that is running by Apache Process, So here is my Try.

I used Strace to get the opened files by the apache process. (Get PID of
Apache process that is taking time, though you can also get it From top command)

# pstree -p -n | grep http
(This will show each files that is being processed by that Apache Proc)

# strace -p
The list of files could also be get using lsof, but that could not be of full use, as you need the files continuus

Counting Hits from Web Server Access log
# awk ‘{print $1}’ /opt/rmohan.com/access_log | grep -vE ‘^:|^common|^-‘ | sort | uniq -c | sort -nr > /var/www/reports/ips/rmohan.txt

or

# awk ‘$1>10000 {print $1}’ /opt/rmohan.com/access_log | uniq -c | sort -nr > /var/www/reports/ips/rmohan.txt

Restart Apache and check whether the modules are running by issuing the ‘httpd2 -M’ command:

reverseproxy:/var/log/apache2 # httpd2 -M
Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
authz_host_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
authz_groupfile_module (shared)
authn_file_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgi_module (shared)
dir_module (shared)
include_module (shared)
log_config_module (shared)
mime_module (shared)
negotiation_module (shared)
setenvif_module (shared)
status_module (shared)
userdir_module (shared)
asis_module (shared)
cache_module (shared)
disk_cache_module (shared)
imagemap_module (shared)
proxy_module (shared)
proxy_connect_module (shared)
proxy_http_module (shared)
rewrite_module (shared)
ssl_module (shared)
unique_id_module (shared)
authz_default_module (shared)
security2_module (shared)
Syntax OK

Monitor Your Website in Real-Time with Apachetop

As a webmaster, I’ve often wanted to be able to see real-time hits as they arrive. Sure, Google Analytics is a wonderful package for looking at trends over time, but there’s a delay of a few hours there, and you really can’t see data like requests per second or total bytes.

This is where the apachetop utility comes in. It’s a very simple command line utility that you can use to monitor traffic real-time. It accomplishes this by parsing the apache logfiles and displaying meaningful output to the screen.

Using Apachetop

Once you’ve installed the utility (instructions below), you can launch it by simply running apachetop from the command line. Since apachetop sometimes defaults to the wrong directory for the logfiles, you can pass in the -f parameter to specify the location of the logfile. This is also helpful when you have many virtual hosts on the same box.

apachetop -f /var/www/vhosts/howtogeek.com/statistics/logs/access_log

This is what you’ll see after a few requests have come in:

Monitoring Timeframe

The first thing to note is that the default time range for data shown is 30 seconds, so don’t expect the total counts to continue to climb forever. You can change this by passing in a few different arguments.

apachetop -H hits (Will display stats on the last x number of hits)

apachetop -T secs (Will display stats on the last x number of seconds)

I’ve been using a range of 5-10 minutes in my testing, and it really shows some useful feedback. There’s other options you can try out as well.

Filters

The next thing to note is that you can filter what gets shown in the view. To access the filters, use the f key, and you should see a small line pop up.

Hit the a key to add a filter and the line should switch. Now you can choose to filter by URL, referrer, or host.

I’m going to choose URL by hitting the u key. The filter dialog will show up near the bottom:

Since all of my articles are under the subdirectory /howto/, I’m going to enter that. Now apachetop will only show the hits relevant to hits to the articles, instead of every hit for every image.

Viewing Request Details

If you use the up/down keys, you’ll notice the cursor move up and down to allow you to select a request. (notice the * char)

If you hit the Right arrow key, you’ll be taken to the details page for that request. From here you can see the actual hosts hitting your site, as well as the referrers. I’m not going to show the hosts, since I don’t want to give out user’s IP address, but you can see the referrer here:

To go back to the list, just use the Left arrow key.

Switch Between Hosts, Referrers and URLs

If you use the d key, you can easily switch between the different views.

For instance, here I can see what traffic StumbleUpon is sending me, and then I can use the details view(right arrow) to see the exact articles that are getting hit from stumbleupon.

Help

At any point you can hit the ? or the h keys to take you to the help screen, which will give you a quick view of all the options.

I find the sort by very useful.

Installing on Ubuntu

sudo apt-get install apachetop

Installing from Source on CentOS

wget http://www.webta.org/apachetop/apachetop-0.12.6.tar.gz

yum install readline-devel

yum install ncurses-devel

tar xvzf apachetop-0.12.6.tar.gz

cd apachetop-0.12.6

./configure

make

The binary can be found in src/apachetop, and you can copy it anywhere you’d like.

Installing from Source on Ubuntu

wget http://www.webta.org/apachetop/apachetop-0.12.6.tar.gz

sudo apt-get install ncurses-dev

sudo apt-get install libreadline5-dev

tar xvzf apachetop-0.12.6.tar.gz

cd apachetop-0.12.6

./configure

make

Forward Proxy and reverse proxy

Forward Proxy and reverse proxy

Proxy server types and uses for HTTP Server (powered by Apache)

This topic provides information about proxy server types and uses.
Important: Information for this topic supports the latest PTF levels for HTTP Server for iSeries . It is recommended that you install the latest PTFs to upgrade to the latest level of the HTTP Server for iSeries. Some of the topics documented here are not available prior to this update. See http://www.ibm.com/servers/eserver/iseries/software/http/services/service.htm for more information.

Proxy servers receive requests intended for other servers and then act to fulfill, forward, redirect, or reject the requests. Exactly which service is carried out for a particular request is based on a number of factors which include: the proxy server’s capabilities, what is requested, information contained in the request, where the request came from, the intended destination, and in some cases, who sent the request.
The two most attractive reasons to use a proxy server are its ability to enhance network security and lessen network traffic. A proxy server enhances network security by providing controls for receiving and forwarding (or rejecting) requests between isolated networks, for example, forwarding requests across a firewall. A proxy server lessens network traffic by rejecting unwanted requests, forwarding requests to balance and optimize server workload, and fulfilling requests by serving data from cache rather than unnecessarily contacting the true destination server.
HTTP Server (powered by Apache) has proxy server capabilities built in. Activating these services is simply a matter of configuration. This topic explains three common proxy concepts: forward proxy, reverse proxy, and proxy chaining.
Parent topic: Concepts of functions of HTTP Server

Forward proxy

A forward proxy is the most common form of a proxy server and is generally used to pass requests from an isolated, private network to the Internet through a firewall. Using a forward proxy, requests from an isolated network, or intranet, can be rejected or allowed to pass through a firewall. Requests may also be fulfilled by serving from cache rather than passing through the Internet. This allows a level of network security and lessens network traffic.
A forward proxy server will first check to make sure a request is valid. If a request is not valid, or not allowed (blocked by the proxy), it will reject the request resulting in the client receiving an error or a redirect. If a request is valid, a forward proxy may check if the requested information is cached. If it is, the forward proxy serves the cached information. If it is not, the request is sent through a firewall to an actual content server which serves the information to the forward proxy. The proxy, in turn, relays this information to the client and may also cache it, for future requests.

Forward Proxy

The above image shows a forward proxy configuration. An intranet client initiates a request that is valid but is not cached on Server A (Proxy Server). The request is sent through the firewall to the Internet server, Server B (Content Server), which has the information the client is requesting. The information is sent back through the firewall where it is cached on Server A and served to the client. Future requests for the same information will be fulfilled by the cache, lessening network traffic (proxy caching is optional and not necessary for forward proxy to function on your HTTP Server).

For information on how to configure a forward proxy, see Set up forward proxy for HTTP Server (powered by Apache).


Reverse proxy

A reverse proxy is another common form of a proxy server and is generally used to pass requests from the Internet, through a firewall to isolated, private networks. It is used to prevent Internet clients from having direct, unmonitored access to sensitive data residing on content servers on an isolated network, or intranet. If caching is enabled, a reverse proxy can also lessen network traffic by serving cached information rather than passing all requests to actual content servers. Reverse proxy servers may also balance workload by spreading requests across a number of content servers. One advantage of using a reverse proxy is that Internet clients do not know their requests are being sent to and handled by a reverse proxy server. This allows a reverse proxy to redirect or reject requests without making Internet clients aware of the actual content server (or servers) on a protected network.

Reverse proxy

A reverse proxy server will first check to make sure a request is valid. If a request is not valid, or not allowed (blocked by the proxy), it will not continue to process the request resulting in the client receiving an error or a redirect. If a request is valid, a reverse proxy may check if the requested information is cached. If it is, the reverse proxy serves the cached information. If it is not, the reverse proxy will request the information from the content server and serve it to the requesting client. It also caches the information for future requests.

The above image shows a reverse proxy configuration. An Internet client initiates a request to Server A (Proxy Server) which, unknown to the client, is actually a reverse proxy server. The request is allowed to pass through the firewall and is valid but is not cached on Server A. The reverse proxy (Server A) requests the information from Server B (Content Server), which has the information the Internet client is requesting. The information is served to the reverse proxy, where it is cached, and relayed through the firewall to the client. Future requests for the same information will be fulfilled by the cache, lessening network traffic and load on the content server (proxy caching is optional and not necessary for proxy to function on your HTTP Server). In this example, all information originates from one content server (Server B).
For information on how to configure a reverse proxy, see Set up reverse proxy for HTTP Server (powered by Apache).

Proxy chaining

A proxy chain uses two or more proxy servers to assist in server and protocol performance and network security. Proxy chaining is not a type of proxy, but a use of reverse and forward proxy servers across multiple networks. In addition to the benefits to security and performance, proxy chaining allows requests from different protocols to be fulfilled in cases where, without chaining, such requests would not be possible or permitted. For example, a request using HTTP is sent to a server that can only handle FTP requests. In order for the request to be processed, it must pass through a server that can handle both protocols. This can be accomplished by making use of proxy chaining which allows the request to be passed from a server that is not able to fulfill such a request (perhaps due to security or networking issues, or its own limited capabilities) to a server that can fulfill such a request.
The first proxy server in a chain will check to make sure a request is valid. If a request is not valid, or not allowed (blocked by the proxy), it will reject the request resulting in the client receiving an error or a redirect. If a request is valid, the proxy may check if the requested information is cached and simply serve it from there. If the requested information is not in cache, the proxy will pass the request on to the next proxy server in the chain. This server also has the ability to fulfill, forward, redirect, or reject the request. If it acts to forward the request then it too passes the request on to yet another proxy server. This process is repeated until the request reaches the last proxy server in the chain. The last server in the chain is required to handle the request by contacting the content server, using whatever protocol is required, to obtain the information. The information is then relayed back through the chain until it reaches the requesting client.

Proxy chaining

The above image shows a proxy chaining configuration. The intranet client makes a request to Server C (Content Server FTP). Server A (Proxy Server HTTP) does not contain the requested information in cache, so the request is passed through the firewall to Server B (proxy server HTTP/FTP). Server B has both HTTP and FTP protocols and is able to change the HTTP request to an FTP request. Server C receives the FTP request and passes back the requested information to Server B. Server B, in turn, passes the fulfilled request back to the intranet client using the HTTP protocol. The request is sent through the firewall and Server A where the request is cached and given to the intranet client.

Apache as Forward Proxy:
An ordinary forward proxy is an intermediate server that sits between the client and the origin server. In order to get content from the origin server, the client sends a request to the proxy naming the origin server as the target and the proxy then requests the content from the origin server and returns it to the client. The client must be specially configured to use the forward proxy to access other sites.

A typical usage of a forward proxy is to provide Internet access to internal clients that are otherwise restricted by a firewall. The forward proxy can also use caching (mod_cache) to reduce network usage.

The forward proxy is activated using the ProxyRequests directive. Because forward proxies allow clients to access arbitrary sites through your server and to hide their true origin, it is essential that you secure your server so that only authorized clients can access the proxy before activating a forward proxy.

ProxyRequests On
ProxyVia On


Order deny,allow
Deny from all
Allow from 192.168.1

Apache as Reverse Proxy:
A reverse proxy (or gateway), by contrast, appears to the client just like an ordinary web server. No special configuration on the client is necessary. The client makes ordinary requests for content the reverse proxy then decides where to send those requests, and returns the content as if it was itself the origin.

A typical usage of a reverse proxy is to provide Internet users access to a server that is behind a firewall. Reverse proxies can also be used to balance load among several back-end servers, or to provide caching for a slower back-end server. In addition, reverse proxies can be used simply to bring several servers into the same URL space.

A reverse proxy is activated using the ProxyPass directive or the flag to the RewriteRule directive. It is not necessary to turn ProxyRequests on in order to configure a reverse proxy.

ProxyRequests Off


Order deny,allow
Allow from all

ProxyPass /foo http://foo.example.com/bar
ProxyPassReverse /foo http://foo.example.com/bar

Configuring Apache to be a forward proxy

This configuration makes Apache act as an HTTP proxy:


ProxyRequests On
ProxyVia On
#ProxyRemote * http://…:8080 Uncomment to route requests through another proxy

Order deny,allow
Deny from all
Allow from all # Not a good idea, set to allowed IP ranges

CacheRoot “/tmp”
CacheMaxExpire 24
CacheLastModifiedFactor 0.1
CacheDefaultExpire 1

ServerName my-proxy

ErrorLog “/var/log/apache2/proxy-error.log”
CustomLog “/var/log/apache2/proxy-access.log” common

Also read this.

Tips

You can use mod_rewrite to rewrite requests. To rewrite root (/) to /temporary_outage you could use the following rewrite:

RewriteCond %{HTTP_HOST} ^(www\.)?xxx\.com
RewriteRule /$ http://%{HTTP_HOST}/temporary_outage/ [P,L]

Forward Proxy works

# webproxy server1

NameVirtualHost *:80

ServerName server1
ProxyPass / http://realserver1/
ProxyHTMLURLMap http://realserver1 /

ProxyPassReverse /
ProxyHTMLInterp On
ProxyHTMLURLMap / /
RequestHeader unset Accept-Encoding

# webproxy server2
NameVirtualHost *:80

ServerName server2
ProxyPass / http://realserver2/
ProxyHTMLURLMap http://realserver2 /

ProxyPassReverse /
ProxyHTMLInterp On
ProxyHTMLURLMap / /
RequestHeader unset Accept-Encoding

# realserver2 reverse proxy
NameVirtualHost *:80

ServerName realserver2

Order deny,allow
Allow from all

ProxyPreserveHost On
ProxyPass / http://localhost:32101/
ProxyPassReverse / http://localhost:32101/

Apache Performance Tuning

Apache Performance Tuning

Forewarning:

“Premature optimization is the root of all evil.” — Donald Knuth.

Select MPM
Chose the right MPM for the right job:
prefork [default MPM for Apache 2.0 and 1.3]:
• Apache 1.3-based.
• Multiple processes, 1 thread per process, processes handle requests.
• Used for security and stability.
• Has higher memory consumption and lower performance over the newer Apache 2.0-based threaded MPMs.
worker:
• Apache 2.0-based.
• Multiple processes, many threads per process, threads handle requests.
• Used for lower memory consumption and higher performance.
• Does not provide the same level of isolation request-to-request, as a process-based MPM does.
winnt:
• The only MPM choice under Windows.
• 1 parent process, exactly 1 child process with many threads, threads handle requests.
• Best solution under Windows, as on this platform, threads are always “cheaper” to use over processes.
Configure MPM
Core Features and Multi-Processing Modules
Default Configuration

StartServers 8
MinSpareServers 5
MaxSpareServers 20
MaxClients 150
MaxRequestsPerChild 1000


StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0


ThreadsPerChild 250
MaxRequestsPerChild 0

Directives

MaxClients, for prefork MPM

MaxClients sets a limit on the number of simultaneous connections/requests that will be served.

I consider this directive to be the critical factor to a well functioning server. Set this number too low and resources will go to waste. Set this number too high and an influx of connections will bring the server to a stand still. Set this number just right and your server will fully utilize the available resources.

An approximation of this number should be derived by dividing the amount of system memory (physical RAM) available by the maximum size of an apache/httpd process; with a generous amount spared for all other processes.

MaxClients ? (RAM – size_all_other_processes)/(size_apache_process)

Use ‘ps -ylC httpd –sort:rss’ to find process size. Divide number by 1024 to get megabytes. Also try ‘top’.
Use ‘free -m’ for a general overview. The key figure to look at is the buffers/cache used value.

Use ‘vmstat 2 5’ to display the number of runnable, blocked, and waiting processes; and swap in and swap out.
Example:
• System: VPS (Virtual Private Server), CentOS 4.4, with 128MB RAM
• Apache: v2.0, mpm_prefork, mod_php, mod_rewrite, mod_ssl, and other modules
• Other Services: MySQL, Bind, SendMail
• Reported System Memory: 120MB
• Reported httpd process size: 7-13MB
• Assumed memory available to Apache: 90MB
Optimal settings:
• StartServers 5
• MinSpareServers 5
• MaxSpareServers 10
• ServerLimit 15
• MaxClients 15
• MaxRequestsPerChild 2000

With the above configuration, we start with 5-10 processes and set a top limit of 15. Anything above this number will cause serious swapping and thrashing under a load; due to the low amount of RAM available to the [virtual] Server. With a dedicated Server, the default values [ServerLimit 256] will work with 1-2GB of RAM.

When calculating MaxClients, take into consideration that the reported size of a process and the effective size are two different values. In this setup, it might be safe to use 20 or more workers… Play with different values and check your system stats.

Note that when more connections are attempted than there are workers, the connections are placed into a queue. The default queue size value is 511 and can be adjusted with the ListenBackLog directive.
ThreadsPerChild, for winnt MPM
On the Windows side, the only useful directive is ThreadsPerChild, which is usually set to a value of 250 [defaults to 64 without a value]. If you expect more, or less, concurrent connections/requests, set this directive appropriately. Check process size with Task Manager, under different values and server load.
MaxRequestsPerChild
Directive MaxRequestsPerChild is used to recycle processes. When this directive is set to 0, an unlimited amount of requests are allowed per process.
While some might argue that this increases server performance by not burdening Apache with having to destroy and create new processes, there is the other side to the argument…
Setting this value to the amount of requests that a website generates per day, divided by the number of processes, will have the benefit of keeping memory leaks and process bloat to a minimum [both of which are a common problem]. The goal here is to recycle each process once per day, as apache threads gradually increase their memory allocation as they run.
Note that under the winnt MPM model, recycling the only request serving process that Apache contains, can present a problem for some sites with constant and heavy traffic.
Requests vs. Client Connections
On any given connection, to load a page, a client may request many URLs: page, site css files, javascript files, image files, etc.
Multiple requests from one client in rapid succession can have the same effect on a Server as “concurrent” connections [threaded MPMs and directive KeepAlive taken into consideration]. If a particular website requires 10 requests per page, 10 concurrent clients will require MPM settings that are geared more towards 20-70 clients. This issue manifests itself most under a process-based MPM [prefork].

Separate Static and Dynamic Content

Use separate servers for static and dynamic content. Apache processes serving dynamic content will carry overhead and swell to the size of the content being served, never decreasing in size. Each process will incur the size of any loaded PHP or Perl libraries. A 6MB-30MB process size [or 10% of server’s memory] is not unusual, and becomes a waist of resources for serving static content.
For a more efficient use of system memory, either use mod_proxy to pass specific requests onto another Apache Server, or use a lightweight server to handle static requests:
• lighttpd [has experimental win32 builds]
• tux [patched into RedHat, runs inside the Linux kernel and is at the top of the charts in performance]
The Server handling the static content goes up front.
Note that configuration settings will be quite different between a dynamic content Server and a static content Server

mod_deflate

Reduce bandwidth by 75% and improve response time by using mod_deflate.
LoadModule deflate_module modules/mod_deflate.so

AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml application/x-javascript

Loaded Modules
Reduce memory footprint by loading only the required modules.
Some also advise to statically compile in the needed modules, over building DSOs (Dynamic Shared Objects). Very bad advice. You will need to manually rebuild Apache every time a new version or security advisory for a module is put out, creating more work, more build related headaches, and more downtime.
mod_expires
Include mod_expires for the ability to set expiration dates for specific content; utilizing the ‘If-Modified-Since’ header cache control sent by the user’s browser/proxy. Will save bandwidth and drastically speed up your site for [repeat] visitors.
Note that this can also be implemented with mod_headers.
KeepAlive
Enable HTTP persistent connections to improve latency times and reduce server load significantly [25% of original load is not uncommon].
prefork MPM:
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 80
worker and winnt MPMs:
KeepAlive On
KeepAliveTimeout 15
MaxKeepAliveRequests 80

With the prefork MPM, it is recommended to set ‘KeepAlive’ to ‘Off’. Otherwise, a client will tie up an entire process for that span of time. Though in my experience, it is more useful to simply set the ‘KeepAliveTimeout’ value to something very low [2 seconds seems to be the ideal value]. This is not a problem with the worker MPM [thread-based], or under Windows [which only has the thread-based winnt MPM].
With the worker and winnt MPMs, the default 15 second timeout is setup to keep the connection open for the next page request; to better handle a client going from link to link. Check logs to see how long a client remains on each page before moving on to another link. Set value appropriately [do not set higher than 60 seconds].

SymLinks
Make sure ‘Options +FollowSymLinks -SymLinksIfOwnerMatch’ is set for all directories. Otherwise, Apache will issue an extra system call per filename component to substantiate that the filename is NOT a symlink; and more system calls to match an owner.

Options FollowSymLinks

AllowOverride
Set a default ‘AllowOverride None’ for your filesystem. Otherwise, for a given URL to path translation, Apache will attempt to detect an .htaccess file under every directory level of the given path.

AllowOverride None

ExtendedStatus
If mod_status is included, make sure that directive ‘ExtendedStatus’ is set to ‘Off’. Otherwise, Apache will issue several extra time-related system calls on every request made.
ExtendedStatus Off

ExtendedStatus
If mod_status is included, make sure that directive ‘ExtendedStatus’ is set to ‘Off’. Otherwise, Apache will issue several extra time-related system calls on every request made.
ExtendedStatus Off
Timeout
Lower the amount of time the server will wait before failing a request.
Timeout 45

Other/Specific
Cache all PHP pages, using Squid, and/or a PHP Accelerator and Encoder application, such as APC. Also take a look at mod_cache under Apache 2.2.
Convert/pre-render all PHP pages that do not change request-to-request, to static HTML pages. Use ‘wget’ or ‘HTTrack’ to crawl your site and perform this task automatically.
Pre-compress content and pre-generate headers for static pages; send-as-is using mod_asis. Can use ‘wget’ or ‘HTTrack’ for this task. Make sure to set zlib Compression Level to a high value (6-9). This will take a considerable amount of load off the server.
Use output buffering under PHP to generate output and serve requests without pauses.
Avoid content negotiation for faster response times.
Make sure log files are being rotated. Apache will not handle large (2gb+) files very well.
Gain a significant performance improvement by using SSL session cache.
Outsource your images to Amazon’s Simple Storage Service (S3).
Measuring Web Server Performance

Apache capacity planning -2

1. Apache server performance
Apache server performance can be improved by adding additional hardware resources such as RAM, faster CPU etc. But, most of the time, the same result can be achieved by custom configuration of the server. This article looks into getting maximum performance out of Apache with the existing hardware resources, specifically on the Linux systems. Of course, it is assumed that there is enough hardware resources, especially enough RAM that the server isn’t swapping frequently. First two sections look into various Compile-Time and Run-Time configuration options. Run-Time section assumes that Apache is compiled with prefork MPM. HTTP compression and caching is discussed next. Finally, using separate servers for serving static and dynamic contents are being discussed. Basic knowledge of compiling and configuring Apache, and Linux are assumed.



2 Compile-Time Configuration Options

2.1 Load only the required modules:

The Apache HTTP Server is a modular program where the administrator can choose the functionality to include in the server by selecting a set of modules [2]. The modules can be either statically compiled to the httpd binary or else can be compiled as Dynamic Shared Objects (DSOs). DSO modules can be either compiled when the server is built or else can use the apxs utility to compile and add at a later date. The module mod_so must be statically compiled into the Apache core to enable DSO support.

Run apache with only the required modules. This reduces the memory footprint and hence the server performance. Statically compiling modules will save RAM that’s used for supporting dynamically loaded modules, but one has to recompile Apache whenever a module is to be added or dropped. This is where the DSO mechanism comes handy. Once the mod_so module is statically compiled, any other module can be added or dropped using the LoadModule command in httpd.conf file – of course, you will have to compile the modules using apxs if it wasn’t compiled when the server was built.

2.2 Choose appropriate MPM:

Apache server ships with a selection of Multi-Processing Modules (MPMs) which are responsible for binding to network ports on the machine, accepting requests, and dispatching children to handle the requests [3]. Only one MPM can be loaded into the server at any time.

Choosing an MPM depends on various factors such as whether the OS supports threads, how much memory is available, scalability versus stability, whether non-thread-safe third-party modules are used, etc.. Linux systems can choose to use a threaded MPM like worker or a non-threaded MPM like prefork:

Worker MPM uses multiple child processes. It’s multi-threaded within each child and each thread handles a single connection. Worker is fast and highly scalable and the memory footprint is comparatively low. It’s well suited for multiple processors. On the other hand, worker is less tolerant to faulty modules and faulty threads can affect all the threads in a child process.

Prefork MPM uses multiple child processes, each child handles one connection at a time. Prefork is well suited for single or double CPU systems, speed is comparable to that of worker and it’s highly tolerant to faulty modules and crashing children. But the memory usage is high, more traffic leads to more memory usage.

3 Run-Time Configuration Options

3.1 DNS lookup:

The HostnameLookups directive enables DNS lookup so that hostnames can be logged instead of the IP address. This adds latency to every request since the DNS lookup has to be completed before the request is finished. HostnameLookups is Off by default in Apache 1.3 and above. Leave it Off and use post-processing program such as logresolve to resolve IP addresses in Apache’s access logfiles. Logresolve ships with Apache.
When using Allow from or Deny from directives, use IP address instead of a domain name or a hostname. Otherwise a double DNS lookup is performed to make sure that the domain name or the hostname is not being spoofed.

3.2 AllowOverride:

If AllowOverride is not set to ‘None’, then Apache will attempt to open .htaccess file (as specified by AccessFileName directive) in each directory that it visits. For example:
DocumentRoot /var/www/html

AllowOverride all

If a request is made for URI /index.html, then Apache will attempt to open /.htaccess, /var/.htaccess, /var/www/.htaccess, and /var/www/html/.htaccess. These additional file system lookups add to the latency. If .htaccess is required for a particular directory, then enable it for that directory alone.


3.3 FollowSymLinks and SymLinksIfOwnerMatch:

If FollowSymLinks option is set, then the server will follow symbolic links in this directory. If SymLinksIfOwnerMatch is set, then the server will follow symbolic links only if the target file or directory is owned by the same user as the link.
If SymLinksIfOwnerMatch is set, then Apache will have to issue additional system calls to verify whether the ownership of the link and the target file match. Additional system calls are also needed when FollowSymLinks is NOT set. For example:
DocumentRoot /vaw/www/html

Options SymLinksIfOwnerMatch

For a request made for URI /index.html, Apache will perform lstat() on /var, /var/www, /var/www/html, and /var/www/html/index.html. These additional system calls will add to the latency. The lstat results are not cached, so they will occur on every request.
For maximum performance, set FollowSymLinks everywhere and never set SymLinksIfOwnerMatch. Or else, if SymLinksIfOwnerMatch is required for a directory, then set it for that directory alone.

3.4 Content Negotiation:

Avoid content negotiation for fast response. If content negotiation is required for the site, use type-map files rather than Options MultiViews directive. With MultiViews, Apache has to scan the directory for files, which add to the latency.
3.5 MaxClients:
The MaxClients sets the limit on maximum simultaneous requests that can be supported by the server. No more than this much number of child processes are spawned. It shouldn’t be set too low such that new connections are put in queue, which eventually time-out and the server resources are left unused. Setting this too high will cause the server to start swapping and the response time will degrade drastically. Appropriate value for MaxClients can be calculated as: MaxClients = Total RAM dedicated to the web server / Max child process size —- [4] Child process size for serving static file is about 2-3M. For dynamic content such as PHP, it may be around 15M. The RSS column in
“ps -ylC httpd –sort:rss”
shows non-swapped physical memory usage by Apache processes in kilo Bytes.
If there are more concurrent users than MaxClients, the requests will be queued up to a number based on ListenBacklog directive. Increase ServerLimit to set MaxClients above 256.

3.6 MinSpareServers, MaxSpareServers, and StartServers:

MaxSpareServers and MinSpareServers determine how many child processes to keep while waiting for requests. If the MinSpareServers is too low and a bunch of requests come in, then Apache will have to spawn additional child processes to serve the requests. Creating child processes is relatively expensive. If the server is busy creating child processes, it won’t be able to serve the client requests immediately. MaxSpareServers shouldn’t be set too high, it can cause resource problems since the child processes consume resources.
Tune MinSpareServers and MaxSpareServers such that Apache need not frequently spwan more than 4 child processes per second (Apache can spwan a maximum of 32 child processes per second). When more than 4 children are spawned per second, a message will be logged in the ErrorLog.
The StartServers directive sets the number of child server processes created on startup. Apache will continue creating child process until the MinSpareServers setting is reached. Doesn’t have much effect on performance if the server isn’t restarted frequently. If there are lot of requests and Apache is restarted frequently, set this to a relatively high value.

3.7 MaxRequestsPerChild:

The MaxRequestsPerChild directive sets the limit on the number of requests that an individual child server process will handle. After MaxRequestsPerChild requests, the child process will die. It’s set to 0 by default, that means the child process will never expire. It is appropriate to set this to a value of few thousands. This can help prevent memory leakage since the process dies after serving a certain number of requests. Do not set this too low, since creating new processes does have overhead.

3.8 KeepAlive and KeepAliveTimeout:
The KeepAlive directive allows multiple requests to be sent over the same TCP connection. This is particularly useful while serving HTML pages with lot of images. If KeepAlive is set to Off, then for each images, a separate TCP connection has to be made. Overhead due to establishing TCP connection can be eliminated by turning On KeepAlive.
KeepAliveTimeout determines how long to wait for the next request. Set this to a low value, perhaps between two to five seconds. If it is set too high, child processed are tied up waiting for the client when they could be used for serving new clients.

4 HTTP Compression & Caching

HTTP compression is completely specified in HTTP/1.1. The server uses gzip or deflate encoding method to the response payload before it is sent to the client. Client then decompresses the payload. There is no need to install any additional software at the client side since all major browsers support this. Using compression will save bandwidth and improve response time, studies have found a mean compression gain of 75.2 % [5]. HTTP Compression can be enabled in Apache using mod_deflate module. Payload is compressed only if the browser requests compression, otherwise uncompressed content is served. A compression aware browser inform the server that it prefers compressed content through the HTTP request header – “Accept-Encoding: gzip,deflate”. Then the server responds with compressed payload and the response header set to ”
Content-Encoding:
gzip
Following example uses telnet to view request and response headers:
bash-3.00$ telnet www.webperformance.org 80
Trying 24.60.234.27…
Connected to www.webperformance.org (24.60.234.27).
Escape character is ‘^]’.
HEAD / HTTP/1.1
Host: www.webperformance.org
Accept-Encoding: gzip,deflate

HTTP/1.1 200 OK
Date: Sat, 31 Dec 2005 02:29:22 GMT
Server: Apache/2.0
X-Powered-By: PHP/5.1.1
Cache-Control: max-age=0
Expires: Sat, 31 Dec 2005 02:29:22 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 20
Content-Type: text/html; charset=ISO-8859-1

In caching, a copy of the data is stored at the client or in a proxy server so that it need not be retrieved frequently from the server. This will save bandwidth, decrease load on the server and reduce latency. Cache control is done through HTTP headers. In Apache, this can be accomplished through mod_expires and mod_headers modules. Also there is server side caching, in which the frequently accessed contents are stored in memory so that it can be served fast. The module mod_cache can be used for server side caching, it is production stable in Apache version 2.2.

5 Separate server for static and dynamic content

Apache processes serving dynamic content takes about 3M to 20M of RAM. It grows to accommodate the content it’s serving and never decreases until the process dies. Say an Apache process grows to 20M to serve a dynamic content. After completing the request, it is free to serve any other request. If a request for an image comes in, then this 20M process is serving a static content which could as well be served by a 1M process. Memory is used inefficiently.
Use a tiny Apache (with minimum modules statically compiled) as the front-end server to serve static contents. Request for dynamic contents are forwarded to the heavy Apache (compiled with all required modules). Using a light front-end server has the advantage that the static contents are served fast without much memory usage and only the dynamic contents are passed over to the heavy server.
Request forwarding can be achieved by using mod_proxy and rewrite_module modules. Suppose there is a lightweight Apache server listening to port 80 and the heavyweight Apache listening on port 8088. Then the following configuration in the lightweight Apache can be used to forward all request except request for images to the heavyweight server.
ProxyPassReverse / http://%{HTTP_HOST}:8088/
RewriteEngine on —- [9]
RewriteCond %{REQUEST_URI} !.*\.(gif|png|jpg)$
RewriteRule ^/(.*) http://%{HTTP_HOST}:8088/$1 [P]
All requests, except for images, are proxied to the backend server. Response is received by the frontend server and then supplied to the client. As far as client is concerned, all the response seem to come from a single server.

6 Conclusion

Configuring Apache for maximum performance is tricky, there are no hard and fast rules. Understand the web server requirements and experiment with various available options. Use tools like ab and httperf to measure the web server performance. Light weight servers such as tux , thttpd can also be used as the front-end server. If a database server is used, make sure it is optimized so that it won’t create any bottleneck. In case of MySQL, mtop can be used to monitor slow queries. Performance of PHP scripts can be improved by using a PHP caching product such as Turck MMCache. It eliminates overhead due to compiling by caching the PHP scripts

Hardening guide for Apache

Hardening guide for Apache

From:
ServerSignature OnTo:
ServerSignature Off
HostnameLookups Off

From:
# ServerTokensTo:
ServerTokens Prod

From:
ServerAdmin you@yourhost.comTo:
ServerAdmin webmaster@yourcompany.com

From:
LogLevel warnTo:
LogLevel notice

From:
IndexOptions FancyIndexing VersionSortTo:
# IndexOptions FancyIndexing VersionSort
#

To:
# AddIcon

From:
DefaultIcon /icons/unknown.gifTo:
# DefaultIcon /icons/unknown.gif

From:
Alias /icons/ “/var/apache2/icons/”To:
# Alias /icons/ “/var/apache2/icons/”

From:
AliasMatchTo:
# AliasMatch

From:
ScriptAliasTo:
# ScriptAlias

From:
LoadModule proxy_ftp_module libexec/mod_proxy_ftp.soTo:
# LoadModule proxy_ftp_module libexec/mod_proxy_ftp.so

From
LoadModule imap_module libexec/mod_imap.soTo:
# LoadModule imap_module libexec/mod_imap.so

From:
LoadModule cgi_module libexec/mod_cgi.soTo:
# LoadModule cgi_module libexec/mod_cgi.so

From:
LoadModule suexec_module libexec/mod_suexec.soTo:
# LoadModule suexec_module libexec/mod_suexec.so

From:
LoadModule autoindex_module libexec/mod_autoindex.soTo:
# LoadModule autoindex_module libexec/mod_autoindex.so

From:
LoadModule info_module libexec/mod_info.soTo:
# LoadModule info_module libexec/mod_info.so

From:
LoadModule status_module libexec/mod_status.soTo:
# LoadModule status_module libexec/mod_status.so

From:
LoadModule status_module libexec/mod_status.soTo:
# LoadModule status_module libexec/mod_status.so

From:
LoadModule userdir_module libexec/mod_userdir.soTo:
# LoadModule userdir_module libexec/mod_userdir.so

From:
LoadModule cern_meta_module modules/mod_cern_meta.soTo:
# LoadModule cern_meta_module modules/mod_cern_meta.so

From:
LoadModule dav_module modules/mod_dav.soTo:
# LoadModule dav_module modules/mod_dav.so

From:

Options FollowSymLinks
AllowOverride None

To:

Options None
AllowOverride None
Order deny,allow
deny from all

From:
<Directory “/var/apache2/htdocs”>To:
<Directory “/www”>

deny from all

From:
Options Indexes FollowSymLinksTo:
Options -FollowSymLinks -Includes -Indexes -MultiViews
# Add the following sections to the end of the httpd.conf file:
LimitRequestBody 10000
LimitRequestFields 40
LimitRequestFieldSize 100
LimitRequestLine 500
# Remove the sections bellow from the file httpd.conf
<Directory “/usr/apache2/manual”>
<Directory “/var/apache2/cgi-bin”>
# Edit using VI the file /usr/apache2/include/ap_release.h and change the following strings:
From:
#define AP_SERVER_BASEVENDOR “Apache Software Foundation”To:
#define AP_SERVER_BASEVENDOR “Restricted server”
From:
#define AP_SERVER_BASEPRODUCT “Apache”To:
#define AP_SERVER_BASEPRODUCT “Secure Web Server”
# Starting Apache from command line:
/usr/apache2/bin/apachectl start
# Run the command bellow to start the Apache service at server start-up:
svcadm enable apache2

Security Testing your Apache Configuration with Nikto

Nikto: Scan Apache for Security Holes

Nikto is an Open Source (GPL) web server scanner which performs comprehensive tests against web servers for multiple items,
including over 3500 potentially dangerous files/CGIs,
versions on over 900 servers, and version specific problems on over 250 servers. Scan items and plugins are frequently updated and
can be automatically updated (if desired).”

Nikto does require the LibWhisker Perl module, but this is built into the program so it does not need to be installed.
You will want to install the Net::SSLeay Perl module if you want to test SSL.


Install mod_security Apache Intrusion Detection And Prevention Engine

ModSecurity operates embedded into the web server (httpd), acting as a powerful umbrella – shielding web applications from attacks

mod_security configuration files

1. /etc/httpd/conf.d/mod_security.conf – main configuration file for the mod_security Apache module.
2. /etc/httpd/modsecurity.d/ – all other configuration files for the mod_security Apache.
3. /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf – Configuration contained in this file should be customized for your specific requirements before deployment.
4. /var/log/httpd/modsec_debug.log – Use debug messages for debugging mod_security rules and other problems.
5. /var/log/httpd/modsec_audit.log – All requests that trigger a ModSecurity events (as detected) or a serer error are logged (“RelevantOnly”) are logged into this file.

cp modsecurity_crs_10_config.conf.example modsecurity_crs_10_config.conf

vi /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf

There are five rules directories:

* activated_rules
* base_rules
* experimental_rules
* optional_rules
* slr_rules

Make sure SecRuleEngine set to “On” to protect webserver for the attacks:

SecRuleEngine On

Turn on other required options and policies as per your requirements. Finally, restart httpd:
# service httpd restart
Make sure everything is working:
# tail -f /var/log/httpd/error_log

mod_evasive is an evasive maneuvers module for Apache that provides evasive action in the event of an HTTP DoS attack or brute force attack. It is also designed to be a detection and network management tool, and can be easily configured to talk to ipchains, firewalls, routers, and more. mod_evasive presently reports abuse via email and syslog facilities. This guide assumes you already have your LAMP server configured.
Guides for setting up a LMAP stack can be found under our LAMP guides section.

disable TRACE and TRACK in the main scope of httpd.conf

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* – [F]
RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* – [F]

ServerTokens Prod
ServerSignature Off
TraceEnable Off



1.2 ModSecurity

1.3 ModSecurity Core Rules Overview

Performance
Quality
Regression tests
Real traffic testing
Generic Detection
Event Information
Plug and Play
Protocol compliance:
Attack Detection:

## For RHEL/CentOS 6.2/6.1/6/5.8 ##
# cd /usr/src
# wget http://www.modsecurity.org/download/modsecurity-apache_2.6.6.tar.gz
# tar xzf modsecurity-apache_2.6.6.tar.gz
# cd modsecurity-apache_2.6.6
# ./configure
# make install
# cp modsecurity.conf-recommended /etc/httpd/conf.d/modsecurity.conf

CentOS 6.x 32-bit (x86/i386):

rpm -Uvh http://mirror.overthewire.com.au/pub/epel/6/i386/epel-release-6-7.noarch.rpm

CentOS 6.x 64-bit (x64):

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm

CentOS 5.x 32-bit (x86/i386):

rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

CentOS 5.x 64-bit (x64):

rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm


yum install mod_security





Downloading OWASP Mod_Security Core Rule Set


## For RHEL/CentOS 6.2/6.1/6/5.8 ##
# cd /etc/httpd/
# wget http://downloads.sourceforge.net/project/mod-security/modsecurity-crs/0-CURRENT/modsecurity-crs_2.2.5.tar.gz
# tar xzf modsecurity-crs_2.2.5.tar.gz
# mv modsecurity-crs_2.2.5 modsecurity-crs
# cd modsecurity-crs
# cp modsecurity_crs_10_setup.conf.example modsecurity_crs_10_config.conf








CentOS / Redhat (RHEL) / Fedora Linux disable a module

Under Redhat based Linux distributions you need to modify *.conf file stored in /etc/httpd/conf.d/ directory. Apache scans for files with the .conf suffix at start up.

So if the system does not need to use mod_python, rename 'python.conf' to 'python.bak' and restart Apache with the command 'service httpd restart in order to disable that particular module and save memory.
# cd /etc/httpd/conf.d/
# mv perl.conf no.perl.bak
# /etc/init.d/httpd restart
Enable a module

To re-enable modules, simply rename them to their original names and restart Apache to get back module functionality:
# cd /etc/httpd/conf.d/
# mv no.perl.bak perl.conf
# /etc/init.d/httpd restart



More about /etc/httpd/conf.d/ directory

This directory holds Apache 2.0 module-specific configuration files; any files in this directory which have the ".conf" extension will be processed as Apache configuration files. Files are processed in alphabetical order, so if using configuration directives which depend on, say, mod_perl being loaded, ensure that
these are placed in a filename later in the sort order than "perl.conf".

    manual.conf : This configuration file allows the manual to be accessed at http://localhost/manual/
    perl.conf : mod_perl incorporates a Perl interpreter into the Apache web server, so that the Apache web server can directly execute Perl code.
    php.conf : php5 module for php
    proxy_ajp.conf : When loaded, the mod_proxy_ajp module adds support for proxying to an AJP/1.3 backend server such as Tomcat.
    python.conf : mod_python is a module that embeds the Python language interpreter within the server, allowing Apache handlers to be written in Python.
    squid.conf : Access to squid cache manager
    ssl.conf : Apache SSL server configuration
    webalizer.conf : Webalizer stats configuration
    welcome.conf : This configuration file enables the default "Welcome" page if there is no default index page present for
    the root URL.


mod_dav_svn
mod_perl-devel
mod_auth_kerb
mod_nss
mod_auth_kerb
mod_auth_mysql
mod_auth_pgsql
mod_authz_ldap
mod_dnssd
mod_revocator
mod_wsgi


grep -v '\#' /etc/httpd/conf/httpd.conf


yum install  httpd-devel mod_security mod_ssl php

#LoadModule ldap_module modules/mod_ldap.so
#LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
#LoadModule logio_module modules/mod_logio.so
#LoadModule logio_module modules/mod_logio.so
#LoadModule env_module modules/mod_env.so
#LoadModule ext_filter_module modules/mod_ext_filter.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
#LoadModule dav_module modules/mod_dav.so
#LoadModule info_module modules/mod_info.so
#LoadModule dav_fs_module modules/mod_dav_fs.so
#LoadModule speling_module modules/mod_speling.so
#LoadModule userdir_module modules/mod_userdir.so
#LoadModule substitute_module modules/mod_substitute.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
#LoadModule cache_module modules/mod_cache.so
#LoadModule disk_cache_module modules/mod_disk_cache.so
#LoadModule cgi_module modules/mod_cgi.so
#LoadModule version_module modules/mod_version.so



LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so



http://www.thegeekstuff.com/2011/03/apache-hardening/
http://hackathology.blogspot.sg/2007/11/basics-of-modsecurity.html
http://www.tecmint.com/protect-apache-using-mod_security-and-mod_evasive-on-rhel-centos-fedora/

Here’s what I’ve added to tune the Linux TCP stack in /etc/sysctl.conf:


    net.ipv4.tcp_abort_on_overflow = 1
    net.ipv4.tcp_fin_timeout = 15
    net.ipv4.tcp_low_latency = 1
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_max_syn_backlog = 2048
    net.ipv4.tcp_synack_retries = 3
    net.ipv4.tcp_sack = 0
    net.ipv4.ip_conntrack_max = 65535
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.ip_local_port_range = 1024 65000
    net.ipv4.tcp_keepalive_intvl = 15
    net.ipv4.tcp_keepalive_probes = 4
    net.ipv4.tcp_keepalive_time = 1800

Apache capacity planning

Apache capacity planning

    LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” proxy
    SetEnvIf X-Forwarded-For “^.*\..*\..*\..*” forwarded
    CustomLog “logs/access_log” combined env=!forwarded
    CustomLog “logs/access_log” proxy env=forwarded

    The apache MPM

    http://articles.slicehost.com/2010/5/19/configuring-the-apache-mpm-on-centos

    http://www.howtoforge.com/configuring_apache_for_maximum_performance

    The apache MPM

    Part of the apache web server installation is the “MPM”, which stands for “Multi-Processing Method”.
    The MPM determines the mechanism apache uses to handle multiple connections. Now that we have an idea of where apache keeps its configs we’ll cover in detail
    how the main MPMs are configured and how you might optimize their settings for your environment.
    The difference

    The first thing to know is that there are several MPMs that apache can use, but the main MPMs are “worker” and “prefork”.

    The worker MPM primarily handles connections by creating new threads within a child process, while the prefork MPM spawns a new process to handle each connection.
    The worker MPM is considered more efficient, but some modules aren’t stable when running under the worker MPM.
    The yum apache package defaults to the prefork MPM for the best compatibility with modules.
    Most users won’t notice a difference in performance between the MPMs, but it’s good to know they’re there.
    If you find your site is having trouble scaling, for example, you might want to switch to the worker MPM even though it isn’t recommended by a module you’re using. PHP,
    for instance, will switch apache to the prefork MPM when aptitude installs it, but newer versions of PHP can be compiled with worker MPM support.
    For a change like that, you’ll want to consult your module’s documentation to see what it may have to say about apache MPMs.

    The prefork MPM

    Default:


    StartServers 8
    MinSpareServers 5
    MaxSpareServers 20
    ServerLimit 256
    MaxClients 256
    MaxRequestsPerChild 4000

    StartServers

    This is the number of child server processes created at startup, ready to handle incoming connections.
    If you’re expecting heavy traffic you might want to increase this number so the server is ready to handle a lot of connections right when it’s started.

    MinSpareServers

    The minimum number of child server processes to keep in reserve.
    MaxSpareServers

    Maximum number of child server processes that will be held in reserve. Any more than the maximum will be killed.

    ServerLimit

    The ServerLimit directive sets an absolute limit on the MaxClients directive. The reasons for this aren’t interesting enough to go into here, so the main thing to know about this directive is that
    it should usually be set to the same value as MaxClients, and probably shouldn’t be set at all if you set MaxClients lower than 256.

    MaxClients

    Sets the maximum simultaneous requests that Apache will handle. Anything over this number will be queued until a process is free to action the request.

    MaxClients is not the same as the maximum number of visitors you can have. It is the maximum number of requests that can be fielded at the same time.

    Remember the KeepAliveTimeout? This was set low so the connections used by idle web clients can be recycled more quickly to handle new web clients.
    Each active connection uses memory and counts toward the MaxClients total. If you hit the number of connections in the MaxClients setting,
    web clients will be stuck waiting for a connection slot to free up.

    The trick with MaxClients is that you want the number to be high enough that visitors don’t have to wait before connecting to your site,
    but not so high that apache needs to grab more memory than is available on your server. If you go over the available memory for your server
    it will start dipping into swap memory, which is slow and ugly and trust me you don’t want to do that.
    For the prefork MPM, a new process is started when apache handles a new connection. That means MaxClients sets the maximum number of processes apache
    will create to handle incoming clients. Memory can definitely be a limiting factor here.

    MaxRequestsPerChild

    Sets how many requests a child process will handle before terminating. The default is zero, which means it will never die.

    Why change this if the Max numbers are set as shown above? Well, it can help in managing your Slice memory usage.

    If you change the default you give a child a finite number of actions before it will die.

    This will, in effect, reduce the number of processes in use when the server is not busy, thus freeing memory.

    Freeing it for what though? If other software needed memory then it would also need it when the server is under load. It is unlikely you will have anything that requires memory only when the server is quiet.

    The worker MPM

    Defaults:


    StartServers 2
    MaxClients 150
    MinSpareThreads 25
    MaxSpareThreads 75
    ThreadsPerChild 25
    MaxRequestsPerChild 0

    Configuring Apache for Maximum Performance

    2.1 Load only the required modules:

    Run apache with only the required modules. This reduces the memory footprint and hence the server performance. Statically compiling modules will save RAM that’s used for supporting dynamically loaded modules,
    but one has to recompile Apache whenever a module is to be added or dropped

    2.2 Choose appropriate MPM:

    orker MPM uses multiple child processes. It’s multi-threaded within each child and each thread handles a single connection.
    Worker is fast and highly scalable and the memory footprint is comparatively low. It’s well suited for multiple processors. On the other hand,
    worker is less tolerant to faulty modules and faulty threads can affect all the threads in a child process.

    Prefork MPM uses multiple child processes, each child handles one connection at a time. Prefork is well suited for single or double CPU systems,
    speed is comparable to that of worker and it’s highly tolerant to faulty modules and crashing children. But the memory usage is high,
    more traffic leads to more memory usage.

    3.1 DNS lookup:

    The HostnameLookups directive enables DNS lookup so that hostnames can be logged instead of the IP address. This adds latency to every request since the DNS lookup has to be completed before the request is finished.
    HostnameLookups is Off by default in Apache 1.3 and above. Leave it Off and use post-processing program such as logresolve to resolve IP addresses in Apache’s access logfiles.
    .Logresolve ships with Apache.

    When using Allow from or Deny from directives, use IP address instead of a domain name or a hostname.
    Otherwise a double DNS lookup is performed to make sure that the domain name or the hostname is not being spoofed.

    3.2 AllowOverride:
    If AllowOverride is not set to ‘None’, then Apache will attempt to open .htaccess file (as specified by AccessFileName directive) in each directory that it visits. For example:

    DocumentRoot /var/www/html

    AllowOverride all

    3.3 FollowSymLinks and SymLinksIfOwnerMatch:
    If FollowSymLinks option is set, then the server will follow symbolic links in this directory. If SymLinksIfOwnerMatch is set, then the server will follow symbolic links only if the target file or directory is owned by the same user as the link.

    If SymLinksIfOwnerMatch is set, then Apache will have to issue additional system calls to verify whether the ownership of the link and the target file match.
    Additional system calls are also needed when FollowSymLinks is NOT set. For example:

    DocumentRoot /vaw/www/html

    Options SymLinksIfOwnerMatch

    3.4 Content Negotiation:

    Avoid content negotiation for fast response. If content negotiation is required for the site, use type-map files rather than Options MultiViews directive. With MultiViews,
    Apache has to scan the directory for files, which add to the latency.

    3.5 MaxClients:

    The MaxClients sets the limit on maximum simultaneous requests that can be supported by the server. No more than this much number of child processes are spawned.
    It shouldn’t be set too low such that new connections are put in queue, which eventually time-out and the server resources are left unused.
    Setting this too high will cause the server to start swapping and the response time will degrade drastically.
    Appropriate value for MaxClients can be calculated as: MaxClients = Total RAM dedicated to the web server / Max child process size —-
    [4] Child process size for serving static file is about 2-3M. For dynamic content such as PHP, it may be around 15M. The RSS column in

    3.6 MinSpareServers, MaxSpareServers, and StartServers:

    3.7 MaxRequestsPerChild:

    3.8 KeepAlive and KeepAliveTimeout:

    4 HTTP Compression & Caching

    5 Separate server for static and dynamic content

Extending Swap on an LVM2 Logical Volume

Extending Swap on an LVM2 Logical Volume

1)Check swap space and its utilization
cat /proc/swaps # free

2) Scan (all disks) for Logical Volumes
lvscan
[root@localhost ~]# lvscan
ACTIVE ‘/dev/VolGroup/lv_root’ [47.44 GiB] inherit
ACTIVE ‘/dev/VolGroup/lv_home’ [46.19 GiB] inherit
ACTIVE ‘/dev/VolGroup/lv_swap’ [5.88 GiB] inherit

3) Disable devices and files for paging and swapping
swapoff -v /dev/VolGroup/lv_swap

4) Resize a logical volume Adding 1 GB
lvm lvresize /dev/VolGroup/lv_swap -L +1G

5) Set up a Linux swap area
mkswap /dev/VolGroup/lv_swap

6) Enable devices and files for paging and swapping
swapon -va

7) Check swap space and its utilization
cat /proc/swaps # free

8) Scan (all disks) for Logical Volumes
lvscan

Reducing Swap on an LVM2 Logical Volume

Reducing Swap on an LVM2 Logical Volume
1) Check swap space and its utilization
cat /proc/swaps # free

2) Scan (all disks) for Logical Volumes
lvscan

[root@localhost ~]# lvscan
ACTIVE ‘/dev/VolGroup/lv_root’ [47.44 GiB] inherit
ACTIVE ‘/dev/VolGroup/lv_home’ [46.19 GiB] inherit
ACTIVE ‘/dev/VolGroup/lv_swap’ [5.88 GiB] inherit

3) Disable devices and files for paging and swapping
swapoff -v /dev/vg0/lv_swap

4)Reduce the size of a logical volume
lvm lvreduce /dev/VolGroup/lv_swap -L -1G

5) Set up a Linux swap area
mkswap /dev/VolGroup/lv_swap

6) Enable devices and files for paging and swapping

swapon -va

7) Check swap space and its utilization
cat /proc/swaps # free
8) Scan (all disks) for Logical Volumes
lvscan

Centos 6 SFTP chroot Jail

Centos 6 SFTP chroot Jail

User and Group setup

First you will want to establish the sftponly group

groupadd sftponly

Then create the user with the correct home directories and group

useradd -d /var/www/vhosts/bob -s /bin/false -G sftponly bob

Don’t forget at this point to also add password to these new accounts.

SSHd configuration changes

Now we need to make changes in /etc/ssh/sshd_config to enable SFTP chroot jails in SSH.

Comment out the following line in /etc/ssh/sshd_config:

Subsystem sftp /usr/lib/openssh/sftp-server

and replace it with this line:

Subsystem sftp internal-sftp

Then add the following set of lines to the very bottom of the file:

Match Group sftponly

ChrootDirectory /var/www/vhosts/%u

X11Forwarding no

AllowTCPForwarding no

ForceCommand internal-sftp

This creates a special login group that then chroot jailed all users in that group into their own home directory.

Once these file changes are saved you will need to restart SSHd for the changes to take effect, using the following command:

service sshd restart

Permissions cleanup and testing

Last issue to address is the permissions settings, for this example the directories /var/www/vhosts/bob and /var/www/vhosts/ted should both be owned by root. The directory /var/www/vhosts/ted/site1 should be owned by ted and the directory /var/www/vhosts/bob/site1 should be owned by bob.

chown root /var/www/vhosts/bob
ls -la
chmod go-w /var/www/vhosts/bob
chown bob:sftponly /var/www/vhosts/bob/fileupload/
chown bob:sftponly /opt/app/vhosts/rbc/writable/
chown bob:sftponly /opt/app/vhosts/rbc/codeupload/
chmod ug+rwx codeupload fileupload writable

tail -f /var/log/secure
tail -f /var/log/audit/audit.log