Categories

A sample text widget

Etiam pulvinar consectetur dolor sed malesuada. Ut convallis euismod dolor nec pretium. Nunc ut tristique massa.

Nam sodales mi vitae dolor ullamcorper et vulputate enim accumsan. Morbi orci magna, tincidunt vitae molestie nec, molestie at mi. Nulla nulla lorem, suscipit in posuere in, interdum non magna.

Installing node.js on amazon EC2 instance

Launch an EC2 instance of type Amazon Linux AMI from your aws console.

SSH into your ec2 instance
ssh -i ec2-user@my_ec2_ip_address

Update the instance
sudo yum update -y

#install developer tools
sudo yum groupinstall -y "Development Tools"

Install the node using nvm as it allows you to switch between any version of the node ????
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh | bash
. ~/.nvm/nvm.sh
nvm install 4.4.5
node -e "console.log('Running Node.js ' + process.version)"

Install pm2
npm install pm2 -g --save

pull the source code in ec2 instance. run your node app using pm2
pm2 start app.js --name="api"

Monitoring Memory Utilisation of aws EC2 instance in CloudWatch

Connect to your EC2 instance using ssh.
ssh -i ec2-user@my_ec2_ip_address

Install following perl packages
sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https

Download the monitoring scripts
curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O

Install the monitoring scripts
unzip CloudWatchMonitoringScripts-1.2.1.zip
rm CloudWatchMonitoringScripts-1.2.1.zip
cd aws-scripts-mon

run the following command to check the memory utilisation.
./mon-get-instance-stats.pl --recent-hours=12

In order to push this data periodically to cloud watch, we need to create an IAM user with relevant permissions, then schedule a cron job to periodically push the data to aws cloud watch.

Create a new IAM user and provide him access to the ec2 instance and cloud watch. Make a note of the aws credentials i.e. access key and secret key. You need this later. Make sure the IAM user has following access.

cloudwatch:PutMetricData
cloudwatch:GetMetricStatistics
cloudwatch:ListMetrics
ec2:DescribeTags

For the purpose of the demo, you can provide full access to EC2 and CloudWatch (This is not recommended for production.)

run the following command
cp awscreds.template awscreds.con

open the file  awscreds.conf and enter the key & secret.
The content of the file should look something like this

AWSAccessKeyId=AKAWSACESSKEYSA
AWSSecretKey=z/NOAWSSECRETkeyzt

Run the following command to push the data to cloud watch
./mon-put-instance-data.pl --mem-util --mem-used --mem-avail

Now, configure the cron tab to automate this process at eveny 5 minutes interval.
crontab -e
*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/ --from-cron

In your aws console, go to cloud watch section, and select Metrics => custom namespaces => Linux System. Select the required metrics.

 

AWS S3 permissions

A quick tip on aws S3 permissions to a specific folder in a bucket.

{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:Put*",
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::nameofmybucket/profile/*"
]
}]
}

Try to avoid bucket root level permission as much as possible. Also, try to use cognito if the client wants to directly access aws resources.

Setting up redis on aws EC2 instance

Setting up redis on aws EC2 instance

Launch an EC2 instance of type Amazon Linux AMI from your aws console.

SSH into your ec2 instance
ssh -i ec2-user@my_ec2_ip

Update the instance
sudo yum update -y

Install the developer tools to compile the redis source
sudo yum groupinstall -y "Development Tools"

Install tcl
sudo yum install -y tcl

Download the latest redis source
wget http://download.redis.io/redis-stable.tar.gz

unzip the source
tar xvzf redis-stable.tar.gz

go to the redis folder and run the make command to compile the source
cd redis-stable
make
make test
sudo make install

to run the server, go to the src folder and run the daemonize command.
cd src
./redis-server --daemonize yes

Run the redis-cli command to check if server resonds.
redis-cli
ping

If the response is PONG then you are good to go ????

554 5.7.1 : Relay access denied centos postfix

554 5.7.1 : Relay access denied centos postfix

 

Based on the Postfix documentation, setting mynetworks = all doesn’t appear to be valid, and if you specifically set mynetworks to something, then mynetworks_style is ignored.

Reset mynetworks back to its default value if you want to use mynetworks_style=host, or explicitly set mynetworks to reflect where mail is being sent from (usually localhost and your local subnet). For example:

mynetworks = 127.0.0.0/8 192.168.1.0/24
mynetworks_style = subnet

Based on the Postfix documentation, setting mynetworks = all doesn’t appear to be valid, and if you specifically set mynetworks to something, then mynetworks_style is ignored.

Reset mynetworks back to its default value if you want to use mynetworks_style=host, or explicitly set mynetworks to reflect where mail is being sent from (usually localhost and your local subnet). For example:

mynetworks = 127.0.0.0/8 192.168.1.0/24
mynetworks_style = subnet

How to resolve and prevent 554 5.7.1 : Relay access denied email errors in Linux and Windows mail servers

 

When a mail user tries to send a mail, and gets a bounce.
When mails from a remote domain is rejected by the server, and mail users report it to the server owner.
In either case, we’ve seen the error recorded in mail server logs. It looks something like this:

Jan 23 03:10:57 mysev postfix/smtpd[15921]: NOQUEUE: reject: RCPT from mail-wg0-f53.google.com[74.125.82.53]:
554 5.7.1 <user1@vdomain1.com>: Relay access denied; from=<sender+caf_=sender=sender@gmail.com> to=<user1@domain1.com> proto=ESMTP helo=<mail-wg0-f53.google.com>

1. User authentication system could be broken
All modern mail servers have a way to authenticate a user before it accepts a mail to be sent. So, if we notice ALL of the mail server users getting this error, we immediately check the user authentication settings of mail server.
For example, in Postfix mail server, the below setting enables SMTP authentication. If this is disabled in the configuration file, all the users will receive “554 5.7.1 : Relay access denied“.
smtpd_recipient_restrictions = permit_sasl_authenticated

tc server run level RHEL7

ln -s /usr//tcServer/APP/bin/init.d.sh /etc/init.d/tc_APP
/sbin/chkconfig –add tc_APP
/sbin/chkconfig –level 2345 tc_APP on

#!/bin/sh
#
# —————————————————————————
# tc Runtime application server bootup script
#
# Copyright (c) 2009-2011 VMware, Inc. All rights reserved.
# —————————————————————————
# chkconfig: 2345 80 20
# description: Start up the tc Runtime application server
# version: 2.9.2.RELEASE
# build date: 20130514153934

# Source function library.

# The user account that will run the tc Runtime instance
TC_RUNTIME_USER=”tcserver”

# DO NOT EDIT BEYOND THIS LINE
RETVAL=$?

setup () {
PRG=”$0″

while [ -h “$PRG” ]; do
ls=`ls -ld “$PRG”`
link=`expr “$ls” : ‘.*-> \(.*\)$’`
if expr “$link” : ‘/.*’ > /dev/null; then
PRG=”$link”
else
PRG=`dirname “$PRG”`/”$link”
fi
done

# Get standard environment variables
PRGDIR=`dirname “$PRG”`

#Absolute path
PRGDIR=`cd “$PRGDIR” ; pwd -P`
}

stop() {
if [ -x “$PRGDIR/tcruntime-ctl.sh” ]; then
echo “Stopping tcServer”
/bin/su $TC_RUNTIME_USER $PRGDIR/tcruntime-ctl.sh stop
RETVAL=$?
else
echo “Startup script $PRGDIR/tcruntime-ctl.sh doesn’t exist or is not executable.”
RETVAL=255
fi
}

status() {
if [ -x “$PRGDIR/tcruntime-ctl.sh” ]; then
echo “Status-ing tcServer”
/bin/su $TC_RUNTIME_USER $PRGDIR/tcruntime-ctl.sh status
RETVAL=$?
else
echo “Startup script $PRGDIR/tcruntime-ctl.sh doesn’t exist or is not executable.”
RETVAL=255
fi
}

start() {
if [ -x “$PRGDIR/tcruntime-ctl.sh” ]; then
echo “Starting tcServer”
/bin/su $TC_RUNTIME_USER $PRGDIR/tcruntime-ctl.sh start
RETVAL=$?
else
echo “Startup script $PRGDIR/tcruntime-ctl.sh doesn’t exist or is not executable.”
RETVAL=255
fi
echo “TEST: $PRGDIR : $TC_RUNTIME_USER : `ls -l $PRGDIR/tcruntime-ctl.sh`” > /tmp/test.log

}

#setup
PRGDIR=”/usr/mware/tcServerApps/VEESQ01/bin”

case “$1″ in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
status)
status
;;
*)
echo $”Usage: $0 {start|stop|restart|status}”
exit 1
;;
esac

exit $RETVAL

install wildfly 10 on centos 7

Introduction
WildFly formerly known as JBoss web and application server is an open source high performance web application server written in Java.

It is specially designed for high throughput and is able to handle millions of connections. WildFly is a simple, flexible, lightweight, managed application runtime that can help you build amazing applications easily. WildFly’s architecture is based on pluggable subsystems, so you can easily add or remove. This will allow you to reduce the overall disk footprint and memory overhead of the server. WildFly supports two modes: a single JVM (standalone mode) and a multi-JVM (domain mode) that can be used to synchronize configuration across any number of processes and hosts.

In this tutorial we will learn how to install and configure the WildFly application server on a server running CentOS 7.

Requirements
A server running CentOS 7.
A non-root user with sudo privilege setup on your server.
A static IP address [example: 192.168.1.22] configured on your server.

header

wget http://download.java.net/java/GA/jdk9/9/binaries/jdk-9+181_linux-x64_bin.tar.gz
UPDATED FOR JDK 8u151

TAR GZ:

wget –no-check-certificate -c –header “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz
UPDATED FOR JDK 8u144

TAR GZ:

wget –no-cookies –no-check-certificate –header “Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie” “http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz”
RPM:

wget -c –header “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.rpm
UPDATED FOR JDK 8u131

RPM:

wget -c –header “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm
TAR GZ:

wget -c –header “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
RPM using curl:

curl -v -j -k -L -H “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm > jdk-8u112-linux-x64.rpm
In all cases above, subst ‘i586’ for ‘x64’ to download the 32-bit build.

-j -> junk cookies
-k -> ignore certificates
-L -> follow redirects
-H [arg] -> headers

To start the wildfly instance, we need to run the script named ‘standalone.sh’ located in the ‘bin’ directory,
$ cd /data/wildfly-10.1.0.Final/bin
$ sh standalone.sh
This will start the wildfly instance. To access the instance, open the web browser & enter the following URL in the web browser
http://192.168.1.100:8080
This will open default wildfly page on the browser.
Step 4 – Accessing management console
To access the management console, URL is
http://192.168.1.100:9990
But we can’t access the page just yet, we need to create a management user to access the console. To create a management user, firstly stop the running instance by pressing ‘ctrl + c’ & then in ‘bin’ directory itself, run script named ‘add-user.sh’ ,
$ cd /data/wildfly-10.1.0.Final/bin
$ sh add-user.sh
Once the script starts, follow the onscreen steps to create a management user. After the username and password has been set up, same can be used to access the management console from the URL mentioned above.
This was our step by step tutorial on wildfly 10.1.0 installation. Please feel free to send your queries/questions or suggestions using the comment box below.

Read more: http://linuxtechlab.com/wildfly-10-10-1-0-installation/#ixzz4wrx5sWFT

#!/bin/bash
#title :wildfly-install.sh
#usage :/bin/bash wildfly-install.sh
#tested-version1 :10.0.0.CR3
#tested-distros1 :Ubuntu 15.10; Debian 7,8; CentOS 7; Fedora 22
#tested-version2 :10.0.0.Final
#tested-distros2 :Debian 8

WILDFLY_VERSION=10.0.0.Final
WILDFLY_FILENAME=wildfly-$WILDFLY_VERSION
WILDFLY_ARCHIVE_NAME=$WILDFLY_FILENAME.tar.gz
WILDFLY_DOWNLOAD_ADDRESS=http://download.jboss.org/wildfly/$WILDFLY_VERSION/$WILDFLY_ARCHIVE_NAME

INSTALL_DIR=/data/app
WILDFLY_FULL_DIR=$INSTALL_DIR/$WILDFLY_FILENAME
WILDFLY_DIR=$INSTALL_DIR/wildfly

WILDFLY_USER=”wildfly”
WILDFLY_SERVICE=”wildfly”
WILDFLY_MODE=”standalone”

WILDFLY_STARTUP_TIMEOUT=240
WILDFLY_SHUTDOWN_TIMEOUT=30

SCRIPT_DIR=”$( cd “$( dirname “${BASH_SOURCE[0]}” )” && pwd )”

if [[ $EUID -ne 0 ]]; then
echo “This script must be run as root.”
exit 1
fi

echo “Downloading: $WILDFLY_DOWNLOAD_ADDRESS…”
[ -e “$WILDFLY_ARCHIVE_NAME” ] && echo ‘Wildfly archive already exists.’
if [ ! -e “$WILDFLY_ARCHIVE_NAME” ]; then
wget -q $WILDFLY_DOWNLOAD_ADDRESS
if [ $? -ne 0 ]; then
echo “Not possible to download Wildfly.”
exit 1
fi
fi

echo “Cleaning up…”
rm -f “$WILDFLY_DIR”
rm -rf “$WILDFLY_FULL_DIR”
rm -rf “/var/run/$WILDFLY_SERVICE/”
rm -f “/etc/init.d/$WILDFLY_SERVICE”

echo “Installation…”
mkdir $WILDFLY_FULL_DIR
tar -xzf $WILDFLY_ARCHIVE_NAME -C $INSTALL_DIR
ln -s $WILDFLY_FULL_DIR/ $WILDFLY_DIR
useradd -s /sbin/nologin $WILDFLY_USER
chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR
chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR/

#mkdir -p /var/log/$WILDFLY_SERVICE

echo “Registrating Wildfly as service…”
# if should use systemd
if [ -x /bin/systemctl ]; then
# Script from $WILDFLY_DIR/docs/contrib/scripts/systemd/launch.sh didn’t work for me
cat > $WILDFLY_DIR/bin/launch.sh << "EOF" #!/bin/sh if [ "x$WILDFLY_HOME" = "x" ]; then WILDFLY_HOME="/data/app/wildfly" fi if [ "x$1" = "xdomain" ]; then echo 'Starting Wildfly in domain mode.' $WILDFLY_HOME/bin/domain.sh -c $2 -b $3 #>> /var/log/$WILDFLY_SERVICE/server-`date +%Y-%m-%d`.log
else
echo ‘Starting Wildfly in standalone mode.’
$WILDFLY_HOME/bin/standalone.sh -c $2 -b $3
#>> /var/log/$WILDFLY_SERVICE/server-`date +%Y-%m-%d`.log
fi
EOF
# $WILDFLY_HOME is not visible here
sed -i -e ‘s,WILDFLY_HOME=.*,WILDFLY_HOME=’$WILDFLY_DIR’,g’ $WILDFLY_DIR/bin/launch.sh
#sed -i -e ‘s,$WILDFLY_SERVICE,’$WILDFLY_SERVICE’,g’ $WILDFLY_DIR/bin/launch.sh
chmod +x $WILDFLY_DIR/bin/launch.sh

cp $WILDFLY_DIR/docs/contrib/scripts/systemd/wildfly.service /etc/systemd/system/$WILDFLY_SERVICE.service
WILDFLY_SERVICE_CONF=/etc/default/$WILDFLY_SERVICE
# To install multiple instances of Wildfly replace all hardcoding in systemd file
sed -i -e ‘s,EnvironmentFile=.*,EnvironmentFile=’$WILDFLY_SERVICE_CONF’,g’ /etc/systemd/system/$WILDFLY_SERVICE.service
sed -i -e ‘s,User=.*,User=’$WILDFLY_USER’,g’ /etc/systemd/system/$WILDFLY_SERVICE.service
sed -i -e ‘s,PIDFile=.*,PIDFile=/var/run/wildfly/’$WILDFLY_SERVICE’.pid,g’ /etc/systemd/system/$WILDFLY_SERVICE.service
sed -i -e ‘s,ExecStart=.*,ExecStart=’$WILDFLY_DIR’/bin/launch.sh $WILDFLY_MODE $WILDFLY_CONFIG $WILDFLY_BIND,g’ /etc/systemd/system/$WILDFLY_SERVICE.service
systemctl daemon-reload
#systemctl enable $WILDFLY_SERVICE.service
fi

# if non-systemd Debian-like distribution
if [ ! -x /bin/systemctl -a -r /lib/lsb/init-functions ]; then
cp $WILDFLY_DIR/docs/contrib/scripts/init.d/wildfly-init-debian.sh /etc/init.d/$WILDFLY_SERVICE
sed -i -e ‘s,NAME=wildfly,NAME=’$WILDFLY_SERVICE’,g’ /etc/init.d/$WILDFLY_SERVICE
WILDFLY_SERVICE_CONF=/etc/default/$WILDFLY_SERVICE
fi

# if non-systemd RHEL-like distribution
if [ ! -x /bin/systemctl -a -r /etc/init.d/functions ]; then
cp $WILDFLY_DIR/docs/contrib/scripts/init.d/wildfly-init-redhat.sh /etc/init.d/$WILDFLY_SERVICE
WILDFLY_SERVICE_CONF=/etc/default/wildfly.conf
chmod 755 /etc/init.d/$WILDFLY_SERVICE
fi

# if neither Debian nor RHEL like distribution
if [ ! -x /bin/systemctl -a ! -r /lib/lsb/init-functions -a ! -r /etc/init.d/functions ]; then
cat > /etc/init.d/$WILDFLY_SERVICE << "EOF" #!/bin/sh ### BEGIN INIT INFO # Provides: ${WILDFLY_SERVICE} # Required-Start: $local_fs $remote_fs $network $syslog # Required-Stop: $local_fs $remote_fs $network $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start/Stop ${WILDFLY_FILENAME} ### END INIT INFO WILDFLY_USER=${WILDFLY_USER} WILDFLY_DIR=${WILDFLY_DIR} case "$1" in start) echo "Starting ${WILDFLY_FILENAME}..." start-stop-daemon --start --background --chuid $WILDFLY_USER --exec $WILDFLY_DIR/bin/standalone.sh exit $? ;; stop) echo "Stopping ${WILDFLY_FILENAME}..." start-stop-daemon --start --quiet --background --chuid $WILDFLY_USER --exec $WILDFLY_DIR/bin/jboss-cli.sh -- --connect command=:shutdown exit $? ;; log) echo "Showing server.log..." tail -500f $WILDFLY_DIR/standalone/log/server.log ;; *) echo "Usage: /etc/init.d/wildfly {start|stop}" exit 1 ;; esac exit 0 EOF sed -i -e 's,${WILDFLY_USER},'$WILDFLY_USER',g; s,${WILDFLY_FILENAME},'$WILDFLY_FILENAME',g; s,${WILDFLY_SERVICE},'$WILDFLY_SERVICE',g; s,${WILDFLY_DIR},'$WILDFLY_DIR',g' /etc/init.d/$WILDFLY_SERVICE chmod 755 /etc/init.d/$WILDFLY_SERVICE fi if [ ! -z "$WILDFLY_SERVICE_CONF" ]; then echo "Configuring service..." echo JBOSS_HOME=\"$WILDFLY_DIR\" > $WILDFLY_SERVICE_CONF
echo JBOSS_USER=$WILDFLY_USER >> $WILDFLY_SERVICE_CONF
echo WILDFLY_HOME=\”$WILDFLY_DIR\” > $WILDFLY_SERVICE_CONF
echo WILDFLY_USER=\”$WILDFLY_USER\” > $WILDFLY_SERVICE_CONF
echo STARTUP_WAIT=$WILDFLY_STARTUP_TIMEOUT >> $WILDFLY_SERVICE_CONF
echo SHUTDOWN_WAIT=$WILDFLY_SHUTDOWN_TIMEOUT >> $WILDFLY_SERVICE_CONF
echo WILDFLY_CONFIG=$WILDFLY_MODE.xml >> $WILDFLY_SERVICE_CONF
echo WILDFLY_MODE=$WILDFLY_MODE >> $WILDFLY_SERVICE_CONF
echo WILDFLY_BIND=0.0.0.0 >> $WILDFLY_SERVICE_CONF
fi

echo “Configuring application server…”
sed -i -e ‘s,,,g’ $WILDFLY_DIR/$WILDFLY_MODE/configuration/$WILDFLY_MODE.xml
sed -i -e ‘s,,,g’ $WILDFLY_DIR/$WILDFLY_MODE/configuration/$WILDFLY_MODE.xml
sed -i -e ‘s,,,g’ $WILDFLY_DIR/$WILDFLY_MODE/configuration/$WILDFLY_MODE.xml
sed -i -e ‘s,,,g’ $WILDFLY_DIR/$WILDFLY_MODE/configuration/$WILDFLY_MODE.xml
sed -i -e ‘s,,,g’ $WILDFLY_DIR/$WILDFLY_MODE/configuration/$WILDFLY_MODE.xml

[ -x /bin/systemctl ] && systemctl start $WILDFLY_SERVICE || service $WILDFLY_SERVICE start

echo “Done.”

To start the wildfly instance, we need to run the script named ‘standalone.sh’ located in the ‘bin’ directory,
$ cd /data/wildfly/bin
$ sh standalone.sh
This will start the wildfly instance. To access the instance, open the web browser & enter the following URL in the web browser
http://192.168.1.22:8080
This will open default wildfly page on the browser.

To access the management console, URL is
http://192.168.1.22:9990
But we can’t access the page just yet, we need to create a management user to access the console. To create a management user, firstly stop the running instance by pressing ‘ctrl + c’ & then in ‘bin’ directory itself, run script named ‘add-user.sh’ ,
$ cd /data/wildfly/bin
$ sh add-user.sh
Once the script starts, follow the onscreen steps to create a management user. After the username and password has been set up, same can be used to access the management console from the URL mentioned above.
This was our step by step tutorial on wildfly 10.1.0 installation. Please feel free to send your queries/questions or suggestions using the comment box below.

DDoS Protection With IPtables

DDoS Protection With IPtables

There are different ways of building your own anti-DDoS rules for iptables. We will be discussing the most effective iptables DDoS protection methods in this comprehensive tutorial.
This guide will teach you how to:
Select the best iptables table and chain to stop DDoS attacks
Tweak your kernel settings to mitigate the effects of DDoS attacks
Use iptables to block most TCP-based DDoS attacks
Use iptables SYNPROXY to block SYN floods
Please note that this article is written for professionals who deal with Linux servers on a daily basis.
If you just want to protect your online application from DDoS attacks, you can use our remote protection, a VPS with DDoS protection or a DDoS protected bare metal server.
While one can do a lot with iptables to block DDoS attacks, there isn’t a way around actual hardware firewalls (we recently reviewed RioRey DDoS mitigation hardware) to detect and stop large DDoS floods.
However, it isn’t impossible to filter most bad traffic at line rate using iptables!
We’ll only cover protection from TCP-based attacks. Most UDP-based attacks are amplified reflection attacks that will exhaust the network interface card of any common server.
The only mitigation approach that makes sense against these types of attacks is to block them at the edge or core network or even at the carrier already.
Did you know we now offer VPS with unmetered bandwidth and DDoS protection in Chicago, Illinois and Bucharest, Romania?
If they are able to reach your server, there isn’t much you can do against those multi-Gbit/s attacks except to move to a DDoS protected network.
anti-DDoS iptables
What Is IPtables?
netfilter iptables (soon to be replaced by nftables) is a user-space command line utility to configure kernel packet filtering rules developed by netfilter.
It’s the default firewall management utility on Linux systems – everyone working with Linux systems should be familiar with it or have at least heard of it.
iptables can be used to filter certain packets, block source or destination ports and IP addresses, forward packets via NAT and a lot of other things.
Most commonly it’s used to block destination ports and source IP addresses.
Why Your IPtables Anti-DDoS Rules Suck
To understand why your current iptables rules to prevent DDoS attacks suck, we first have to dig into how iptables works.
iptables is a command line tool used to set up and control the tables of IP packet filter rules. There are different tables for different purposes.
IPtables Tables
Filter: The filter table is the default and most commonly used table that rules go to if you don’t use the -t (–table) option.
Nat: This table is used for Network Address Translation (NAT). If a packet creates a new connection, the nat table gets checked for rules.
Mangle: The mangle table is used to modify or mark packets and their header information.
Raw: This table’s purpose is mainly to exclude certain packets from connection tracking using the NOTRACK target.
As you can see there are four different tables on an average Linux system that doesn’t have non-standard kernel modules loaded. Each of these tables supports a different set of iptables chains.
IPtables Chains
PREROUTING: raw, nat, mangle
Applies to packets that enter the network interface card (NIC)
INPUT: filter, mangle
Applies to packets destined to a local socket
FORWARD: filter, mangle
Applies to packets that are being routed through the server
OUTPUT: raw, filter, nat, mangle
Applies to packets that the server sends (locally generated)
POSTROUTING: nat, mangle
Applies to packets that leave the server
Depending on what kind of packets you want to block or modify, you select a certain iptables table and a chain that the selected table supports.
Of course, we’re still missing an explanation of iptables targets (ACCEPT, DROP, REJECT, etc.), but we’re assuming that if you’re reading this article, you already know how to deal with iptables.
We’re going to explain why your iptables rules suck to stop DDoS and not teach you how to use iptables. Let’s get back to that.
If you want to block a DDoS attack with iptables, performance of the iptables rules is extremely important. Most TCP-based DDoS attack types use a high packet rate, meaning the sheer number of packets per second is what causes the server to go down.
That’s why you want to make sure that you can process and block as many packets per second as possible.
You’ll find that most if not all guides on how to block DDoS attacks using iptables use the filter table and the INPUT chain for anti-DDoS rules.
The issue with this approach is that the INPUT chain is only processed after the PREROUTING and FORWARD chains and therefore only applies if the packet doesn’t match any of these two chains.
This causes a delay in the filtering of the packet which consumes resources. In conclusion, to make our rules as effective as possible, we need to move our anti-DDoS rules as far up the chains as possible.
The first chain that can apply to a packet is the PREROUTING chain, so ideally we’ll want to filter the bad packets in this chain already.
However, the filter table doesn’t support the PREROUTING chain. To get around this problem, we can simply use the mangle table instead of the filter table for our anti-DDoS iptables rules.
It supports most if not all rules that the filter table supports while also supporting all iptables chains.
So you want to know why your iptables DDoS protection rules suck? It’s because you use the filter table and the INPUT chain to block the bad packets!
The best solution to dramatically increase the performance of your iptables rules and therefore the amount of (TCP) DDoS attack traffic they can filter is to use the mangle table and the PREROUTING chain!
The Best Linux Kernel Settings to Mitigate DDoS
Another common mistake is that people don’t use optimized kernel settings to better mitigate the effects of DDoS attacks.
Note that this guide focuses on CentOS 7 as the operating system of choice. CentOS 7 includes a recent version of iptables and support of the new SYNPROXY target.
We won’t cover every single kernel setting that you need to adjust in order to better mitigate DDoS with iptables.
Instead, we provide a set of CentOS 7 kernel settings that we would use. Just put the below in your /etc/sysctl.conf file and apply the settings with sysctl -p.
Anti-DDoS Kernel Settings (sysctl.conf)
kernel.printk = 4 4 1 7
kernel.panic = 10
kernel.sysrq = 0
kernel.shmmax = 4294967296
kernel.shmall = 4194304
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
vm.swappiness = 20
vm.dirty_ratio = 80
vm.dirty_background_ratio = 5
fs.file-max = 2097152
net.core.netdev_max_backlog = 262144
net.core.rmem_default = 31457280
net.core.rmem_max = 67108864
net.core.wmem_default = 31457280
net.core.wmem_max = 67108864
net.core.somaxconn = 65535
net.core.optmem_max = 25165824
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384
net.ipv4.neigh.default.gc_interval = 5
net.ipv4.neigh.default.gc_stale_time = 120
net.netfilter.nf_conntrack_max = 10000000
net.netfilter.nf_conntrack_tcp_loose = 0
net.netfilter.nf_conntrack_tcp_timeout_established = 1800
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 20
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 20
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 20
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 20
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 10
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_no_pmtu_disc = 1
net.ipv4.route.flush = 1
net.ipv4.route.max_size = 8048576
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.udp_rmem_min = 16384
net.ipv4.tcp_wmem = 4096 87380 33554432
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 400000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 10
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1
These sysctl.conf settings help to maximize the performance of your server under DDoS as well as the effectiveness of the iptables rules that we’re going to provide in this guide.
The Actual IPtables Anti-DDoS Rules
Considering you now know that you need to use the mangle table and the PREROUTING chain as well as optimized kernel settings to mitigate the effects of DDoS attacks, we’ll now move on to a couple of example rules to mitigate most TCP DDoS attacks.
DDoS attacks are complex.
There are many different types of DDoS and it’s close to impossible to maintain signature-based rules against all of them.
But luckily there is something called connection tracking (nf_conntrack kernel module), which can help us to mitigate almost any TCP-based DDoS attack that doesn’t use SYN packets that seem legitimate.
This includes all types of ACK and SYN-ACK DDoS attacks as well as DDoS attacks that use bogus TCP flags.
We’ll start with just five simple iptables rules that will already drop many TCP-based DDoS attacks.
Block Invalid Packets
iptables -t mangle -A PREROUTING -m conntrack –ctstate INVALID -j DROP
This rule blocks all packets that are not a SYN packet and don’t belong to an established TCP connection.
Block New Packets That Are Not SYN
iptables -t mangle -A PREROUTING -p tcp ! –syn -m conntrack –ctstate NEW -j DROP
This blocks all packets that are new (don’t belong to an established connection) and don’t use the SYN flag. This rule is similar to the “Block Invalid Packets” one, but we found that it catches some packets that the other one doesn’t.
Block Uncommon MSS Values
iptables -t mangle -A PREROUTING -p tcp -m conntrack –ctstate NEW -m tcpmss ! –mss 536:65535 -j DROP
The above iptables rule blocks new packets (only SYN packets can be new packets as per the two previous rules) that use a TCP MSS value that is not common. This helps to block dumb SYN floods.
Block Packets With Bogus TCP Flags
iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,SYN FIN,SYN -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags SYN,RST SYN,RST -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,RST FIN,RST -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,ACK FIN -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ACK,URG URG -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ACK,FIN FIN -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ACK,PSH PSH -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL ALL -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL NONE -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL FIN,PSH,URG -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL SYN,FIN,PSH,URG -j DROP
iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
The above ruleset blocks packets that use bogus TCP flags, ie. TCP flags that legitimate packets wouldn’t use.
Block Packets From Private Subnets (Spoofing)
iptables -t mangle -A PREROUTING -s 224.0.0.0/3 -j DROP
iptables -t mangle -A PREROUTING -s 169.254.0.0/16 -j DROP
iptables -t mangle -A PREROUTING -s 172.16.0.0/12 -j DROP
iptables -t mangle -A PREROUTING -s 192.0.2.0/24 -j DROP
iptables -t mangle -A PREROUTING -s 192.168.0.0/16 -j DROP
iptables -t mangle -A PREROUTING -s 10.0.0.0/8 -j DROP
iptables -t mangle -A PREROUTING -s 0.0.0.0/8 -j DROP
iptables -t mangle -A PREROUTING -s 240.0.0.0/5 -j DROP
iptables -t mangle -A PREROUTING -s 127.0.0.0/8 ! -i lo -j DROP
These rules block spoofed packets originating from private (local) subnets. On your public network interface you usually don’t want to receive packets from private source IPs.
These rules assume that your loopback interface uses the 127.0.0.0/8 IP space.
These five sets of rules alone already block many TCP-based DDoS attacks at very high packet rates.
With the kernel settings and rules mentioned above, you’ll be able to filter ACK and SYN-ACK attacks at line rate.
Additional Rules
iptables -t mangle -A PREROUTING -p icmp -j DROP
This drops all ICMP packets. ICMP is only used to ping a host to find out if it’s still alive. Because it’s usually not needed and only represents another vulnerability that attackers can exploit, we block all ICMP packets to mitigate Ping of Death (ping flood), ICMP flood and ICMP fragmentation flood.
iptables -A INPUT -p tcp -m connlimit –connlimit-above 80 -j REJECT –reject-with tcp-reset
This iptables rule helps against connection attacks. It rejects connections from hosts that have more than 80 established connections. If you face any issues you should raise the limit as this could cause troubles with legitimate clients that establish a large number of TCP connections.
iptables -A INPUT -p tcp -m conntrack –ctstate NEW -m limit –limit 60/s –limit-burst 20 -j ACCEPT
iptables -A INPUT -p tcp -m conntrack –ctstate NEW -j DROP
Limits the new TCP connections that a client can establish per second. This can be useful against connection attacks, but not so much against SYN floods because the usually use an endless amount of different spoofed source IPs.
iptables -t mangle -A PREROUTING -f -j DROP
This rule blocks fragmented packets. Normally you don’t need those and blocking fragments will mitigate UDP fragmentation flood. But most of the time UDP fragmentation floods use a high amount of bandwidth that is likely to exhaust the capacity of your network card, which makes this rule optional and probably not the most useful one.
iptables -A INPUT -p tcp –tcp-flags RST RST -m limit –limit 2/s –limit-burst 2 -j ACCEPT
iptables -A INPUT -p tcp –tcp-flags RST RST -j DROP
This limits incoming TCP RST packets to mitigate TCP RST floods. Effectiveness of this rule is questionable.
Mitigating SYN Floods With SYNPROXY
SYNPROXY is a new target of iptables that has been added in Linux kernel version 3.12 and iptables 1.4.21. CentOS 7 backported the feature and it’s available in its 3.10 default kernel.
The purpose of SYNPROXY is to check whether the host that sent the SYN packet actually establishes a full TCP connection or just does nothing after it sent the SYN packet.
If it does nothing, it discards the packet with minimal performance impact.
While the iptables rules that we provided above already block most TCP-based attacks, the attack type that can still slip through them if sophisticated enough is a SYN flood.
It’s important to note that the performance of the rules will always be better if we find a certain pattern or signature to block, such as packet length (-m length), TOS (-m tos), TTL (-m ttl) or strings and hex values (-m string and -m u32 for the more advanced users).
But in some rare cases that’s not possible or at least not easy to achieve. So, in these cases, you can make use of SYNPROXY.
Here are iptables SYNPROXY rules that help mitigate SYN floods that bypass our other rules:

hese rules apply to all ports. If you want to use SYNPROXY only on certain TCP ports that are active (recommended – also you should block all TCP ports that are not in use using the mangle table and PREROUTING chain), you can just add –dport 80 to each of the rules if you want to use SYNPROXY on port 80 only.
To verify that SYNPROXY is working, you can do watch -n1 cat /proc/net/stat/synproxy. If the values change when you establish a new TCP connection to the port you use SYNPROXY on, it works.
The Complete IPtables Anti-DDoS Rules
If you don’t want to copy & paste each single rule we discussed in this article, you can use the below ruleset for basic DDoS protection of your Linux server.
### 1: Drop invalid packets ###
/sbin/iptables -t mangle -A PREROUTING -m conntrack –ctstate INVALID -j DROP

### 2: Drop TCP packets that are new and are not SYN ###
/sbin/iptables -t mangle -A PREROUTING -p tcp ! –syn -m conntrack –ctstate NEW -j DROP

### 3: Drop SYN packets with suspicious MSS value ###
/sbin/iptables -t mangle -A PREROUTING -p tcp -m conntrack –ctstate NEW -m tcpmss ! –mss 536:65535 -j DROP

### 4: Block packets with bogus TCP flags ###
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,SYN FIN,SYN -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags SYN,RST SYN,RST -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,RST FIN,RST -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags FIN,ACK FIN -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ACK,URG URG -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ACK,FIN FIN -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ACK,PSH PSH -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL ALL -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL NONE -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL FIN,PSH,URG -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL SYN,FIN,PSH,URG -j DROP
/sbin/iptables -t mangle -A PREROUTING -p tcp –tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP

### 5: Block spoofed packets ###
/sbin/iptables -t mangle -A PREROUTING -s 224.0.0.0/3 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 169.254.0.0/16 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/12 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 192.0.2.0/24 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 192.168.0.0/16 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 10.0.0.0/8 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 0.0.0.0/8 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 240.0.0.0/5 -j DROP
/sbin/iptables -t mangle -A PREROUTING -s 127.0.0.0/8 ! -i lo -j DROP

### 6: Drop ICMP (you usually don’t need this protocol) ###
/sbin/iptables -t mangle -A PREROUTING -p icmp -j DROP

### 7: Drop fragments in all chains ###
/sbin/iptables -t mangle -A PREROUTING -f -j DROP

### 8: Limit connections per source IP ###
/sbin/iptables -A INPUT -p tcp -m connlimit –connlimit-above 111 -j REJECT –reject-with tcp-reset

### 9: Limit RST packets ###
/sbin/iptables -A INPUT -p tcp –tcp-flags RST RST -m limit –limit 2/s –limit-burst 2 -j ACCEPT
/sbin/iptables -A INPUT -p tcp –tcp-flags RST RST -j DROP

### 10: Limit new TCP connections per second per source IP ###
/sbin/iptables -A INPUT -p tcp -m conntrack –ctstate NEW -m limit –limit 60/s –limit-burst 20 -j ACCEPT
/sbin/iptables -A INPUT -p tcp -m conntrack –ctstate NEW -j DROP

### 11: Use SYNPROXY on all ports (disables connection limiting rule) ###
# Hidden – unlock content above in “Mitigating SYN Floods With SYNPROXY” section
Bonus Rules
Here are some more iptables rules that are useful to increase the overall security of a Linux server:
### SSH brute-force protection ###
/sbin/iptables -A INPUT -p tcp –dport ssh -m conntrack –ctstate NEW -m recent –set
/sbin/iptables -A INPUT -p tcp –dport ssh -m conntrack –ctstate NEW -m recent –update –seconds 60 –hitcount 10 -j DROP

### Protection against port scanning ###
/sbin/iptables -N port-scanning
/sbin/iptables -A port-scanning -p tcp –tcp-flags SYN,ACK,FIN,RST RST -m limit –limit 1/s –limit-burst 2 -j RETURN
/sbin/iptables -A port-scanning -j DROP
Conclusion
This tutorial demonstrates some of the most powerful and effective methods to stop DDoS attacks using iptables.
We’ve successfully mitigated DDoS attacks that peaked at multiple million packets per second using these iptables rules.
Every single guide on the same topic that we had researched provided inefficient methods to stop DDoS traffic or only a very limited number of iptables rules.
If used correctly, iptables is an extremely powerful tool that’s able to block different types of DDoS attacks at line-rate of 1GigE NICs and close to line-rate of 10GigE NICs.

wireshark cmd

wireshark cmd

ip.dst==192.168.6.58
ip.src==192.168.6.58

ip.dst==192.168.6.58 and http
ip.dst==192.168.6.58 and ssl

up vote
20
down vote
Filtering IP Address in Wireshark:

(1)single IP filtering:

ip.addr==X.X.X.X

ip.src==X.X.X.X

ip.dst==X.X.X.X

(2)Multiple IP filtering based on logical conditions:

OR condition:

(ip.src==192.168.2.25)||(ip.dst==192.168.2.25)

AND condition:

(ip.src==192.168.2.25) && (ip.dst==74.125.236.16)

The filtering capabilities of Wireshark are very comprehensive. You can filter on just about any field of any protocol, even down to the HEX values in a data stream. Sometimes though, the hardest part about setting a filter in Wireshark is remembering the syntax! So below are the top 10 display filters that I use in Wireshark. Please comment below and add any common ones that you use as well.

1. ip.addr == 10.0.0.1 [Sets a filter for any packet with 10.0.0.1, as either the source or dest]

2. ip.addr==10.0.0.1 && ip.addr==10.0.0.2 [sets a conversation filter between the two defined IP addresses]

3. http or dns [sets a filter to display all http and dns]

4. tcp.port==4000 [sets a filter for any TCP packet with 4000 as a source or dest port]

5. tcp.flags.reset==1 [displays all TCP resets]

6. http.request [displays all HTTP GET requests]

7. tcp contains traffic [displays all TCP packets that contain the word ‘traffic’. Excellent when searching on a specific string or user ID]

8. !(arp or icmp or dns) [masks out arp, icmp, dns, or whatever other protocols may be background noise. Allowing you to focus on the traffic of interest]

9. udp contains 33:27:58 [sets a filter for the HEX values of 0x33 0x27 0x58 at any offset]

10. tcp.analysis.retransmission [displays all retransmissions in the trace. Helps when tracking down slow application performance and packet loss]

Recovering Windows Password – Sticky Keys Technique

he “sticky keys” technique for resetting the password on a Windows system is described in many places around the web, but all tend to over-complicate the steps. I’m describing it here for my own reference, so I have a reminder of how it works.

The technique basically provides access to a command prompt from the Windows login page – the page right at the start with an icon for each user. This page appears to run with administrator privileges, so the command prompt can be used to reset any password of any user without knowing any further details.

The command prompt is invoked by moving cmd.exe into the place of the command that handles the “sticky keys” functionality in the login page.

The steps are:

Rename c:windowssystem32sethc.exe out of the way (to sethc-bak.exe for example).
Copy c:windowssystem32cmd.exe to c:windowssystem32sethc.exe
Boot up the PC to the account selection screen.
Press the shift key five times. This will bring up a command prompt window.
Reset the password at the command prompt: net user your_user_name new_password
Log in using the new password.
Remove c:windowssystem32sethc.exe and restore the saved sethc-bak.exe
The first step involves getting to the file system on the machine. If you just have one machine and Windows 7 install disks handy, then you can boot up the recovery console to a command prompt and rename the files from there.

You may not be able to do step 7 until after a reboot. You may not be able to remove sethd.exe while Windows is running, but you should be able to rename it to sethd-delete.exe or similar to make room to restore the original file.

When I’m recovering the passwords for a laptop, however, I find it easier to just slip the hard drive out and pop it into another machine by USB and rename and copy the files from Windows Explorer. One of these USB-to-SATA docking stations comes in very handy for doing that:

Page 2 of 16412345...102030...Last »