SlideShare a Scribd company logo
1 of 89
Download to read offline
Application Layer DDoS
A Practical Approach & Mitigation Techniques
Fakrul Alam
bdHUB Limited
fakrul@bdhub.com
Tools used to demonstrate DDoS attack is for educational /
knowledge sharing purpose only. No intention to generate DDoS
attack on production network.
Disclaimer
XXIV
Agenda
• Background
• Application / Layer 7 DDoS
• Practical Approach (Case Study)
• Mitigation
• Simulation
• Key findings & Issues
BACKGROUND
XXIV
Background : What is DDoS
• Denial of Service (DoS) / Distributed Denial of Service (DDoS)
is the act of performing an attack which prevents the system
from providing services to legitimate users
• Denial of Service attacks take many forms, and utilize many
attack vectors
• When successful, the targeted host may stop providing any
service, provide limited services only or provide services to
some users only
• DDoS attack sometime refer as Distributed Reflection Denial of
Service (DrDoS) Attack
XXIV
Background : DDoS Attack Phases
• Phase One: Target Acquisition
• Phase Two: Groundwork
• Phase Three: ATTACK
XXIV
Background : About Botnets
A Botnet can

generate

1 Million
Times
the available

bandwidth

of a business
It takes just

64,000 PCs
infected with

a virus like

Conficker

to generate

10 gigabits
Per second
of traffic
Mariposa, the 

largest known

Botnet, affected

12 million PCs
It could have

generate a

DDoS attack 

as large as

31.2
Terabytes
Per second
XXIV
Background : DDoS Insurance
• Insurance is money you pay to be protected from
• Happen / Might Not Happen
• You can be prepared
• Incident response plan
• Tools
• Gear
• Partnerships
• It may not be sufficient – you should have picked the higher premium policy…
XXIV
Background : Types of Attacks
• Volume Based Attacks
• Protocol Attacks
• Application Layer Attacks
XXIV
Background : Statistics
Source: PROLEXIC Q3 2013 Attack Report
XXIV
Background : DDoS Attack Surface
• Past DDOS attacks were mainly Layer 3 / Layer 4 attacks
Layer 7
Application
Layer 6
Presentation
Layer 5
Session
Layer 4
Transport
Layer 3
Network
Layer 2
Data Link
Layer 1
Physical
Application
Telnet / FTP / NFS / WWW
Session
RPC
Transport
Sockets / Streams
Network
IP + ARP / RARP / ICMP
Physical Protocol
Ethernet / FDD / PPP
Transmission Media
Coax / Fiber
TCP UDP
XXIV
Layer 3 DDoS Attack
• Layer 3 - muscle-based attacks
• Flood of TCP/UDP/ICMP/IGMP packets, overloading
infrastructure due to high rate processing/discarding of
packets and filling up the packet queues, or saturating
pipes
• Introduce a packet workload most gear isn't designed for
• Example - UDP flood to non-listening port
XXIV
Layer 4 DDoS Attack
• Layer 4 – slightly more sophisticated
• DoS attacks consuming extra memory, CPU cycles, and triggering
responses
• TCP SYN flood
• TCP new connections flood
• TCP concurrent connections exhaustion
• TCP/UDP garbage data flood to listening services (ala LOIC)
• Example – SYN flood
XXIV
Layer 7 DDoS
• Layer 7 - The Evil
• DoS attacks abusing application-server memory and performance
limitations – masquerading as legitimate transactions
• HTTP page flood
• HTTP bandwidth consumption
• DNS query flood
• SIP INVITE flood
• Low rate, high impact attacks – e.g. Slowloris, HTTP POST DoS
XXIV
Background : DDoS Attack Costs
Damage to Your Brand Loss of Revenue
Bad Customer
Exeprience
If your site is down,
account holders will
question if you provide
safe service
If your website is down,
you loos revenue
Call center agent get
overwhelmed
Ruins years of work
building your brand
No online banking, bill
pay, forms or application,
account opening
Account holders
frustration
People seek alternatives
XXIV
Background : DDoS Additional Threat
• DDoS attacks are more frequently being used to hide
security breaches and data theft
• Attention focuses on attack
• Log files get massive, to difficult to analyze quickly
• Servers and routers are rebooted, often destroying
forensic evidence
• Attacks end long before any intrusion is identified
XXIV
• SPAMHAUS : MARCH 27 2013
Background : DDoS Attack Volume
XXIV
Background : DDoS Attack Volume (DNS)
1 Attacker Laptop controlling
+ 5-7 compromised server on
+ 3 networks that allowed spoofing of
+ 9 Gbps DNS request to
+ 0.1% of open resolvers resulted in
= 300 Gbps+ DDoS attack traffic
[source: cloudflare]
dig ANY isc.org @OpenResolverIP
+edns=0 +notcp +bufsize=4096
XXIV
fkfkfkfa.com. 84930 IN A 204.46.43.16
fkfkfkfa.com. 84930 IN A 204.46.43.17
fkfkfkfa.com. 84930 IN A 204.46.43.18
fkfkfkfa.com. 84930 IN A 204.46.43.19
fkfkfkfa.com. 84930 IN A 204.46.43.20
fkfkfkfa.com. 84930 IN A 204.46.43.21
fkfkfkfa.com. 84930 IN A 204.46.43.22
fkfkfkfa.com. 84930 IN A 204.46.43.23
fkfkfkfa.com. 84930 IN A 204.46.43.24
fkfkfkfa.com. 84930 IN A 204.46.43.25
fkfkfkfa.com. 84930 IN A 204.46.43.26
fkfkfkfa.com. 84930 IN A 204.46.43.27
fkfkfkfa.com. 84930 IN A 204.46.43.28
fkfkfkfa.com. 84930 IN A 204.46.43.29
fkfkfkfa.com. 84930 IN A 204.46.43.30
fkfkfkfa.com. 84930 IN A 204.46.43.31
fkfkfkfa.com. 84930 IN A 204.46.43.32
fkfkfkfa.com. 84930 IN A 204.46.43.33
fkfkfkfa.com. 84930 IN A 204.46.43.34
fkfkfkfa.com. 84930 IN A 204.46.43.35
fkfkfkfa.com. 84930 IN A 204.46.43.36
fkfkfkfa.com. 84930 IN A 204.46.43.37
fkfkfkfa.com. 84930 IN A 204.46.43.38
fkfkfkfa.com. 84930 IN A 204.46.43.39
fkfkfkfa.com. 84930 IN A 204.46.43.40
fkfkfkfa.com. 84930 IN A 204.46.43.41
fkfkfkfa.com. 84930 IN A 204.46.43.42
fkfkfkfa.com. 84930 IN A 204.46.43.43
fkfkfkfa.com. 84930 IN A 204.46.43.44
fkfkfkfa.com. 84930 IN A 204.46.43.45
fkfkfkfa.com. 84930 IN NS us3.fkfkfkfa.com.
fkfkfkfa.com. 84930 IN A 204.46.43.46
fkfkfkfa.com. 84930 IN NS us4.fkfkfkfa.com.
!
;; AUTHORITY SECTION:
fkfkfkfa.com. 84930 IN NS us3.fkfkfkfa.com.
fkfkfkfa.com. 84930 IN NS us4.fkfkfkfa.com.
!
;; Query time: 83 msec
;; SERVER: 103.12.178.xxx#53(103.12.178.xxx)
;; WHEN: Sat Dec 28 17:46:24 2013
;; MSG SIZE rcvd: 4002
Background : DDoS Attack Volume (DNS)
• dig ANY fkfkfkfa.com @103.12.178.XXX
+edns=0 +notcp +bufsize=4096
4002/64 = 62x
amplification
XXIV
fn-vm1.ndc.nasa.gov 123 198.123.30.132 6 3 4 190 6 27
m000044.ppp.asahi-net. 123 198.123.30.132 1 3 4 190 27 27
106.66.131.129 45826 198.123.30.132 1 3 3 190 27 27
4.26.19.130 289 198.123.30.132 1 3 1 190 27 27
host-24-100-155-153.ne 32769 198.123.30.132 65 3 3 190 5 28
78-62-146-36.static.ze 33494 198.123.30.132 1 3 3 190 28 28
77.68.208.137 32776 198.123.30.132 1 3 4 190 28 28
75-139-155-2.dhcp.gwnt 123 198.123.30.132 1 3 4 190 28 28
d58-106-139-87.rdl802. 123 198.123.30.132 1 3 4 190 28 28
c-71-202-159-26.hsd1.c 39741 198.123.30.132 1 3 4 190 28 28
f6-bamboo-r4.ndc.nasa. 123 198.123.30.132 8 3 4 190 5 28
77.68.201.102 32823 198.123.30.132 1 3 4 190 29 29
117.195.46.81 20609 198.123.30.132 12 3 3 190 29 29
87-194-180-251.bethere 123 198.123.30.132 4 3 4 190 7 29
d54C0EAF1.access.telen 123 198.123.30.132 1 3 4 190 29 29
77.68.132.75 1060 198.123.30.132 1 3 4 190 29 29
187-162-58-152.static. 123 198.123.30.132 1 3 4 190 29 29
ynrfw23001.yr.com 480 198.123.30.132 13 3 3 590 4 29
115-39-41-60.mie1.comm 4913 198.123.30.132 1 3 3 190 29 29
primary.idb.com.au 199 198.123.30.132 1 3 4 190 29 29
206-248-138-67.dsl.tek 123 198.123.30.132 1 3 4 190 29 29
pool-173-66-167-144.wa 60137 198.123.30.132 2 3 1 590 14 29
dslsonicwall.btconline 23891 198.123.30.132 1 3 1 190 30 30
108-192-128-52.lightsp 123 198.123.30.132 1 3 4 190 30 30
static-50-44-202-198.o 11547 198.123.30.132 1 3 1 190 30 30
117.136.19.162 21829 198.123.30.132 1 3 3 190 30 30
cm68.eta123.maxonline. 123 198.123.30.132 1 3 4 190 30 30
89.169.64.236 123 198.123.30.132 6 3 4 190 6 30
121.61-66-87.adsl-dyn. 123 198.123.30.132 1 3 4 190 30 30
pool-98-118-84-72.bstn 123 198.123.30.132 1 3 4 190 30 30
218.56.20.142 1100 198.123.30.132 1 3 3 190 30 30
66.162.156.150 41755 198.123.30.132 4 3 4 190 10 30
173-11-46-77-Minnesota 4847 198.123.30.132 2 3 4 190 17 30
61-195-152-9.cust.bit- 123 198.123.30.132 1 3 4 190 31 31
rionicapital1.pndsl.co 50071 198.123.30.132 1 3 3 190 31 31
vespa.ndc.nasa.gov 123 198.123.30.132 1 3 4 190 31 31
host87-93-dynamic.183- 32850 198.123.30.132 1 3 3 190 31 31
odin000956850.ndc.nasa 123 198.123.30.132 1 3 4 190 31 31
241.wolainfo.com.pl 123 198.123.30.132 1 3 3 190 31 31
112.90.239.142 43793 198.123.30.132 1 3 3 190 31 31
cpe-071-070-131-156.nc 123 198.123.30.132 1 3 4 190 31 31
88.230.191.53.dynamic. 23460 198.123.30.132 1 3 3 190 31 31
ARouen-652-1-378-137.w 28247 198.123.30.132 2 3 3 190 16 31
77.68.194.63 32825 198.123.30.132 1 3 4 190 32 32
77.68.154.162 1042 198.123.30.132 1 3 4 190 32 32
Background : DDoS Attack Volume (NTP)
• ntpdc -c monlist ntp.nasa.gov
XXIV
Background : DDoS Attack Volume (NTP)
XXIV
Background : DDoS Attack Volume
• Source Address Spoofing
• Ingress Filtering / BCP 38 (http://tools.ietf.org/html/bcp38)
• Secure DNS
• http://www.cymru.com/Documents/secure-bind-
template.html
• Secure NTP
• http://www.team-cymru.org/ReadingRoom/Templates/
secure-ntp-template.html
XXIV
Background : DDoS or Network Stress Test!
LAYER 7 DDOS
XXIV
Layer 7 DDoS : Overview
• Application layer DoS attacks are evolving as part of the
evolution of application attacks
• The denied service is the application itself (rather than the
host) – effectively preventing usage of the system.
• Take advantage of flaws in the code to perform the DoS
• The benefit for the attacker – does not require the same
effort to achieve as a DDoS attack
XXIV
Layer 7 DDoS : Overview
DoS can be achieved in various ways:
• Application Crashing
• Data Destruction
• Resource Depletion
XXIV
Layer 7 DDoS : Application Crashing
• Common way of performing a Denial of Service attack
• In many cases, certain types of inputs may yield an error in the
application which it did not anticipate, and will cause it to
crash:
• Buffer Overflows
• Malformed data – causing parser exception
• Terminating with error
• SQL Injection
XXIV
Layer 7 DDoS : Data Destruction
• One way to cause a DoS attack is by tampering with the
data instead of the service itself
• If a site is vulnerable to SQL Injection, for instance, it may
be possible to DELETE all data from all tables
• Although the Web site will keep being ‘online’, it will
actually be useless without the information from the
Database
XXIV
Layer 7 DDoS : Resource Depletion
• Resource Depletion is a technique of performing DoS attacks on any
site or application
• Classical Resource Depletion simply utilizes very large amounts of
attacker resources which includes
• Memory
• CPU
• Disk Space
• Sophisticated attacks pinpoint the weak points of the application to
achieve maximum effect using minimal resources
XXIV
Effectiveness of Layer 7 DDoS
• Higher Obscurity
• Higher Efficiency
• Higher Lethality
XXIV
Layer 7 DDoS Web Attack
• Causes related to your inefficient codes
• Protocol Weakness
• HTTP GET
• HTTP POST
XXIV
HTTP GET DDoS Attack
XXIV
HTTP POST DDoS Attack
XXIV
Layer 7 DDoS Tools
• Slowloris abuses handling of
HTTP request headers
ssslooowly…
• Written by RSnake
• Iteratively injects one custom
header at a time and goes to
sleep
• Web server vainly awaits the line
space that will never come
• R-U-Dead-Yet? abuses HTTP
web form fields
• Iteratively injects one custom byte
into a web application post field
and goes to sleep
• Application threads become
zombies awaiting ends of posts till
death lurks upon the website
XXIV
Low Orbit Ion Cannon (LOIC)
• LOIC v.1.0.0.0
• LOIC v.1.1.1.16
XXIV
Low Orbit Ion Cannon (LOIC) - Android
XXIV
And Others Also!!
CASE STUDY PRACTICAL APPROACH
XXIV
Case Study
• Time: August 2012
• Country: Bangladesh
• Site: www.prothom-alo.com
• Ranked top 10 sites in
Bangladesh (Source: Alexa)
XXIV
Initial Findings
• Massive HTTP GET Flood
• Site is not accessible
• There is no major changes in bandwidth utilization
• Proper monitoring not in place to identify the actual attack
• Attack source is from Russia, China and some countries
from Africa
XXIV
Initial Findings : Logs
186.58.179.33 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/
4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR
2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" !
189.76.197.117 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.19) Gecko/20081201 Firefox/
2.0.0.19”!
186.58.179.33 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/
4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR
2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618)”!
186.6.168.11 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/
4.0 (compatible; MSIE 5.0; Windows 2000) Opera 6.03 [en]" 197.0.165.121 - - [21/Oct/
2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.4.154.25 Safari/525.19”!
189.76.197.117 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.19) Gecko/20081201 Firefox/
2.0.0.19”!
197.0.165.121 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/
5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/
0.4.154.25 Safari/525.19”
XXIV
Approach 1
• Solution from hosting company
• Conventional host based firewall using IPTABLES.
• Fine tune TCP parameters
• Enable SYN Cookies
• echo 1 > /proc/sys/net/ipv4/tcp_syncookies!
• Enable socket reuse
• echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle!
• echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse!
• Increase local port range
• echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
XXIV
Issue with Approach 1
• Solution from hosting company required additional $$$$
which is significantly high
• Hard to justify management
• Host based firewall works only in Layer 3 & Layer 4
• Not capable to filter Layer 7 DDoS Attack
XXIV
Approach 2
• Split DNS
• DNS configured to resolve host based on GEOIP.
• External uses request redirected to external server hosted
in USA
• One new server co-located in Bangladesh
• Internal (Bangladesh) traffic has been redirected to new
server
• Load has been distributed
XXIV
Issue with Approach 2
• Issue with Split DNS
• 4.2.2.2, 8.8.8.8 and other Open DNS
• Lots of users from Bangladesh is using open DNS like
4.2.2.2 & 8.8.8.8.
• For those users DNS is still resolving USA data center
server IP
XXIV
Approach 3
• Anycast
• Failed
• Most of the upstream provider and datacenter doesn’t
allow anycast
• It’s good in handling volumetric attack
XXIV
Approach 4
• Reverse Web Proxy
• Use Reverse Proxy as frontend
• Anti DDoS plugins along with other parameters
• Minimize the attack vector
• Distribute end user load and mitigation solution
MIGRATION SOLUTION ARCHITECTURE
XXIV
Solution Architecture
Web Server
Collocated in Bangladesh
Web Server
Collocated in USA
Reverse Web Proxy
(Front End)
Spit DNS: Decentralize load based on GEOIP

Reverse Web Proxy with DDoS plugins as front end
XXIV
Reverse Web Proxy
• A reverse proxy is a type of proxy server that retrieves
resources on behalf of a client from one or more servers.
These resources are then returned to the client as though
they originated from the server itself (or servers
themselves) –Wikipedia
XXIV
Reverse Web Proxy
XXIV
Why NGINX
• Event Driven
• Asynchronous
• Single Threaded
XXIV
Nginx DDoS Plugins
• Available plugins:
• testcookie-nginx-module [http://kyprizel.github.io/
testcookie-nginx-module/]
• Roboo : HTTP Robot Mitigator [http://www.ecl-
labs.org/2011/03/17/roboo-http-mitigator.html]
XXIV
testcookie-nginx-module
• testcookie-nginx-module is a simple robot mitigation
module using cookie based challenge/response
technique.
• Challenge cookies can be set using different methods:
• "Set-Cookie" + 307/302 HTTP Location redirect
• "Set-Cookie" + HTML meta refresh redirect
• If you need Captcha or Flash, check testcookie-flash-
processor
XXIV
testcookie-nginx-module
XXIV
Roboo : HTTP Robot Mitigator
• Uses advanced non-interactive HTTP challenge/response mechanisms to detect & mitigate
HTTP Robots
• Weeds out the larger percentage of HTTP robots which do not use real browsers or
implement full browser stacks, resulting in the mitigation of various web threats:
• HTTP Denial of Service tools - e.g. Low Orbit Ion Cannon
• Vulnerability Scanning - e.g. Acunetix Web Vulnerability Scanner, Metasploit Pro, Nessus
• Web exploits
• Automatic comment posters/comment spam as a replacement of conventional
CAPTCHA methods
• Spiders, Crawlers and other robotic evil
• Available at https://github.com/yuri-gushin/Roboo
XXIV
Roboo : HTTP Robot Mitigator
• Will respond to each GET or POST request from an unverified source
with a challenge:
• Challenge can be Javascript or Flash based, optionally Gzip
compressed
• A real browser with full HTTP, HTML, Javascript and Flash player
stacks will re-issue the original request after setting a special HTTP
cookie that marks the host as “verified”
• Marks verified sources using an HTTP Cookie
• Integrates with Nginx web server and reverse proxy as an embedded
Perl module
XXIV
Configuration Snap (nginx.conf)
XXIV
Configuration Snap (Roboo)
XXIV
Configuration Snap (testcookie-nginx-
module)
XXIV
Key Configuration Parameters
Variables Description
worker_processes This number should be, at maximum, the number of CPU cores on
your system.
worker_connections Determines how many clients will be served by each worker
process.

(Max clients = worker_connections * worker_processes)
perl_modules /opt/
local/share/nginx;!
perl_require Roboo.pm;
Enabling Roboo Plugings
map $http_user_agent Define http agent (httrack|WinHTTrack|htmlparser|libwww|Python)
XXIV
Key Configuration Parameters
Variables Description
$http_referer (babes|click|forsale|jewelry|nudit)
limit_req_zone
$binary_remote_addr
zone=req_limit_per_ip:10m
rate=1r/s;!
limit_conn_zone
$binary_remote_addr
zone=conn_limit_per_ip:
10m;!
creates zone “req_limit_per_ip” allocating 10MB for this session
then limits queries for remote ip address to 1 request per second
include /etc/nginx/
allow_only.conf
Can define IP address for where site is only accessible
XXIV
Logs : Roboo
challenged.log
202.4.100.35 - - [28/Nov/2013:14:05:10 +0600] "GET / HTTP/1.1" 200
669 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/
537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36"!
202.4.100.35 - - [28/Nov/2013:14:05:11 +0600] "GET /Anti-Robot-
GET-2babb27395588042480c.swf HTTP/1.1" 200 1025 "http://
www1.prothom-alo.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57
Safari/537.36”!
verified.log
202.4.100.35 - - [28/Nov/2013:14:05:12 +0600] "GET / HTTP/1.1" 200
31942 "http://www1.prothom-alo.com/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
31.0.1650.57 Safari/537.36"
XXIV
Logs : testcookie-nginx-module
202.4.100.35 - - [30/Nov/2013:18:06:53 +0600]
"GET /?ckattempt=1 HTTP/1.1" 200 31643 "-"
"Mozilla/5.0 (iPhone; CPU iPhone OS 7_0_4 like Mac
OS X) AppleWebKit/537.51.1 (KHTML, like Gecko)
Version/7.0 Mobile/11B554a Safari/9537.53"
SIMULATION
XXIV
Solution Architecture
Web Server : NGINX Attack Server
Web Server (web.bdnog.org)
192.168.1.150
• All the hardwares are configured in Virtual Box
• DDoS launched in closed network
• Please don’t try in production network
Reverse Proxy (nx.bdnog.org)
192.168.1.100
Attack Server (attack.bdnog.org)
192.168.1.200
Web Server : Apache
1
2
3
XXIV
Simulation : Phase 1
Web Server : NGINX Attack Server
Web Server (web.bdnog.org)
192.168.1.150
• Step 1: Launch attack on web.bdnog.org
• Step 2: Check web server performance
Reverse Proxy (nx.bdnog.org)
192.168.1.100
Attack Server (attack.bdnog.org)
192.168.1.200
Web Server : Apache
1
2
3
XXIV
Simulation : Phase 2
Web Server : NGINX Attack Server
Web Server (web.bdnog.org)
192.168.1.150
• Step 1: Change DNS entry for web.bdnog.org to
192.168.1.100 (nx.bdnog.org)
• Step 2: nx.bdnog.org works as reverse proxy
• Step 3: Check web server performance
Reverse Proxy (nx.bdnog.org)
192.168.1.100
Attack Server (attack.bdnog.org)
192.168.1.200
Web Server : Apache
1
2
3
XXIV
Simulation : Available Tools
DDOSIM

Layer 7 DDoS Simulator
http://sourceforge.net/projects/
ddosim/
BONESI

The DDoS Botnet Simulator
https://code.google.com/p/bonesi/
Slowhttptest

L7 DoS simulator

http://code.google.com/p/
slowhttptest/
XXIV
Tools Used : ddosim
• ddosim is a tool that can be used in a laboratory
environment to simulate a distributed denial of service
(DDOS) attack against a target server
• ddosim simulates several zombie hosts (having random IP
addresses) which create full TCP connections to the
target server.
• After completing the connection, ddosim starts the
conversation with the listening application (e.g. HTTP
server).
XXIV
Tools Used : ddosim
• ddosim is written in C++ and runs on Linux. Its current
functionalities include:
• HTTP DDoS with valid requests
• HTTP DDoS with invalid requests (similar to a DC++
attack)
• SMTP DDoS
• TCP connection flood on random port
XXIV
Tools Used : ddosim
• Running DDOSIM out of lab is not really possible because
it simulates distributed (multiple source IPs) attacks using
a connection-oriented protocol (TCP) which needs at least
the 3way handshake before sending any useful data.
• So the communication must be bidirectional. The packets
(TCP SYN-ACK) sent by the server must reach the
attacker (having random IP address)
XXIV
Tools Used : ddosim
1. Establish 10 TCP connections from random IP
addresses to www server and send invalid HTTP requests
./ddosim -d 192.168.1.2 -p 80 -c 10
-r HTTP_INVALID -i eth0!
2. Establish infinite connections at higher speed to www
server and make HTTP valid requests:
./ddosim -d 192.168.1.2 -p 80 -c 0
-w 0 -t 10 -r HTTP_VALID -i eth0
XXIV
Simulation
FINDINGS
XXIV
Findings
• Proper monitoring
• Log analysis (logstalgia)
• Off-loading & Splitting Traffic / DDoS Mitigation in broader
scale
XXIV
Monitoring
• Monitoring NGINX/Apache with Observium
Request Statistics
Server Status
XXIV
Log Analysis (logstalgia)
XXIV
Off-loading & Splitting Traffic
NGINX
ISP-1
NGINX
ISP-2
NGINX
ISP-3
NGINX
ISP-4
NGINX
ISP-5
DDoS Front End
Main Server Node
Main Server Node
Users Traffic, DNS Round
Robin
XXIV
Issues
• Scalability
• Performance Optimization
• Integrate DDoS mitigation solution with routing
infrastructure
• Integrate ExaBGP / BGP FlowSpec
XXIV
Scripts (Finding the BOT)
# more /var/log/apache2/access.log | grep ”bdnog.org"
| grep "GET / HTTP”!
!
203.188.170.218 - - [26/Oct/2013:17:58:25 +0600] "GET / HTTP/1.1" 200
537 "http://www.24livenewspaper.com/site/index.php?url=www.bdnog.org"
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"!
94.109.96.192 - - [26/Oct/2013:17:58:26 +0600] "GET / HTTP/1.1" 200
537 "http://www.bdnog.org/" "Mozilla/5.0 (Linux; U; Android 2.3.4;
en-gb; SonyEricssonWT19iv Build/4.0.2.A.0.58) AppleWebKit/533.1
(KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"!
203.188.170.218 - - [26/Oct/2013:17:58:26 +0600] "GET / HTTP/1.1" 200
537 "http://www.24livenewspaper.com/site/index.php?url=www.bdnog.org"
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)”"!
XXIV
Scripts (Finding the BOT)
# more /var/log/apache2/access.log | grep ”bdnog
.org" | grep "GET / HTTP" | cut -d " " -f1!
!
94.109.96.192!
203.188.170.218!
94.109.96.192!
203.188.170.218!
94.109.96.192!
203.188.170.218!
XXIV
Scripts (Finding the BOT)
# more /var/log/apache2/access.log | grep
”bdnog.org" | grep "GET / HTTP" | cut -d " " -
f1 | sort | uniq -c| awk '{if($1>K){print $2}}’!
!
114.130.136.182!
117.18.229.59!
117.18.231.60!
175.140.219.213!
202.134.10.135
Replace K with value
XXIV
Scripts (Finding the BOT)
# ipset create blacklist hash:net!
# more /var/log/apache2/access.log | grep ”bdnog
.org" | grep "GET / HTTP" | cut -d " " -f1 |
sort | uniq -c| awk '{if($1>100){print $2}}' |
xargs -tl -I _ ipset -A blacklist _!
!
ipset -A blacklist 114.130.136.182!
ipset -A blacklist 117.18.229.59!
ipset -A blacklist 117.18.231.60!
ipset -A blacklist 175.140.219.213
XXIV
Scripts (Finding the BOT)
# ipset list!
Name: blacklist!
Type: hash:net!
Header: family inet hashsize 1024 maxelem 65536!
Size in memory: 16984!
References: 0!
Members:!
114.130.136.182!
203.188.170.218!
202.134.10.135!
37.160.132.237
XXIV
Scripts (Finding the BOT)
/sbin/iptables -X DDOS_HTTP_FILTER!
/sbin/iptables -N DDOS_HTTP_FILTER!
/sbin/iptables -A DDOS_HTTP_FILTER -p tcp
--syn --dport 80 -m set --match-set
blacklist src -j DROP
XXIV
Special Thanks
• GZ Kabir, BDCOM
• Sumon Ahmed Sabir, Fiber@Home
• Technical Team of Prothom Alo.Com
• Attackers
QUESTION

More Related Content

Viewers also liked

Bangladesh Cyber Incident Trends 2013 & bdCERT Update
Bangladesh Cyber Incident Trends 2013 & bdCERT UpdateBangladesh Cyber Incident Trends 2013 & bdCERT Update
Bangladesh Cyber Incident Trends 2013 & bdCERT UpdateFakrul Alam
 
BSides Algiers - Layer7 DoS Attacks - Oussama Elhamer
BSides Algiers - Layer7 DoS Attacks - Oussama ElhamerBSides Algiers - Layer7 DoS Attacks - Oussama Elhamer
BSides Algiers - Layer7 DoS Attacks - Oussama ElhamerShellmates
 
High Performance Security: Mitigating DDoS Attacks Without Losing Your Edge
High Performance Security: Mitigating DDoS Attacks Without Losing Your EdgeHigh Performance Security: Mitigating DDoS Attacks Without Losing Your Edge
High Performance Security: Mitigating DDoS Attacks Without Losing Your EdgeHostway|HOSTING
 
Defending Against Application DoS attacks
Defending Against Application DoS attacksDefending Against Application DoS attacks
Defending Against Application DoS attacksRoberto Suggi Liverani
 
Layer 7 ddos
Layer 7 ddosLayer 7 ddos
Layer 7 ddosfangjiafu
 
Community tools to fight against DDoS
Community tools to fight against DDoSCommunity tools to fight against DDoS
Community tools to fight against DDoSFakrul Alam
 
Global Cyber Security trend & impact of Internet on the society of Bangladesh...
Global Cyber Security trend & impact of Internet on the society of Bangladesh...Global Cyber Security trend & impact of Internet on the society of Bangladesh...
Global Cyber Security trend & impact of Internet on the society of Bangladesh...Fakrul Alam
 
Electornic evidence collection
Electornic evidence collectionElectornic evidence collection
Electornic evidence collectionFakrul Alam
 
NetScout nGeniusONE overview
NetScout nGeniusONE overviewNetScout nGeniusONE overview
NetScout nGeniusONE overviewBAKOTECH
 

Viewers also liked (9)

Bangladesh Cyber Incident Trends 2013 & bdCERT Update
Bangladesh Cyber Incident Trends 2013 & bdCERT UpdateBangladesh Cyber Incident Trends 2013 & bdCERT Update
Bangladesh Cyber Incident Trends 2013 & bdCERT Update
 
BSides Algiers - Layer7 DoS Attacks - Oussama Elhamer
BSides Algiers - Layer7 DoS Attacks - Oussama ElhamerBSides Algiers - Layer7 DoS Attacks - Oussama Elhamer
BSides Algiers - Layer7 DoS Attacks - Oussama Elhamer
 
High Performance Security: Mitigating DDoS Attacks Without Losing Your Edge
High Performance Security: Mitigating DDoS Attacks Without Losing Your EdgeHigh Performance Security: Mitigating DDoS Attacks Without Losing Your Edge
High Performance Security: Mitigating DDoS Attacks Without Losing Your Edge
 
Defending Against Application DoS attacks
Defending Against Application DoS attacksDefending Against Application DoS attacks
Defending Against Application DoS attacks
 
Layer 7 ddos
Layer 7 ddosLayer 7 ddos
Layer 7 ddos
 
Community tools to fight against DDoS
Community tools to fight against DDoSCommunity tools to fight against DDoS
Community tools to fight against DDoS
 
Global Cyber Security trend & impact of Internet on the society of Bangladesh...
Global Cyber Security trend & impact of Internet on the society of Bangladesh...Global Cyber Security trend & impact of Internet on the society of Bangladesh...
Global Cyber Security trend & impact of Internet on the society of Bangladesh...
 
Electornic evidence collection
Electornic evidence collectionElectornic evidence collection
Electornic evidence collection
 
NetScout nGeniusONE overview
NetScout nGeniusONE overviewNetScout nGeniusONE overview
NetScout nGeniusONE overview
 

More from Fakrul Alam

bdNOG Update in APRICOT 2015
bdNOG Update in APRICOT 2015bdNOG Update in APRICOT 2015
bdNOG Update in APRICOT 2015Fakrul Alam
 
RPKI Deployment Status in Bangladesh
RPKI Deployment Status in BangladeshRPKI Deployment Status in Bangladesh
RPKI Deployment Status in BangladeshFakrul Alam
 
Bangladesh Cyber Security Status in Global Perspective
Bangladesh Cyber Security Status in Global PerspectiveBangladesh Cyber Security Status in Global Perspective
Bangladesh Cyber Security Status in Global PerspectiveFakrul Alam
 
Bangladesh Cyber Incident Trends 2012 & bdCERT Update
Bangladesh Cyber Incident Trends 2012 & bdCERT UpdateBangladesh Cyber Incident Trends 2012 & bdCERT Update
Bangladesh Cyber Incident Trends 2012 & bdCERT UpdateFakrul Alam
 
RPKI (Resource Public Key Infrastructure)
RPKI (Resource Public Key Infrastructure)RPKI (Resource Public Key Infrastructure)
RPKI (Resource Public Key Infrastructure)Fakrul Alam
 
bdNOG Update (APRICOT 2014)
bdNOG Update (APRICOT 2014)bdNOG Update (APRICOT 2014)
bdNOG Update (APRICOT 2014)Fakrul Alam
 
DDoS Attacks : Preparation Detection Mitigation
DDoS Attacks : Preparation Detection MitigationDDoS Attacks : Preparation Detection Mitigation
DDoS Attacks : Preparation Detection MitigationFakrul Alam
 
IPv6 deployment status in Bangladesh
IPv6 deployment status in BangladeshIPv6 deployment status in Bangladesh
IPv6 deployment status in BangladeshFakrul Alam
 

More from Fakrul Alam (8)

bdNOG Update in APRICOT 2015
bdNOG Update in APRICOT 2015bdNOG Update in APRICOT 2015
bdNOG Update in APRICOT 2015
 
RPKI Deployment Status in Bangladesh
RPKI Deployment Status in BangladeshRPKI Deployment Status in Bangladesh
RPKI Deployment Status in Bangladesh
 
Bangladesh Cyber Security Status in Global Perspective
Bangladesh Cyber Security Status in Global PerspectiveBangladesh Cyber Security Status in Global Perspective
Bangladesh Cyber Security Status in Global Perspective
 
Bangladesh Cyber Incident Trends 2012 & bdCERT Update
Bangladesh Cyber Incident Trends 2012 & bdCERT UpdateBangladesh Cyber Incident Trends 2012 & bdCERT Update
Bangladesh Cyber Incident Trends 2012 & bdCERT Update
 
RPKI (Resource Public Key Infrastructure)
RPKI (Resource Public Key Infrastructure)RPKI (Resource Public Key Infrastructure)
RPKI (Resource Public Key Infrastructure)
 
bdNOG Update (APRICOT 2014)
bdNOG Update (APRICOT 2014)bdNOG Update (APRICOT 2014)
bdNOG Update (APRICOT 2014)
 
DDoS Attacks : Preparation Detection Mitigation
DDoS Attacks : Preparation Detection MitigationDDoS Attacks : Preparation Detection Mitigation
DDoS Attacks : Preparation Detection Mitigation
 
IPv6 deployment status in Bangladesh
IPv6 deployment status in BangladeshIPv6 deployment status in Bangladesh
IPv6 deployment status in Bangladesh
 

Recently uploaded

Cybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best PracticesCybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best PracticesLumiverse Solutions Pvt Ltd
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxmibuzondetrabajo
 
SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predieusebiomeyer
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxMario
 
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书rnrncn29
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxNIMMANAGANTI RAMAKRISHNA
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书rnrncn29
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxAndrieCagasanAkio
 

Recently uploaded (9)

Cybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best PracticesCybersecurity Threats and Cybersecurity Best Practices
Cybersecurity Threats and Cybersecurity Best Practices
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptx
 
SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predi
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptx
 
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptx
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptx
 

Application Layer DDoS-A Practical Approach & Mitigation Techniques

  • 1. Application Layer DDoS A Practical Approach & Mitigation Techniques
  • 3. Tools used to demonstrate DDoS attack is for educational / knowledge sharing purpose only. No intention to generate DDoS attack on production network. Disclaimer
  • 4. XXIV Agenda • Background • Application / Layer 7 DDoS • Practical Approach (Case Study) • Mitigation • Simulation • Key findings & Issues
  • 6. XXIV Background : What is DDoS • Denial of Service (DoS) / Distributed Denial of Service (DDoS) is the act of performing an attack which prevents the system from providing services to legitimate users • Denial of Service attacks take many forms, and utilize many attack vectors • When successful, the targeted host may stop providing any service, provide limited services only or provide services to some users only • DDoS attack sometime refer as Distributed Reflection Denial of Service (DrDoS) Attack
  • 7. XXIV Background : DDoS Attack Phases • Phase One: Target Acquisition • Phase Two: Groundwork • Phase Three: ATTACK
  • 8. XXIV Background : About Botnets A Botnet can generate 1 Million Times the available bandwidth of a business It takes just 64,000 PCs infected with a virus like Conficker to generate 10 gigabits Per second of traffic Mariposa, the largest known Botnet, affected 12 million PCs It could have generate a DDoS attack as large as 31.2 Terabytes Per second
  • 9. XXIV Background : DDoS Insurance • Insurance is money you pay to be protected from • Happen / Might Not Happen • You can be prepared • Incident response plan • Tools • Gear • Partnerships • It may not be sufficient – you should have picked the higher premium policy…
  • 10. XXIV Background : Types of Attacks • Volume Based Attacks • Protocol Attacks • Application Layer Attacks
  • 11. XXIV Background : Statistics Source: PROLEXIC Q3 2013 Attack Report
  • 12. XXIV Background : DDoS Attack Surface • Past DDOS attacks were mainly Layer 3 / Layer 4 attacks Layer 7 Application Layer 6 Presentation Layer 5 Session Layer 4 Transport Layer 3 Network Layer 2 Data Link Layer 1 Physical Application Telnet / FTP / NFS / WWW Session RPC Transport Sockets / Streams Network IP + ARP / RARP / ICMP Physical Protocol Ethernet / FDD / PPP Transmission Media Coax / Fiber TCP UDP
  • 13. XXIV Layer 3 DDoS Attack • Layer 3 - muscle-based attacks • Flood of TCP/UDP/ICMP/IGMP packets, overloading infrastructure due to high rate processing/discarding of packets and filling up the packet queues, or saturating pipes • Introduce a packet workload most gear isn't designed for • Example - UDP flood to non-listening port
  • 14. XXIV Layer 4 DDoS Attack • Layer 4 – slightly more sophisticated • DoS attacks consuming extra memory, CPU cycles, and triggering responses • TCP SYN flood • TCP new connections flood • TCP concurrent connections exhaustion • TCP/UDP garbage data flood to listening services (ala LOIC) • Example – SYN flood
  • 15. XXIV Layer 7 DDoS • Layer 7 - The Evil • DoS attacks abusing application-server memory and performance limitations – masquerading as legitimate transactions • HTTP page flood • HTTP bandwidth consumption • DNS query flood • SIP INVITE flood • Low rate, high impact attacks – e.g. Slowloris, HTTP POST DoS
  • 16. XXIV Background : DDoS Attack Costs Damage to Your Brand Loss of Revenue Bad Customer Exeprience If your site is down, account holders will question if you provide safe service If your website is down, you loos revenue Call center agent get overwhelmed Ruins years of work building your brand No online banking, bill pay, forms or application, account opening Account holders frustration People seek alternatives
  • 17. XXIV Background : DDoS Additional Threat • DDoS attacks are more frequently being used to hide security breaches and data theft • Attention focuses on attack • Log files get massive, to difficult to analyze quickly • Servers and routers are rebooted, often destroying forensic evidence • Attacks end long before any intrusion is identified
  • 18. XXIV • SPAMHAUS : MARCH 27 2013 Background : DDoS Attack Volume
  • 19. XXIV Background : DDoS Attack Volume (DNS) 1 Attacker Laptop controlling + 5-7 compromised server on + 3 networks that allowed spoofing of + 9 Gbps DNS request to + 0.1% of open resolvers resulted in = 300 Gbps+ DDoS attack traffic [source: cloudflare] dig ANY isc.org @OpenResolverIP +edns=0 +notcp +bufsize=4096
  • 20. XXIV fkfkfkfa.com. 84930 IN A 204.46.43.16 fkfkfkfa.com. 84930 IN A 204.46.43.17 fkfkfkfa.com. 84930 IN A 204.46.43.18 fkfkfkfa.com. 84930 IN A 204.46.43.19 fkfkfkfa.com. 84930 IN A 204.46.43.20 fkfkfkfa.com. 84930 IN A 204.46.43.21 fkfkfkfa.com. 84930 IN A 204.46.43.22 fkfkfkfa.com. 84930 IN A 204.46.43.23 fkfkfkfa.com. 84930 IN A 204.46.43.24 fkfkfkfa.com. 84930 IN A 204.46.43.25 fkfkfkfa.com. 84930 IN A 204.46.43.26 fkfkfkfa.com. 84930 IN A 204.46.43.27 fkfkfkfa.com. 84930 IN A 204.46.43.28 fkfkfkfa.com. 84930 IN A 204.46.43.29 fkfkfkfa.com. 84930 IN A 204.46.43.30 fkfkfkfa.com. 84930 IN A 204.46.43.31 fkfkfkfa.com. 84930 IN A 204.46.43.32 fkfkfkfa.com. 84930 IN A 204.46.43.33 fkfkfkfa.com. 84930 IN A 204.46.43.34 fkfkfkfa.com. 84930 IN A 204.46.43.35 fkfkfkfa.com. 84930 IN A 204.46.43.36 fkfkfkfa.com. 84930 IN A 204.46.43.37 fkfkfkfa.com. 84930 IN A 204.46.43.38 fkfkfkfa.com. 84930 IN A 204.46.43.39 fkfkfkfa.com. 84930 IN A 204.46.43.40 fkfkfkfa.com. 84930 IN A 204.46.43.41 fkfkfkfa.com. 84930 IN A 204.46.43.42 fkfkfkfa.com. 84930 IN A 204.46.43.43 fkfkfkfa.com. 84930 IN A 204.46.43.44 fkfkfkfa.com. 84930 IN A 204.46.43.45 fkfkfkfa.com. 84930 IN NS us3.fkfkfkfa.com. fkfkfkfa.com. 84930 IN A 204.46.43.46 fkfkfkfa.com. 84930 IN NS us4.fkfkfkfa.com. ! ;; AUTHORITY SECTION: fkfkfkfa.com. 84930 IN NS us3.fkfkfkfa.com. fkfkfkfa.com. 84930 IN NS us4.fkfkfkfa.com. ! ;; Query time: 83 msec ;; SERVER: 103.12.178.xxx#53(103.12.178.xxx) ;; WHEN: Sat Dec 28 17:46:24 2013 ;; MSG SIZE rcvd: 4002 Background : DDoS Attack Volume (DNS) • dig ANY fkfkfkfa.com @103.12.178.XXX +edns=0 +notcp +bufsize=4096 4002/64 = 62x amplification
  • 21. XXIV fn-vm1.ndc.nasa.gov 123 198.123.30.132 6 3 4 190 6 27 m000044.ppp.asahi-net. 123 198.123.30.132 1 3 4 190 27 27 106.66.131.129 45826 198.123.30.132 1 3 3 190 27 27 4.26.19.130 289 198.123.30.132 1 3 1 190 27 27 host-24-100-155-153.ne 32769 198.123.30.132 65 3 3 190 5 28 78-62-146-36.static.ze 33494 198.123.30.132 1 3 3 190 28 28 77.68.208.137 32776 198.123.30.132 1 3 4 190 28 28 75-139-155-2.dhcp.gwnt 123 198.123.30.132 1 3 4 190 28 28 d58-106-139-87.rdl802. 123 198.123.30.132 1 3 4 190 28 28 c-71-202-159-26.hsd1.c 39741 198.123.30.132 1 3 4 190 28 28 f6-bamboo-r4.ndc.nasa. 123 198.123.30.132 8 3 4 190 5 28 77.68.201.102 32823 198.123.30.132 1 3 4 190 29 29 117.195.46.81 20609 198.123.30.132 12 3 3 190 29 29 87-194-180-251.bethere 123 198.123.30.132 4 3 4 190 7 29 d54C0EAF1.access.telen 123 198.123.30.132 1 3 4 190 29 29 77.68.132.75 1060 198.123.30.132 1 3 4 190 29 29 187-162-58-152.static. 123 198.123.30.132 1 3 4 190 29 29 ynrfw23001.yr.com 480 198.123.30.132 13 3 3 590 4 29 115-39-41-60.mie1.comm 4913 198.123.30.132 1 3 3 190 29 29 primary.idb.com.au 199 198.123.30.132 1 3 4 190 29 29 206-248-138-67.dsl.tek 123 198.123.30.132 1 3 4 190 29 29 pool-173-66-167-144.wa 60137 198.123.30.132 2 3 1 590 14 29 dslsonicwall.btconline 23891 198.123.30.132 1 3 1 190 30 30 108-192-128-52.lightsp 123 198.123.30.132 1 3 4 190 30 30 static-50-44-202-198.o 11547 198.123.30.132 1 3 1 190 30 30 117.136.19.162 21829 198.123.30.132 1 3 3 190 30 30 cm68.eta123.maxonline. 123 198.123.30.132 1 3 4 190 30 30 89.169.64.236 123 198.123.30.132 6 3 4 190 6 30 121.61-66-87.adsl-dyn. 123 198.123.30.132 1 3 4 190 30 30 pool-98-118-84-72.bstn 123 198.123.30.132 1 3 4 190 30 30 218.56.20.142 1100 198.123.30.132 1 3 3 190 30 30 66.162.156.150 41755 198.123.30.132 4 3 4 190 10 30 173-11-46-77-Minnesota 4847 198.123.30.132 2 3 4 190 17 30 61-195-152-9.cust.bit- 123 198.123.30.132 1 3 4 190 31 31 rionicapital1.pndsl.co 50071 198.123.30.132 1 3 3 190 31 31 vespa.ndc.nasa.gov 123 198.123.30.132 1 3 4 190 31 31 host87-93-dynamic.183- 32850 198.123.30.132 1 3 3 190 31 31 odin000956850.ndc.nasa 123 198.123.30.132 1 3 4 190 31 31 241.wolainfo.com.pl 123 198.123.30.132 1 3 3 190 31 31 112.90.239.142 43793 198.123.30.132 1 3 3 190 31 31 cpe-071-070-131-156.nc 123 198.123.30.132 1 3 4 190 31 31 88.230.191.53.dynamic. 23460 198.123.30.132 1 3 3 190 31 31 ARouen-652-1-378-137.w 28247 198.123.30.132 2 3 3 190 16 31 77.68.194.63 32825 198.123.30.132 1 3 4 190 32 32 77.68.154.162 1042 198.123.30.132 1 3 4 190 32 32 Background : DDoS Attack Volume (NTP) • ntpdc -c monlist ntp.nasa.gov
  • 22. XXIV Background : DDoS Attack Volume (NTP)
  • 23. XXIV Background : DDoS Attack Volume • Source Address Spoofing • Ingress Filtering / BCP 38 (http://tools.ietf.org/html/bcp38) • Secure DNS • http://www.cymru.com/Documents/secure-bind- template.html • Secure NTP • http://www.team-cymru.org/ReadingRoom/Templates/ secure-ntp-template.html
  • 24. XXIV Background : DDoS or Network Stress Test!
  • 26. XXIV Layer 7 DDoS : Overview • Application layer DoS attacks are evolving as part of the evolution of application attacks • The denied service is the application itself (rather than the host) – effectively preventing usage of the system. • Take advantage of flaws in the code to perform the DoS • The benefit for the attacker – does not require the same effort to achieve as a DDoS attack
  • 27. XXIV Layer 7 DDoS : Overview DoS can be achieved in various ways: • Application Crashing • Data Destruction • Resource Depletion
  • 28. XXIV Layer 7 DDoS : Application Crashing • Common way of performing a Denial of Service attack • In many cases, certain types of inputs may yield an error in the application which it did not anticipate, and will cause it to crash: • Buffer Overflows • Malformed data – causing parser exception • Terminating with error • SQL Injection
  • 29. XXIV Layer 7 DDoS : Data Destruction • One way to cause a DoS attack is by tampering with the data instead of the service itself • If a site is vulnerable to SQL Injection, for instance, it may be possible to DELETE all data from all tables • Although the Web site will keep being ‘online’, it will actually be useless without the information from the Database
  • 30. XXIV Layer 7 DDoS : Resource Depletion • Resource Depletion is a technique of performing DoS attacks on any site or application • Classical Resource Depletion simply utilizes very large amounts of attacker resources which includes • Memory • CPU • Disk Space • Sophisticated attacks pinpoint the weak points of the application to achieve maximum effect using minimal resources
  • 31. XXIV Effectiveness of Layer 7 DDoS • Higher Obscurity • Higher Efficiency • Higher Lethality
  • 32. XXIV Layer 7 DDoS Web Attack • Causes related to your inefficient codes • Protocol Weakness • HTTP GET • HTTP POST
  • 35. XXIV Layer 7 DDoS Tools • Slowloris abuses handling of HTTP request headers ssslooowly… • Written by RSnake • Iteratively injects one custom header at a time and goes to sleep • Web server vainly awaits the line space that will never come • R-U-Dead-Yet? abuses HTTP web form fields • Iteratively injects one custom byte into a web application post field and goes to sleep • Application threads become zombies awaiting ends of posts till death lurks upon the website
  • 36. XXIV Low Orbit Ion Cannon (LOIC) • LOIC v.1.0.0.0 • LOIC v.1.1.1.16
  • 37. XXIV Low Orbit Ion Cannon (LOIC) - Android
  • 40. XXIV Case Study • Time: August 2012 • Country: Bangladesh • Site: www.prothom-alo.com • Ranked top 10 sites in Bangladesh (Source: Alexa)
  • 41. XXIV Initial Findings • Massive HTTP GET Flood • Site is not accessible • There is no major changes in bandwidth utilization • Proper monitoring not in place to identify the actual attack • Attack source is from Russia, China and some countries from Africa
  • 42. XXIV Initial Findings : Logs 186.58.179.33 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/ 4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" ! 189.76.197.117 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.19) Gecko/20081201 Firefox/ 2.0.0.19”! 186.58.179.33 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/ 4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618)”! 186.6.168.11 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/ 4.0 (compatible; MSIE 5.0; Windows 2000) Opera 6.03 [en]" 197.0.165.121 - - [21/Oct/ 2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.4.154.25 Safari/525.19”! 189.76.197.117 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.19) Gecko/20081201 Firefox/ 2.0.0.19”! 197.0.165.121 - - [21/Aug/2012:00:10:06 +0600] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/ 5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/ 0.4.154.25 Safari/525.19”
  • 43. XXIV Approach 1 • Solution from hosting company • Conventional host based firewall using IPTABLES. • Fine tune TCP parameters • Enable SYN Cookies • echo 1 > /proc/sys/net/ipv4/tcp_syncookies! • Enable socket reuse • echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle! • echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse! • Increase local port range • echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
  • 44. XXIV Issue with Approach 1 • Solution from hosting company required additional $$$$ which is significantly high • Hard to justify management • Host based firewall works only in Layer 3 & Layer 4 • Not capable to filter Layer 7 DDoS Attack
  • 45. XXIV Approach 2 • Split DNS • DNS configured to resolve host based on GEOIP. • External uses request redirected to external server hosted in USA • One new server co-located in Bangladesh • Internal (Bangladesh) traffic has been redirected to new server • Load has been distributed
  • 46. XXIV Issue with Approach 2 • Issue with Split DNS • 4.2.2.2, 8.8.8.8 and other Open DNS • Lots of users from Bangladesh is using open DNS like 4.2.2.2 & 8.8.8.8. • For those users DNS is still resolving USA data center server IP
  • 47. XXIV Approach 3 • Anycast • Failed • Most of the upstream provider and datacenter doesn’t allow anycast • It’s good in handling volumetric attack
  • 48. XXIV Approach 4 • Reverse Web Proxy • Use Reverse Proxy as frontend • Anti DDoS plugins along with other parameters • Minimize the attack vector • Distribute end user load and mitigation solution
  • 50. XXIV Solution Architecture Web Server Collocated in Bangladesh Web Server Collocated in USA Reverse Web Proxy (Front End) Spit DNS: Decentralize load based on GEOIP Reverse Web Proxy with DDoS plugins as front end
  • 51. XXIV Reverse Web Proxy • A reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned to the client as though they originated from the server itself (or servers themselves) –Wikipedia
  • 53. XXIV Why NGINX • Event Driven • Asynchronous • Single Threaded
  • 54. XXIV Nginx DDoS Plugins • Available plugins: • testcookie-nginx-module [http://kyprizel.github.io/ testcookie-nginx-module/] • Roboo : HTTP Robot Mitigator [http://www.ecl- labs.org/2011/03/17/roboo-http-mitigator.html]
  • 55. XXIV testcookie-nginx-module • testcookie-nginx-module is a simple robot mitigation module using cookie based challenge/response technique. • Challenge cookies can be set using different methods: • "Set-Cookie" + 307/302 HTTP Location redirect • "Set-Cookie" + HTML meta refresh redirect • If you need Captcha or Flash, check testcookie-flash- processor
  • 57. XXIV Roboo : HTTP Robot Mitigator • Uses advanced non-interactive HTTP challenge/response mechanisms to detect & mitigate HTTP Robots • Weeds out the larger percentage of HTTP robots which do not use real browsers or implement full browser stacks, resulting in the mitigation of various web threats: • HTTP Denial of Service tools - e.g. Low Orbit Ion Cannon • Vulnerability Scanning - e.g. Acunetix Web Vulnerability Scanner, Metasploit Pro, Nessus • Web exploits • Automatic comment posters/comment spam as a replacement of conventional CAPTCHA methods • Spiders, Crawlers and other robotic evil • Available at https://github.com/yuri-gushin/Roboo
  • 58. XXIV Roboo : HTTP Robot Mitigator • Will respond to each GET or POST request from an unverified source with a challenge: • Challenge can be Javascript or Flash based, optionally Gzip compressed • A real browser with full HTTP, HTML, Javascript and Flash player stacks will re-issue the original request after setting a special HTTP cookie that marks the host as “verified” • Marks verified sources using an HTTP Cookie • Integrates with Nginx web server and reverse proxy as an embedded Perl module
  • 62. XXIV Key Configuration Parameters Variables Description worker_processes This number should be, at maximum, the number of CPU cores on your system. worker_connections Determines how many clients will be served by each worker process. (Max clients = worker_connections * worker_processes) perl_modules /opt/ local/share/nginx;! perl_require Roboo.pm; Enabling Roboo Plugings map $http_user_agent Define http agent (httrack|WinHTTrack|htmlparser|libwww|Python)
  • 63. XXIV Key Configuration Parameters Variables Description $http_referer (babes|click|forsale|jewelry|nudit) limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=1r/s;! limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip: 10m;! creates zone “req_limit_per_ip” allocating 10MB for this session then limits queries for remote ip address to 1 request per second include /etc/nginx/ allow_only.conf Can define IP address for where site is only accessible
  • 64. XXIV Logs : Roboo challenged.log 202.4.100.35 - - [28/Nov/2013:14:05:10 +0600] "GET / HTTP/1.1" 200 669 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/ 537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36"! 202.4.100.35 - - [28/Nov/2013:14:05:11 +0600] "GET /Anti-Robot- GET-2babb27395588042480c.swf HTTP/1.1" 200 1025 "http:// www1.prothom-alo.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36”! verified.log 202.4.100.35 - - [28/Nov/2013:14:05:12 +0600] "GET / HTTP/1.1" 200 31942 "http://www1.prothom-alo.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 31.0.1650.57 Safari/537.36"
  • 65. XXIV Logs : testcookie-nginx-module 202.4.100.35 - - [30/Nov/2013:18:06:53 +0600] "GET /?ckattempt=1 HTTP/1.1" 200 31643 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0_4 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11B554a Safari/9537.53"
  • 67. XXIV Solution Architecture Web Server : NGINX Attack Server Web Server (web.bdnog.org) 192.168.1.150 • All the hardwares are configured in Virtual Box • DDoS launched in closed network • Please don’t try in production network Reverse Proxy (nx.bdnog.org) 192.168.1.100 Attack Server (attack.bdnog.org) 192.168.1.200 Web Server : Apache 1 2 3
  • 68. XXIV Simulation : Phase 1 Web Server : NGINX Attack Server Web Server (web.bdnog.org) 192.168.1.150 • Step 1: Launch attack on web.bdnog.org • Step 2: Check web server performance Reverse Proxy (nx.bdnog.org) 192.168.1.100 Attack Server (attack.bdnog.org) 192.168.1.200 Web Server : Apache 1 2 3
  • 69. XXIV Simulation : Phase 2 Web Server : NGINX Attack Server Web Server (web.bdnog.org) 192.168.1.150 • Step 1: Change DNS entry for web.bdnog.org to 192.168.1.100 (nx.bdnog.org) • Step 2: nx.bdnog.org works as reverse proxy • Step 3: Check web server performance Reverse Proxy (nx.bdnog.org) 192.168.1.100 Attack Server (attack.bdnog.org) 192.168.1.200 Web Server : Apache 1 2 3
  • 70. XXIV Simulation : Available Tools DDOSIM Layer 7 DDoS Simulator http://sourceforge.net/projects/ ddosim/ BONESI The DDoS Botnet Simulator https://code.google.com/p/bonesi/ Slowhttptest L7 DoS simulator http://code.google.com/p/ slowhttptest/
  • 71. XXIV Tools Used : ddosim • ddosim is a tool that can be used in a laboratory environment to simulate a distributed denial of service (DDOS) attack against a target server • ddosim simulates several zombie hosts (having random IP addresses) which create full TCP connections to the target server. • After completing the connection, ddosim starts the conversation with the listening application (e.g. HTTP server).
  • 72. XXIV Tools Used : ddosim • ddosim is written in C++ and runs on Linux. Its current functionalities include: • HTTP DDoS with valid requests • HTTP DDoS with invalid requests (similar to a DC++ attack) • SMTP DDoS • TCP connection flood on random port
  • 73. XXIV Tools Used : ddosim • Running DDOSIM out of lab is not really possible because it simulates distributed (multiple source IPs) attacks using a connection-oriented protocol (TCP) which needs at least the 3way handshake before sending any useful data. • So the communication must be bidirectional. The packets (TCP SYN-ACK) sent by the server must reach the attacker (having random IP address)
  • 74. XXIV Tools Used : ddosim 1. Establish 10 TCP connections from random IP addresses to www server and send invalid HTTP requests ./ddosim -d 192.168.1.2 -p 80 -c 10 -r HTTP_INVALID -i eth0! 2. Establish infinite connections at higher speed to www server and make HTTP valid requests: ./ddosim -d 192.168.1.2 -p 80 -c 0 -w 0 -t 10 -r HTTP_VALID -i eth0
  • 77. XXIV Findings • Proper monitoring • Log analysis (logstalgia) • Off-loading & Splitting Traffic / DDoS Mitigation in broader scale
  • 78. XXIV Monitoring • Monitoring NGINX/Apache with Observium Request Statistics Server Status
  • 80. XXIV Off-loading & Splitting Traffic NGINX ISP-1 NGINX ISP-2 NGINX ISP-3 NGINX ISP-4 NGINX ISP-5 DDoS Front End Main Server Node Main Server Node Users Traffic, DNS Round Robin
  • 81. XXIV Issues • Scalability • Performance Optimization • Integrate DDoS mitigation solution with routing infrastructure • Integrate ExaBGP / BGP FlowSpec
  • 82. XXIV Scripts (Finding the BOT) # more /var/log/apache2/access.log | grep ”bdnog.org" | grep "GET / HTTP”! ! 203.188.170.218 - - [26/Oct/2013:17:58:25 +0600] "GET / HTTP/1.1" 200 537 "http://www.24livenewspaper.com/site/index.php?url=www.bdnog.org" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"! 94.109.96.192 - - [26/Oct/2013:17:58:26 +0600] "GET / HTTP/1.1" 200 537 "http://www.bdnog.org/" "Mozilla/5.0 (Linux; U; Android 2.3.4; en-gb; SonyEricssonWT19iv Build/4.0.2.A.0.58) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"! 203.188.170.218 - - [26/Oct/2013:17:58:26 +0600] "GET / HTTP/1.1" 200 537 "http://www.24livenewspaper.com/site/index.php?url=www.bdnog.org" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)”"!
  • 83. XXIV Scripts (Finding the BOT) # more /var/log/apache2/access.log | grep ”bdnog .org" | grep "GET / HTTP" | cut -d " " -f1! ! 94.109.96.192! 203.188.170.218! 94.109.96.192! 203.188.170.218! 94.109.96.192! 203.188.170.218!
  • 84. XXIV Scripts (Finding the BOT) # more /var/log/apache2/access.log | grep ”bdnog.org" | grep "GET / HTTP" | cut -d " " - f1 | sort | uniq -c| awk '{if($1>K){print $2}}’! ! 114.130.136.182! 117.18.229.59! 117.18.231.60! 175.140.219.213! 202.134.10.135 Replace K with value
  • 85. XXIV Scripts (Finding the BOT) # ipset create blacklist hash:net! # more /var/log/apache2/access.log | grep ”bdnog .org" | grep "GET / HTTP" | cut -d " " -f1 | sort | uniq -c| awk '{if($1>100){print $2}}' | xargs -tl -I _ ipset -A blacklist _! ! ipset -A blacklist 114.130.136.182! ipset -A blacklist 117.18.229.59! ipset -A blacklist 117.18.231.60! ipset -A blacklist 175.140.219.213
  • 86. XXIV Scripts (Finding the BOT) # ipset list! Name: blacklist! Type: hash:net! Header: family inet hashsize 1024 maxelem 65536! Size in memory: 16984! References: 0! Members:! 114.130.136.182! 203.188.170.218! 202.134.10.135! 37.160.132.237
  • 87. XXIV Scripts (Finding the BOT) /sbin/iptables -X DDOS_HTTP_FILTER! /sbin/iptables -N DDOS_HTTP_FILTER! /sbin/iptables -A DDOS_HTTP_FILTER -p tcp --syn --dport 80 -m set --match-set blacklist src -j DROP
  • 88. XXIV Special Thanks • GZ Kabir, BDCOM • Sumon Ahmed Sabir, Fiber@Home • Technical Team of Prothom Alo.Com • Attackers