Cloud computing has changed the way businesses operate, the way businesses make money, and the way business have to protect their assets and information. More and more software applications are moving into the cloud. People are running their proxies in the cloud and soon you will be collecting your logs in the cloud. You shouldn't have to deal with log collection and log management. You should be able to focus your time on getting value out of the logs; to do log analysis and visualization.
In this presentation we will explore how we can leverage the cloud to build security visualization tools. We will discuss some common visualization libraries and have a look at how they can be deployed to solve security problems. We will see how easy it is to quickly stand up such an application. To close the presentation, we will look at a number of security visualization examples that show how security data benefits from visual representations. For example, how can network traffic, firewall data, or IDS data be visualized effectively?
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Cloud Log Analysis and Visualization
1. Cloud-based Log Analysis and Visualization
RMLL 2010, Bordeaux, France
mobile-166 My syslog
Raffael Marty - @zrlram
Tuesday, July 6, 2010
2. Raffael (Raffy) Marty
• Founder @
• Chief Security Strategist and Product Manager @ Splunk
• Manager Solutions @ ArcSight
• Intrusion Detection Research @ IBM Research
• IT Security Consultant @ PriceWaterhouse Coopers
Applied Security Visualization
Publisher: Addison Wesley (August, 2008)
ISBN: 0321510100
Logging as a Service 2 (c) by Raffael Marty
Tuesday, July 6, 2010
3. Agenda
•Introduction •Do it Yourself
•Visualization •AfterGlow
•Google Visualization API
•InfoViz Process
•Visualization Use-Cases
•Visualization Tools
•Visualization Resources
•The Cloud
•Loggly
Logging as a Service 3 (c) by Raffael Marty
Tuesday, July 6, 2010
4. Open Your Eyes
Logging as a Service 4 (c) by Raffael Marty
Tuesday, July 6, 2010
5. Security Is About Seeing
Logging as a Service 5 (c) by Raffael Marty
Tuesday, July 6, 2010
6. Goals
- Learn how you can
- use visualization to help solve security problems
- leverage the cloud to build security visualization tools
Logging as a Service 6 (c) by Raffael Marty
Tuesday, July 6, 2010
7. Information Visualization?
A picture is worth a thousand log records.
Inspire
Explore and
Discover
Answer a Pose a New Increase Communicate Support
Question Question Efficiency Information Decisions
Logging as a Service 7 (c) by Raffael Marty
Tuesday, July 6, 2010
9. InfoViz Process
Collect Process Visualize
•large-scale data collection •Your parsers •Visualization Tools
•and processing •Standard formats •and Libraries
Logging as a Service 9 (c) by Raffael Marty
Tuesday, July 6, 2010
11. Log Management
• Log Collection and Centralization
• Log Storage
• Log Filtering
• Log Aggregation
• Log Search and Extraction
• Log Retention and Archiving
Logging as a Service 11 (c) by Raffael Marty
Tuesday, July 6, 2010
13. Standard Formats
• Multiple formats
Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S
1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF)
Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src
internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group
"internet_access_in"
Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc:
81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00
TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556
• Log Standards
‣ CEE (cee.mitre.org) ‣ SDEE ‣ WELF
‣ IDMEF ‣ CBE ‣ XDAS
Logging as a Service 13 (c) by Raffael Marty
Tuesday, July 6, 2010
14. Normalization
• Parsers
“To analyze or separate (input, for example) into more easily
processed components.” (answers.com)
• Generate a common output format for vis-tools
(e.g., CSV)
• For example
‣ Regex /(d{1,3}.d{1,3}.d{1,3}.d{1,3})/g
‣ http://secviz.org/content/parser-exchange
Logging as a Service 14 (c) by Raffael Marty
Tuesday, July 6, 2010
16. Choose Your Poison
Logging as a Service 16 (c) by Raffael Marty
Tuesday, July 6, 2010
17. Reporting vs. Visualization
• Reporting Libraries • Visualization Libraries
- HighCharts - TheJIT
- Flot - Graphael
- Google Chart API - Protovis
- Open Flash Chart - ProcessingJS
- Flare
JavaScript vs. Flash vs. XYZ
Logging as a Service 17 (c) by Raffael Marty
Tuesday, July 6, 2010
18. HighCharts
• Click-Through
• On load
- near real-time updates • AJAX data input via JSON
• Zoom
http://www.highcharts.com/
Logging as a Service 18 (c) by Raffael Marty
Tuesday, July 6, 2010
19. Google Visualization API
http://code.google.com/apis/visualization/interactive_charts.html
• JavaScript
• Based on DataTables()
• Many graphs
• Playground
- http://code.google.com/apis/ajax/playground
Logging as a Service 19 (c) by Raffael Marty
Tuesday, July 6, 2010
20. ProtoVis
• JavaScript based visualization library
• Charting
• Treemaps
• BoxPlots
• Parallel Coordinates
• etc.
http://vis.stanford.edu/protovis/
Logging as a Service 20 (c) by Raffael Marty
Tuesday, July 6, 2010
21. TheJIT http://thejit.org/
• JavaScript InfoVis Toolkit
• Interactive
• Link Graphs
Logging as a Service 21 (c) by Raffael Marty
Tuesday, July 6, 2010
22. Processing
•Visualization library
•Java based
•Interactive (event handling)
•Number of libraries to
-draw in OpenGL
-read XML files
-write PDF files
•Processing JS
-JavaScript
-HTML 5 Canvas http://processingjs.org/
-Web IDE http://processing.org/
Logging as a Service 22 (c) by Raffael Marty
Tuesday, July 6, 2010
26. The (public) Cloud
What it is Types
• multi-tenancy • SaaS - Software
• elastic • PaaS - Platform
• “infinite” resources • IaaS - Infrastructure
• pay as you go Benefits
• self provisioning • No installation
• No elaborate configurations
It’s not
• No maintenance
• private data center
• Great scalability
• virtualization
• 7x24 availability
Logging as a Service 26 (c) by Raffael Marty
Tuesday, July 6, 2010
27. LaaS - Logging as a Service
• All your data in one place
• Loggly manages your data (index, store, archive, etc.)
• Extremely fast search across all your data
• Data source agnostic (no parsers)
• Data management
• access control
• data segregation
• data overview and summaries
• API access
Logging as a Service 27 (c) by Raffael Marty
Tuesday, July 6, 2010
28. Loggly Architecture
Loggly
Data Sources Clients user interface
mobile-166 My syslog
Data collection
API Data access
Proxies
Distributed
Indexers and Search Machines indexing and
processing
Distributed
data store
Logging as a Service 28 (c) by Raffael Marty
Tuesday, July 6, 2010
29. Loggly APIs
• URL format: http://wiki.loggly.com/api-documentation
http://<subdomain>.loggly.com/api/<resource>
• RESTful API HTTP Based
- Access through: /api/<resource> •GET - read
- JSON, XML, JSONP output •POST - create
• Authentication
•PUT - update
- Basic auth
•DELETE - delete
- oAuth
http://loggly.loggly.com/api/search/?q=error syslog to:
User: guest / Password: loggly logs.loggly.com:514
Logging as a Service 29 (c) by Raffael Marty
Tuesday, July 6, 2010
30. Search
http://[domain].loggly.com/api/search?q=404
{
"data": [
{
"indexed": "2010-07-03T17:17:38.909Z",
"ip": "75.101.249.172",
"text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au]
[|domain] (DF)",
"inputname": "logglyweb",
"timestamp": "2010-07-03 10:17:38"
},
{
"indexed": "2010-07-03T17:17:37.879Z",
"ip": "75.101.249.172",
"text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au]
[|domain] (DF)",
"inputname": "logglyapp",
"timestamp": "2010-07-03 10:17:37"
},
...
Logging as a Service 30 (c) by Raffael Marty
Tuesday, July 6, 2010
31. Parser
Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF)
Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF)
Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF)
(.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+):
(d+.d+.d+.d+).?(d*) [<>]
Regex / Parser (d+.d+.d+.d+).?(d*): (.*)
Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF)
Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF)
(CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF)
Logging as a Service 31 (c) by Raffael Marty
Tuesday, July 6, 2010
32. Visualize
Parser AfterGlow Grapher
CSV file Graph file
digraph structs {
graph [label="AfterGlow 1.5.8", fontsize=8];
node [shape=ellipse, style=filled,
Configuration fontsize=10, width=1, height=1,
fixedsize=true];
edge [len=1.6];
color.source=“green” if ($fields[0] ne “d”)
"aaelenes" -> "Printing Resume" ;
cluster.target=regex_replace("(d+).")."/8" "abbe" -> "Information Encryption" ;
threshold.event=5 "aanna" -> "Patent Access" ;
size.target=$fields[1] "aatharuv" -> "Ping" ;
}
http://afterglow.sf.net
Logging as a Service 32 (c) by Raffael Marty
Tuesday, July 6, 2010
33. AfterGlow Cloud
Grapher Loggly
JSON
CSV
DOT
Graph
Logging as a Service 33 (c) by Raffael Marty
Tuesday, July 6, 2010
34. Google Vis
• JSON to Graphs
• DataTable
- used among all charts
• Interactivity through events
Logging as a Service 34 (c) by Raffael Marty
Tuesday, July 6, 2010
35. <script type="text/javascript">
Google Vis Code
google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']});
google.setOnLoadCallback(call);
var trends = new Array();
function call() {
l!
a
$.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100",
n
type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx',
io
success: function(data) {
trends = data.data
drawChart();
c t
n
}
u
});
f
}
t
function drawChart() {
o
var data = new google.visualization.DataTable();
n
data.addColumn('string', 'Search');
data.addColumn('datetime', 'Date');
is
data.addColumn('number', 'Count');
e
data.addRows(trends);
od
var chart = new google.visualization.MotionChart(document.getElementById('chart_div'));
c
chart.draw(data, {width: 600, height:300, state:state});
is
var view = new google.visualization.DataView(data);
h
view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}]));
T
var table = new google.visualization.Table(document.getElementById('test_dataview'));
table.draw(view, {sortColumn: 1});
var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline'));
time.draw(timedata, {displayAnnotations: true});
}
</script>
Logging as a Service 35 (c) by Raffael Marty
Tuesday, July 6, 2010
41. http://secviz.org
Share, discuss, challenge, and learn about security
visualization.
• List: secviz.org/mailinglist
• Twitter: @secviz
Logging as a Service 41 (c) by Raffael Marty
Tuesday, July 6, 2010
42. Applied Security Visualization
• Bridging the gap between security and visualization
• Hands-on, end to end examples
• Data processing and analysis
Chapters
• Visualization • Compliance
• Data Sources • Insider Threat
• From Data to Graphs • Visualization Tools
Addison Wesley (August, 2008)
• Perimeter Threat ISBN: 0321510100
Logging as a Service 42 (c) by Raffael Marty
Tuesday, July 6, 2010
43. Thank You!
raffael.marty@loggly.com
@zrlram
43
Tuesday, July 6, 2010