SlideShare a Scribd company logo
1 of 43
Download to read offline
Cloud-based Log Analysis and Visualization
                        RMLL 2010, Bordeaux, France
                                               mobile-166   My syslog




                          Raffael Marty - @zrlram
Tuesday, July 6, 2010
Raffael (Raffy) Marty
       • Founder @
       • Chief Security Strategist and Product Manager @ Splunk
       • Manager Solutions @ ArcSight
       • Intrusion Detection Research @ IBM Research
       • IT Security Consultant @ PriceWaterhouse Coopers
                           Applied Security Visualization
                               Publisher: Addison Wesley (August, 2008)
                                           ISBN: 0321510100




                        Logging as a Service                              2   (c) by Raffael Marty
Tuesday, July 6, 2010
Agenda
            •Introduction                            •Do it Yourself

            •Visualization                            •AfterGlow
                                                      •Google Visualization API
            •InfoViz Process
                                                     •Visualization Use-Cases
            •Visualization Tools
                                                     •Visualization Resources
            •The Cloud

            •Loggly

                        Logging as a Service     3                          (c) by Raffael Marty
Tuesday, July 6, 2010
Open Your Eyes




                        Logging as a Service   4         (c) by Raffael Marty
Tuesday, July 6, 2010
Security Is About Seeing




                        Logging as a Service   5   (c) by Raffael Marty
Tuesday, July 6, 2010
Goals
       - Learn how you can
          - use visualization to help solve security problems
          - leverage the cloud to build security visualization tools




                        Logging as a Service     6          (c) by Raffael Marty
Tuesday, July 6, 2010
Information Visualization?

                           A picture is worth a thousand log records.


                                                                                               Inspire
 Explore and
  Discover


                         Answer a         Pose a New    Increase    Communicate     Support
                         Question          Question    Efficiency    Information   Decisions

                        Logging as a Service                 7                            (c) by Raffael Marty
Tuesday, July 6, 2010
Visualization
                        and The Cloud
                                   8




Tuesday, July 6, 2010
InfoViz Process




        Collect                                Process             Visualize
        •large-scale data collection           •Your parsers       •Visualization Tools
        •and processing                        •Standard formats   •and Libraries


                        Logging as a Service         9                          (c) by Raffael Marty
Tuesday, July 6, 2010
Collect
                                  10




Tuesday, July 6, 2010
Log Management
         • Log Collection and Centralization
         • Log Storage
         • Log Filtering
         • Log Aggregation
         • Log Search and Extraction
         • Log Retention and Archiving
                        Logging as a Service   11      (c) by Raffael Marty
Tuesday, July 6, 2010
Process
                                  12




Tuesday, July 6, 2010
Standard Formats
          • Multiple formats
              Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S
              1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF)

              Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src
              internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group
              "internet_access_in"

              Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc:
              81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00
              TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556

          • Log Standards
                   ‣    CEE (cee.mitre.org)       ‣   SDEE                ‣   WELF
                   ‣    IDMEF                     ‣   CBE                 ‣   XDAS
                        Logging as a Service          13                             (c) by Raffael Marty
Tuesday, July 6, 2010
Normalization
          • Parsers
                        “To analyze or separate (input, for example) into more easily
                        processed components.” (answers.com)
          • Generate a common output format for vis-tools
            (e.g., CSV)
          • For example
                   ‣    Regex                   /(d{1,3}.d{1,3}.d{1,3}.d{1,3})/g
                   ‣    http://secviz.org/content/parser-exchange

                         Logging as a Service              14                     (c) by Raffael Marty
Tuesday, July 6, 2010
Visualize
                                15




Tuesday, July 6, 2010
Choose Your Poison




                        Logging as a Service   16     (c) by Raffael Marty
Tuesday, July 6, 2010
Reporting vs. Visualization
          • Reporting Libraries                     • Visualization Libraries
               - HighCharts                          - TheJIT
               - Flot                                - Graphael
               - Google Chart API                    - Protovis
               - Open Flash Chart                    - ProcessingJS
                                                     - Flare



                               JavaScript vs. Flash vs. XYZ
                        Logging as a Service   17                        (c) by Raffael Marty
Tuesday, July 6, 2010
HighCharts



    • Click-Through
    • On load
        - near real-time updates                   • AJAX data input via JSON
    • Zoom
                                                             http://www.highcharts.com/
                        Logging as a Service       18                       (c) by Raffael Marty
Tuesday, July 6, 2010
Google Visualization API


           http://code.google.com/apis/visualization/interactive_charts.html

           • JavaScript
           • Based on DataTables()
           • Many graphs
           • Playground
                -   http://code.google.com/apis/ajax/playground

                        Logging as a Service                 19           (c) by Raffael Marty
Tuesday, July 6, 2010
ProtoVis
          • JavaScript based visualization library
          • Charting
          • Treemaps
          • BoxPlots
          • Parallel Coordinates
          • etc.

                                                       http://vis.stanford.edu/protovis/
                        Logging as a Service      20                          (c) by Raffael Marty
Tuesday, July 6, 2010
TheJIT   http://thejit.org/

          • JavaScript InfoVis Toolkit
          • Interactive
          • Link Graphs




                        Logging as a Service     21            (c) by Raffael Marty
Tuesday, July 6, 2010
Processing
          •Visualization library
          •Java based
          •Interactive (event handling)
          •Number of libraries to
               -draw      in OpenGL
               -read      XML files
               -write     PDF files
          •Processing JS
           -JavaScript
           -HTML 5 Canvas                               http://processingjs.org/
           -Web IDE                                     http://processing.org/
                        Logging as a Service       22                              (c) by Raffael Marty
Tuesday, July 6, 2010
Building Your Own

                                    23




Tuesday, July 6, 2010
Build Your Own




                                                          AfterGlow
                Loggly                         Regexes
                                                          Google Vis

                        Logging as a Service         24            (c) by Raffael Marty
Tuesday, July 6, 2010
Data Collection in
                        the Cloud
                                    25




Tuesday, July 6, 2010
The (public) Cloud
         What it is                            Types
          • multi-tenancy                      • SaaS - Software

          • elastic                            • PaaS - Platform

          • “infinite” resources               • IaaS - Infrastructure

          • pay as you go                      Benefits
          • self provisioning                  • No installation
                                               • No elaborate configurations
         It’s not
                                               • No maintenance
          • private data center
                                               • Great scalability
          • virtualization
                                               • 7x24 availability
                        Logging as a Service               26                  (c) by Raffael Marty
Tuesday, July 6, 2010
LaaS - Logging as a Service
       • All your data in one place
          • Loggly manages your data (index, store, archive, etc.)
       • Extremely fast search across all your data
          • Data source agnostic (no parsers)
       • Data management
          • access control
          • data segregation
          • data overview and summaries
       • API access
                        Logging as a Service    27                   (c) by Raffael Marty
Tuesday, July 6, 2010
Loggly Architecture
                                                                                Loggly
        Data Sources                    Clients                              user interface
                                                                mobile-166            My syslog




                                                                                                  Data collection
                                          API                                                     Data access
         Proxies


                                                                                                  Distributed
                                       Indexers and Search Machines                               indexing and
                                                                                                  processing

                                                                                                  Distributed
                                                                                                  data store




                        Logging as a Service               28                                        (c) by Raffael Marty
Tuesday, July 6, 2010
Loggly APIs
       • URL format:                                     http://wiki.loggly.com/api-documentation

             http://<subdomain>.loggly.com/api/<resource>
       • RESTful API                                           HTTP Based
                - Access through: /api/<resource>              •GET - read
                - JSON, XML, JSONP output                      •POST - create
       • Authentication
                                                               •PUT - update
                - Basic auth
                                                               •DELETE - delete
                - oAuth

         http://loggly.loggly.com/api/search/?q=error                       syslog to:
                 User: guest / Password: loggly                       logs.loggly.com:514

                        Logging as a Service        29                             (c) by Raffael Marty
Tuesday, July 6, 2010
Search
               http://[domain].loggly.com/api/search?q=404
               {
                    "data": [
                        {
                             "indexed": "2010-07-03T17:17:38.909Z",
                             "ip": "75.101.249.172",
                             "text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au]
               [|domain] (DF)",
                             "inputname": "logglyweb",
                             "timestamp": "2010-07-03 10:17:38"
                        },
                        {
                             "indexed": "2010-07-03T17:17:37.879Z",
                             "ip": "75.101.249.172",
                             "text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au]
               [|domain] (DF)",
                             "inputname": "logglyapp",
                             "timestamp": "2010-07-03 10:17:37"
                        },

                         ...



                        Logging as a Service                             30                                               (c) by Raffael Marty
Tuesday, July 6, 2010
Parser
                              Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53:    34388 [1au][|domain] (DF)

   Raw                        Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53:   49962 [1au][|domain] (DF)

                              Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53:   14434 [1au][|domain] (DF)




                                            (.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+):
                                                          (d+.d+.d+.d+).?(d*) [<>]
   Regex / Parser                                          (d+.d+.d+.d+).?(d*): (.*)



                              Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF)
   Normalized                 Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF)
   (CSV)                      Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF)




                        Logging as a Service                                 31                                                      (c) by Raffael Marty
Tuesday, July 6, 2010
Visualize
                                  Parser              AfterGlow              Grapher

                                           CSV file               Graph file



                                                                    digraph structs {
                                                                      graph [label="AfterGlow 1.5.8", fontsize=8];
                                                                      node [shape=ellipse, style=filled,
                                Configuration                           fontsize=10, width=1, height=1,
                                                                        fixedsize=true];
                                                                      edge [len=1.6];
       color.source=“green” if ($fields[0] ne “d”)
                                                                        "aaelenes" -> "Printing Resume" ;
       cluster.target=regex_replace("(d+).")."/8"                  "abbe" -> "Information Encryption" ;
       threshold.event=5                                                "aanna" -> "Patent Access" ;
       size.target=$fields[1]                                           "aatharuv" -> "Ping" ;
                                                                    }




                                           http://afterglow.sf.net
                        Logging as a Service                 32                                      (c) by Raffael Marty
Tuesday, July 6, 2010
AfterGlow Cloud
                                               Grapher   Loggly


                                                         JSON


                                                          CSV


                                                         DOT


                                                         Graph

                        Logging as a Service    33        (c) by Raffael Marty
Tuesday, July 6, 2010
Google Vis
          • JSON to Graphs
          • DataTable
               - used among all charts

          • Interactivity through events




                        Logging as a Service       34       (c) by Raffael Marty
Tuesday, July 6, 2010
<script type="text/javascript">
                                           Google Vis Code
           google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']});
           google.setOnLoadCallback(call);
           var trends = new Array();
           function call() {

                                                                                                 l!
                                                                                                a
               $.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100",


                                                                                              n
                     type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx',



                                                                                            io
                     success: function(data) {
                         trends = data.data
                         drawChart();

                                                                                      c   t
                                                                                    n
                     }


                                                                          u
               });


                                                                         f
           }


                                                                       t
           function drawChart() {


                                                                      o
             var data = new google.visualization.DataTable();


                                                                    n
             data.addColumn('string', 'Search');
             data.addColumn('datetime',    'Date');


                                                            is
             data.addColumn('number', 'Count');


                                                          e
             data.addRows(trends);



                                                   od
                  var chart = new google.visualization.MotionChart(document.getElementById('chart_div'));


                                                 c
                  chart.draw(data, {width: 600, height:300, state:state});



                                        is
                  var view = new google.visualization.DataView(data);


                                      h
                  view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}]));

                                     T
                  var table = new google.visualization.Table(document.getElementById('test_dataview'));
                  table.draw(view, {sortColumn: 1});

                  var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline'));
                  time.draw(timedata, {displayAnnotations: true});
           }
     </script>

                        Logging as a Service                                35                                  (c) by Raffael Marty
Tuesday, July 6, 2010
Visualization Use-Cases

                                      36




Tuesday, July 6, 2010
NetFlow Visualization
          • Treemap
          • Protovis.JS
          • Size: Amount
          • Brightness: Variance
          • Color: Sensor
          • Shows: Scans -
            bright spots


          • Thanks to Chris Horsley

                        Logging as a Service   37     (c) by Raffael Marty
Tuesday, July 6, 2010
Firewall Treemap




                        Logging as a Service   38        (c) by Raffael Marty
Tuesday, July 6, 2010
Firewall Log
                              Port                Source IP   Destination IP




                        Logging as a Service            39                     (c) by Raffael Marty
Tuesday, July 6, 2010
Visualization Resources


                                      40




Tuesday, July 6, 2010
http://secviz.org
                          Share, discuss, challenge, and learn about security
                                             visualization.
           • List: secviz.org/mailinglist
           • Twitter: @secviz




                        Logging as a Service       41                       (c) by Raffael Marty
Tuesday, July 6, 2010
Applied Security Visualization
        • Bridging the gap between security and visualization
        • Hands-on, end to end examples
        • Data processing and analysis


        Chapters
        • Visualization                        • Compliance
        • Data Sources                         • Insider Threat
        • From Data to Graphs                  • Visualization Tools
                                                                       Addison Wesley (August, 2008)
        • Perimeter Threat                                                        ISBN: 0321510100


                        Logging as a Service               42                           (c) by Raffael Marty
Tuesday, July 6, 2010
Thank You!




                        raffael.marty@loggly.com
                                 @zrlram


                                                   43
Tuesday, July 6, 2010

More Related Content

What's hot

Fast and Scalable Python
Fast and Scalable PythonFast and Scalable Python
Fast and Scalable PythonTravis Oliphant
 
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...DataWorks Summit
 
Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)Nicolas Poggi
 
Drag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFiDrag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFi"Constantin \"Cristi\"" Stanca
 
The convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on HadoopThe convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on HadoopDataWorks Summit
 
Lessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloudLessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloudDataWorks Summit
 
Db tech show - hivemall
Db tech show - hivemallDb tech show - hivemall
Db tech show - hivemallMakoto Yui
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemShivaji Dutta
 
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...DataWorks Summit
 
Teradata Partners Conference Oct 2014 Big Data Anti-Patterns
Teradata Partners Conference Oct 2014   Big Data Anti-PatternsTeradata Partners Conference Oct 2014   Big Data Anti-Patterns
Teradata Partners Conference Oct 2014 Big Data Anti-PatternsDouglas Moore
 
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...Databricks
 
Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践Hao Chen
 

What's hot (16)

Fast and Scalable Python
Fast and Scalable PythonFast and Scalable Python
Fast and Scalable Python
 
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
 
Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)
 
How do you decide where your customer was?
How do you decide where your customer was?How do you decide where your customer was?
How do you decide where your customer was?
 
Benchmarking
BenchmarkingBenchmarking
Benchmarking
 
Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale
 
London level39
London level39London level39
London level39
 
Drag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFiDrag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFi
 
The convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on HadoopThe convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on Hadoop
 
Lessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloudLessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloud
 
Db tech show - hivemall
Db tech show - hivemallDb tech show - hivemall
Db tech show - hivemall
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystem
 
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
 
Teradata Partners Conference Oct 2014 Big Data Anti-Patterns
Teradata Partners Conference Oct 2014   Big Data Anti-PatternsTeradata Partners Conference Oct 2014   Big Data Anti-Patterns
Teradata Partners Conference Oct 2014 Big Data Anti-Patterns
 
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
 
Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践
 

Viewers also liked

What Is Log Analyis
What Is Log AnalyisWhat Is Log Analyis
What Is Log AnalyisJim Jansen
 
Warehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis SystemWarehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis SystemJivan Nepali
 
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and SparkCrystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and SparkJivan Nepali
 
0610 w13 ms_61
0610 w13 ms_610610 w13 ms_61
0610 w13 ms_61King Ali
 
A Basic Guide to Server Log Analysis
A Basic Guide to Server Log AnalysisA Basic Guide to Server Log Analysis
A Basic Guide to Server Log AnalysisAndrew Halliday
 
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...DevOpsDays Tel Aviv
 
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesAmit Goel
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and VisualizationSurasak Sanguanpong
 
Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013SATOSHI TAGOMORI
 
Security Insights at Scale
Security Insights at ScaleSecurity Insights at Scale
Security Insights at ScaleRaffael Marty
 
Modern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analiziModern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analiziErtugrul Akbas
 
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans VerileriLog Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans VerileriErtugrul Akbas
 
Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi Ertugrul Akbas
 
LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1Sarah Robinson
 
عربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافیعربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافیmaqsood hasni
 
Google Analytics and Webmaster tool
Google Analytics and Webmaster toolGoogle Analytics and Webmaster tool
Google Analytics and Webmaster toolRUBEN LICERA
 
New Technologies Close the Recruitment Gap
New Technologies Close the Recruitment GapNew Technologies Close the Recruitment Gap
New Technologies Close the Recruitment GapJohn Reites
 
Oracle on Oracle (OoO)
Oracle on Oracle (OoO)Oracle on Oracle (OoO)
Oracle on Oracle (OoO)Oracle
 

Viewers also liked (20)

What Is Log Analyis
What Is Log AnalyisWhat Is Log Analyis
What Is Log Analyis
 
Warehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis SystemWarehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis System
 
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and SparkCrystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
 
0610 w13 ms_61
0610 w13 ms_610610 w13 ms_61
0610 w13 ms_61
 
A Basic Guide to Server Log Analysis
A Basic Guide to Server Log AnalysisA Basic Guide to Server Log Analysis
A Basic Guide to Server Log Analysis
 
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
 
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source Technologies
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
 
Log Data Mining
Log Data MiningLog Data Mining
Log Data Mining
 
Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013
 
Security Insights at Scale
Security Insights at ScaleSecurity Insights at Scale
Security Insights at Scale
 
Modern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analiziModern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analizi
 
Log siem korelasyon
Log siem korelasyonLog siem korelasyon
Log siem korelasyon
 
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans VerileriLog Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
 
Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi
 
LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1
 
عربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافیعربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافی
 
Google Analytics and Webmaster tool
Google Analytics and Webmaster toolGoogle Analytics and Webmaster tool
Google Analytics and Webmaster tool
 
New Technologies Close the Recruitment Gap
New Technologies Close the Recruitment GapNew Technologies Close the Recruitment Gap
New Technologies Close the Recruitment Gap
 
Oracle on Oracle (OoO)
Oracle on Oracle (OoO)Oracle on Oracle (OoO)
Oracle on Oracle (OoO)
 

Similar to Cloud Log Analysis and Visualization

The architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWSThe architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWSTreasure Data, Inc.
 
Open source for customer analytics
Open source for customer analyticsOpen source for customer analytics
Open source for customer analyticsMatthias Funke
 
Graphite tattle
Graphite tattleGraphite tattle
Graphite tattleDraco2002
 
Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012 Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012 Stephen Katz
 
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)John Adams
 
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...Beniamino Murgante
 
Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby ConferenceJohn Woodell
 
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPSONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPSotb
 
RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?Pat Cappelaere
 
EOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introductionEOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introductionOpenAIRE
 
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...Patrick Chanezon
 
Tools & Measurements
Tools & MeasurementsTools & Measurements
Tools & MeasurementsRIPE NCC
 
RIPE NCC Measurements Tools
RIPE NCC Measurements ToolsRIPE NCC Measurements Tools
RIPE NCC Measurements ToolsRIPE NCC
 
IDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on CloudIDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on Cloudstratuslab
 
Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04GeoSolutions
 
RIPEstat Demo - s2e03
RIPEstat Demo - s2e03RIPEstat Demo - s2e03
RIPEstat Demo - s2e03RIPE NCC
 
Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?Tugdual Grall
 
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoWorld
 

Similar to Cloud Log Analysis and Visualization (20)

The architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWSThe architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWS
 
Open source for customer analytics
Open source for customer analyticsOpen source for customer analytics
Open source for customer analytics
 
App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
Graphite tattle
Graphite tattleGraphite tattle
Graphite tattle
 
Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012 Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012
 
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
 
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
 
Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby Conference
 
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPSONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
 
RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?
 
EOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introductionEOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introduction
 
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
 
Tools & Measurements
Tools & MeasurementsTools & Measurements
Tools & Measurements
 
RIPE NCC Measurements Tools
RIPE NCC Measurements ToolsRIPE NCC Measurements Tools
RIPE NCC Measurements Tools
 
IDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on CloudIDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on Cloud
 
Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04
 
RIPEstat Demo - s2e03
RIPEstat Demo - s2e03RIPEstat Demo - s2e03
RIPEstat Demo - s2e03
 
Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?
 
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
 
20100608sigmod
20100608sigmod20100608sigmod
20100608sigmod
 

More from Raffael Marty

Exploring the Defender's Advantage
Exploring the Defender's AdvantageExploring the Defender's Advantage
Exploring the Defender's AdvantageRaffael Marty
 
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...Raffael Marty
 
How To Drive Value with Security Data
How To Drive Value with Security DataHow To Drive Value with Security Data
How To Drive Value with Security DataRaffael Marty
 
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Raffael Marty
 
Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Raffael Marty
 
Understanding the "Intelligence" in AI
Understanding the "Intelligence" in AIUnderstanding the "Intelligence" in AI
Understanding the "Intelligence" in AIRaffael Marty
 
AI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousAI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousRaffael Marty
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousRaffael Marty
 
Delivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationDelivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationRaffael Marty
 
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedAI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedRaffael Marty
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big DataRaffael Marty
 
Big Data Visualization
Big Data VisualizationBig Data Visualization
Big Data VisualizationRaffael Marty
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?Raffael Marty
 
Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityRaffael Marty
 
Visualization for Security
Visualization for SecurityVisualization for Security
Visualization for SecurityRaffael Marty
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?Raffael Marty
 
DAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxDAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxRaffael Marty
 

More from Raffael Marty (20)

Exploring the Defender's Advantage
Exploring the Defender's AdvantageExploring the Defender's Advantage
Exploring the Defender's Advantage
 
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
 
How To Drive Value with Security Data
How To Drive Value with Security DataHow To Drive Value with Security Data
How To Drive Value with Security Data
 
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
 
Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?
 
Understanding the "Intelligence" in AI
Understanding the "Intelligence" in AIUnderstanding the "Intelligence" in AI
Understanding the "Intelligence" in AI
 
Security Chat 5.0
Security Chat 5.0Security Chat 5.0
Security Chat 5.0
 
AI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousAI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are Dangerous
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are Dangerous
 
Delivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationDelivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and Visualization
 
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedAI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big Data
 
Big Data Visualization
Big Data VisualizationBig Data Visualization
Big Data Visualization
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?
 
Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for Security
 
Visualization for Security
Visualization for SecurityVisualization for Security
Visualization for Security
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?
 
DAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxDAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization Linux
 

Recently uploaded

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 

Recently uploaded (20)

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 

Cloud Log Analysis and Visualization

  • 1. Cloud-based Log Analysis and Visualization RMLL 2010, Bordeaux, France mobile-166 My syslog Raffael Marty - @zrlram Tuesday, July 6, 2010
  • 2. Raffael (Raffy) Marty • Founder @ • Chief Security Strategist and Product Manager @ Splunk • Manager Solutions @ ArcSight • Intrusion Detection Research @ IBM Research • IT Security Consultant @ PriceWaterhouse Coopers Applied Security Visualization Publisher: Addison Wesley (August, 2008) ISBN: 0321510100 Logging as a Service 2 (c) by Raffael Marty Tuesday, July 6, 2010
  • 3. Agenda •Introduction •Do it Yourself •Visualization •AfterGlow •Google Visualization API •InfoViz Process •Visualization Use-Cases •Visualization Tools •Visualization Resources •The Cloud •Loggly Logging as a Service 3 (c) by Raffael Marty Tuesday, July 6, 2010
  • 4. Open Your Eyes Logging as a Service 4 (c) by Raffael Marty Tuesday, July 6, 2010
  • 5. Security Is About Seeing Logging as a Service 5 (c) by Raffael Marty Tuesday, July 6, 2010
  • 6. Goals - Learn how you can - use visualization to help solve security problems - leverage the cloud to build security visualization tools Logging as a Service 6 (c) by Raffael Marty Tuesday, July 6, 2010
  • 7. Information Visualization? A picture is worth a thousand log records. Inspire Explore and Discover Answer a Pose a New Increase Communicate Support Question Question Efficiency Information Decisions Logging as a Service 7 (c) by Raffael Marty Tuesday, July 6, 2010
  • 8. Visualization and The Cloud 8 Tuesday, July 6, 2010
  • 9. InfoViz Process Collect Process Visualize •large-scale data collection •Your parsers •Visualization Tools •and processing •Standard formats •and Libraries Logging as a Service 9 (c) by Raffael Marty Tuesday, July 6, 2010
  • 10. Collect 10 Tuesday, July 6, 2010
  • 11. Log Management • Log Collection and Centralization • Log Storage • Log Filtering • Log Aggregation • Log Search and Extraction • Log Retention and Archiving Logging as a Service 11 (c) by Raffael Marty Tuesday, July 6, 2010
  • 12. Process 12 Tuesday, July 6, 2010
  • 13. Standard Formats • Multiple formats Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S 1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group "internet_access_in" Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc: 81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00 TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556 • Log Standards ‣ CEE (cee.mitre.org) ‣ SDEE ‣ WELF ‣ IDMEF ‣ CBE ‣ XDAS Logging as a Service 13 (c) by Raffael Marty Tuesday, July 6, 2010
  • 14. Normalization • Parsers “To analyze or separate (input, for example) into more easily processed components.” (answers.com) • Generate a common output format for vis-tools (e.g., CSV) • For example ‣ Regex /(d{1,3}.d{1,3}.d{1,3}.d{1,3})/g ‣ http://secviz.org/content/parser-exchange Logging as a Service 14 (c) by Raffael Marty Tuesday, July 6, 2010
  • 15. Visualize 15 Tuesday, July 6, 2010
  • 16. Choose Your Poison Logging as a Service 16 (c) by Raffael Marty Tuesday, July 6, 2010
  • 17. Reporting vs. Visualization • Reporting Libraries • Visualization Libraries - HighCharts - TheJIT - Flot - Graphael - Google Chart API - Protovis - Open Flash Chart - ProcessingJS - Flare JavaScript vs. Flash vs. XYZ Logging as a Service 17 (c) by Raffael Marty Tuesday, July 6, 2010
  • 18. HighCharts • Click-Through • On load - near real-time updates • AJAX data input via JSON • Zoom http://www.highcharts.com/ Logging as a Service 18 (c) by Raffael Marty Tuesday, July 6, 2010
  • 19. Google Visualization API http://code.google.com/apis/visualization/interactive_charts.html • JavaScript • Based on DataTables() • Many graphs • Playground - http://code.google.com/apis/ajax/playground Logging as a Service 19 (c) by Raffael Marty Tuesday, July 6, 2010
  • 20. ProtoVis • JavaScript based visualization library • Charting • Treemaps • BoxPlots • Parallel Coordinates • etc. http://vis.stanford.edu/protovis/ Logging as a Service 20 (c) by Raffael Marty Tuesday, July 6, 2010
  • 21. TheJIT http://thejit.org/ • JavaScript InfoVis Toolkit • Interactive • Link Graphs Logging as a Service 21 (c) by Raffael Marty Tuesday, July 6, 2010
  • 22. Processing •Visualization library •Java based •Interactive (event handling) •Number of libraries to -draw in OpenGL -read XML files -write PDF files •Processing JS -JavaScript -HTML 5 Canvas http://processingjs.org/ -Web IDE http://processing.org/ Logging as a Service 22 (c) by Raffael Marty Tuesday, July 6, 2010
  • 23. Building Your Own 23 Tuesday, July 6, 2010
  • 24. Build Your Own AfterGlow Loggly Regexes Google Vis Logging as a Service 24 (c) by Raffael Marty Tuesday, July 6, 2010
  • 25. Data Collection in the Cloud 25 Tuesday, July 6, 2010
  • 26. The (public) Cloud What it is Types • multi-tenancy • SaaS - Software • elastic • PaaS - Platform • “infinite” resources • IaaS - Infrastructure • pay as you go Benefits • self provisioning • No installation • No elaborate configurations It’s not • No maintenance • private data center • Great scalability • virtualization • 7x24 availability Logging as a Service 26 (c) by Raffael Marty Tuesday, July 6, 2010
  • 27. LaaS - Logging as a Service • All your data in one place • Loggly manages your data (index, store, archive, etc.) • Extremely fast search across all your data • Data source agnostic (no parsers) • Data management • access control • data segregation • data overview and summaries • API access Logging as a Service 27 (c) by Raffael Marty Tuesday, July 6, 2010
  • 28. Loggly Architecture Loggly Data Sources Clients user interface mobile-166 My syslog Data collection API Data access Proxies Distributed Indexers and Search Machines indexing and processing Distributed data store Logging as a Service 28 (c) by Raffael Marty Tuesday, July 6, 2010
  • 29. Loggly APIs • URL format: http://wiki.loggly.com/api-documentation http://<subdomain>.loggly.com/api/<resource> • RESTful API HTTP Based - Access through: /api/<resource> •GET - read - JSON, XML, JSONP output •POST - create • Authentication •PUT - update - Basic auth •DELETE - delete - oAuth http://loggly.loggly.com/api/search/?q=error syslog to: User: guest / Password: loggly logs.loggly.com:514 Logging as a Service 29 (c) by Raffael Marty Tuesday, July 6, 2010
  • 30. Search http://[domain].loggly.com/api/search?q=404 { "data": [ { "indexed": "2010-07-03T17:17:38.909Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au] [|domain] (DF)", "inputname": "logglyweb", "timestamp": "2010-07-03 10:17:38" }, { "indexed": "2010-07-03T17:17:37.879Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au] [|domain] (DF)", "inputname": "logglyapp", "timestamp": "2010-07-03 10:17:37" }, ... Logging as a Service 30 (c) by Raffael Marty Tuesday, July 6, 2010
  • 31. Parser Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF) Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF) Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF) (.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+): (d+.d+.d+.d+).?(d*) [<>] Regex / Parser (d+.d+.d+.d+).?(d*): (.*) Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF) Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF) (CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF) Logging as a Service 31 (c) by Raffael Marty Tuesday, July 6, 2010
  • 32. Visualize Parser AfterGlow Grapher CSV file Graph file digraph structs { graph [label="AfterGlow 1.5.8", fontsize=8]; node [shape=ellipse, style=filled, Configuration fontsize=10, width=1, height=1, fixedsize=true]; edge [len=1.6]; color.source=“green” if ($fields[0] ne “d”) "aaelenes" -> "Printing Resume" ; cluster.target=regex_replace("(d+).")."/8" "abbe" -> "Information Encryption" ; threshold.event=5 "aanna" -> "Patent Access" ; size.target=$fields[1] "aatharuv" -> "Ping" ; } http://afterglow.sf.net Logging as a Service 32 (c) by Raffael Marty Tuesday, July 6, 2010
  • 33. AfterGlow Cloud Grapher Loggly JSON CSV DOT Graph Logging as a Service 33 (c) by Raffael Marty Tuesday, July 6, 2010
  • 34. Google Vis • JSON to Graphs • DataTable - used among all charts • Interactivity through events Logging as a Service 34 (c) by Raffael Marty Tuesday, July 6, 2010
  • 35. <script type="text/javascript"> Google Vis Code google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']}); google.setOnLoadCallback(call); var trends = new Array(); function call() { l! a $.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100", n type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx', io success: function(data) { trends = data.data drawChart(); c t n } u }); f } t function drawChart() { o var data = new google.visualization.DataTable(); n data.addColumn('string', 'Search'); data.addColumn('datetime', 'Date'); is data.addColumn('number', 'Count'); e data.addRows(trends); od var chart = new google.visualization.MotionChart(document.getElementById('chart_div')); c chart.draw(data, {width: 600, height:300, state:state}); is var view = new google.visualization.DataView(data); h view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}])); T var table = new google.visualization.Table(document.getElementById('test_dataview')); table.draw(view, {sortColumn: 1}); var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline')); time.draw(timedata, {displayAnnotations: true}); } </script> Logging as a Service 35 (c) by Raffael Marty Tuesday, July 6, 2010
  • 36. Visualization Use-Cases 36 Tuesday, July 6, 2010
  • 37. NetFlow Visualization • Treemap • Protovis.JS • Size: Amount • Brightness: Variance • Color: Sensor • Shows: Scans - bright spots • Thanks to Chris Horsley Logging as a Service 37 (c) by Raffael Marty Tuesday, July 6, 2010
  • 38. Firewall Treemap Logging as a Service 38 (c) by Raffael Marty Tuesday, July 6, 2010
  • 39. Firewall Log Port Source IP Destination IP Logging as a Service 39 (c) by Raffael Marty Tuesday, July 6, 2010
  • 40. Visualization Resources 40 Tuesday, July 6, 2010
  • 41. http://secviz.org Share, discuss, challenge, and learn about security visualization. • List: secviz.org/mailinglist • Twitter: @secviz Logging as a Service 41 (c) by Raffael Marty Tuesday, July 6, 2010
  • 42. Applied Security Visualization • Bridging the gap between security and visualization • Hands-on, end to end examples • Data processing and analysis Chapters • Visualization • Compliance • Data Sources • Insider Threat • From Data to Graphs • Visualization Tools Addison Wesley (August, 2008) • Perimeter Threat ISBN: 0321510100 Logging as a Service 42 (c) by Raffael Marty Tuesday, July 6, 2010
  • 43. Thank You! raffael.marty@loggly.com @zrlram 43 Tuesday, July 6, 2010