SlideShare a Scribd company logo
1 of 20
Download to read offline
Don’t Break the Glass!
In case of emergency…
Intro
• Apache CloudStack committer, PMC member
• Security Guy
• Founder of Stratosec
• …hacker, reverse-engineer
• @johnlkinsella, #CCCNA14
Goal
Provide bread crumbs for you to gain expertise
on CloudStack’s data model so you can react
quickly in an emergency situation.
TOC
• What Should Production Look Like?
• My Nightmare
• My response
• How to prepare
• How we’re trying to make things better
• Homework
What Should Production Look Like?
• Briefly:
– Redundant management servers
– Redundant databases
– Redundant storage (primary and secondary)
– Redundant infrastructure
– Data backups – encrypted
– Pro tip: store backups outside CloudStack.
One Painful February Day…
• Made a change in control panel prior to giving customer access
• Control panel interpreted this as “Delete the Customer.”
• More precisely: “Delete the Customer, VMs and all storage. Now.”
Wait, What??
• Took about 15 seconds to realize what happened
• Looking in CloudStack Management UI, saw VM shutting down,
then being expunged.
Periodic Table…
Emergency Steps
• Shut down ACS management server immediately
• Fire up your favorite SQL tool (I recommend Squirrel SQL)
With your Data Recovery Shrine of Choice
Immediately Establish Contact
Establish Contact With Data Shrine
Know how CloudStack Works
• Steps to delete a VM:
– Shut down
– Destroy
– Expunge volumes
– Expunge VM
The “SQL API”
• DB Tables:
– vm_instance
– volumes
– nics
– user
– account
• Common Fields:
• account_id
• instance_id
• created/updated/removed
• state
• removed
BACK UP DB BEFORE MAKING CHANGES!
Actual SQL
Make change in SQL.
Bonus: changing VM’s IPUPDATE user_ip_address SET allocated=(SELECT allocated FROM (SELECT allocated,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET account_id=(SELECT account_id FROM (SELECT account_id,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET domain_id=(SELECT domain_id FROM (SELECT allocated,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET state='Allocated' WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET network_id=(SELECT network_id FROM (SELECT network_id,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE nics SET ip4_address='184.172.14.193' WHERE instance_id=1797;
UPDATE vm_instance SET private_ip_address='184.172.14.193' WHERE instance_name='i-2-1797-
VM';
Move VM between accounts
UPDATE user_ip_address SET account_id=10 WHERE id=29;
UPDATE user_ip_address SET domain_id=8 WHERE id=29;
UPDATE user_ip_address SET network_id=217 WHERE id=29;
UPDATE vm_network_map SET network_id=217 WHERE vm_id=144;
UPDATE vm_instance SET domain_id=8 WHERE id=29;
UPDATE op_networks SET nics_count=nics_count+1 WHERE id=217;
UPDATE op_networks SET mac_address_seq=mac_address_seq+1 WHERE id=217;
UPDATE nics SET broadcast_uri='vlan://230' WHERE id=179;
UPDATE nics SET network_id=217 WHERE id=179;
UPDATE nics SET isolation_uri='vlan://230' WHERE id=179;
UPDATE firewall_rules SET account_id=10 WHERE ip_address_id=29;
UPDATE firewall_rules SET domain_id=8 WHERE ip_address_id=29;
UPDATE firewall_rules SET network_id=217 WHERE ip_address_id=29;
How you can prepare
• Create written cloud policy, audit production to match
• Test your backups
• Become familiar with CloudStack internals
• Do you really need frequent data expunge processes?
Next: Production Lock
• Feature: Allow our customers to ensure that no automations will
modify or delete their compute or storage instances
– Working for VM
– Adding for storage, networks
– Need to add to UI
Homework
• Pick a user VM in your Cloud. Via the database:
– Find the VM instance record
– Find the VM’s nics in the database
– Find the VM’s storage in the database
– Try deleting a VM via normal means, then before expunge,
restore via SQL
kthxbye!
• http://cloudstack.apache.org
jlk@stratosec.co
@johnlkinsella
http://theresnomon.co

More Related Content

What's hot

How To Check and Delete a File via PowerShell
How To Check and Delete a File via PowerShellHow To Check and Delete a File via PowerShell
How To Check and Delete a File via PowerShellVCP Muthukrishna
 
Flexviews materialized views for my sql
Flexviews materialized views for my sqlFlexviews materialized views for my sql
Flexviews materialized views for my sqlJustin Swanhart
 
DataStax NYC Java Meetup: Cassandra with Java
DataStax NYC Java Meetup: Cassandra with JavaDataStax NYC Java Meetup: Cassandra with Java
DataStax NYC Java Meetup: Cassandra with Javacarolinedatastax
 
MySQL > YourSQL
MySQL > YourSQLMySQL > YourSQL
MySQL > YourSQLcafuego
 
MySQL Performance Schema in Action
MySQL Performance Schema in ActionMySQL Performance Schema in Action
MySQL Performance Schema in ActionSveta Smirnova
 
Why Use EXPLAIN FORMAT=JSON?
 Why Use EXPLAIN FORMAT=JSON?  Why Use EXPLAIN FORMAT=JSON?
Why Use EXPLAIN FORMAT=JSON? Sveta Smirnova
 
Summary tables with flexviews
Summary tables with flexviewsSummary tables with flexviews
Summary tables with flexviewsJustin Swanhart
 
Odv oracle customer_demo
Odv oracle customer_demoOdv oracle customer_demo
Odv oracle customer_demoViaggio Italia
 
Highload Perf Tuning
Highload Perf TuningHighload Perf Tuning
Highload Perf TuningHighLoad2009
 
DBA Commands and Concepts That Every Developer Should Know - Part 2
DBA Commands and Concepts That Every Developer Should Know - Part 2DBA Commands and Concepts That Every Developer Should Know - Part 2
DBA Commands and Concepts That Every Developer Should Know - Part 2Alex Zaballa
 
Moving to the NoSQL side: MySQL JSON functions
 Moving to the NoSQL side: MySQL JSON functions Moving to the NoSQL side: MySQL JSON functions
Moving to the NoSQL side: MySQL JSON functionsSveta Smirnova
 
Oracle performance tuning
Oracle performance tuningOracle performance tuning
Oracle performance tuningvksgarg
 
New features in Performance Schema 5.7 in action
New features in Performance Schema 5.7 in actionNew features in Performance Schema 5.7 in action
New features in Performance Schema 5.7 in actionSveta Smirnova
 
Performance Schema for MySQL troubleshooting
Performance Schema for MySQL troubleshootingPerformance Schema for MySQL troubleshooting
Performance Schema for MySQL troubleshootingSveta Smirnova
 

What's hot (16)

How To Check and Delete a File via PowerShell
How To Check and Delete a File via PowerShellHow To Check and Delete a File via PowerShell
How To Check and Delete a File via PowerShell
 
Flexviews materialized views for my sql
Flexviews materialized views for my sqlFlexviews materialized views for my sql
Flexviews materialized views for my sql
 
DataStax NYC Java Meetup: Cassandra with Java
DataStax NYC Java Meetup: Cassandra with JavaDataStax NYC Java Meetup: Cassandra with Java
DataStax NYC Java Meetup: Cassandra with Java
 
CKAN 2.2 Installation
CKAN 2.2 InstallationCKAN 2.2 Installation
CKAN 2.2 Installation
 
MySQL > YourSQL
MySQL > YourSQLMySQL > YourSQL
MySQL > YourSQL
 
MySQL Performance Schema in Action
MySQL Performance Schema in ActionMySQL Performance Schema in Action
MySQL Performance Schema in Action
 
Why Use EXPLAIN FORMAT=JSON?
 Why Use EXPLAIN FORMAT=JSON?  Why Use EXPLAIN FORMAT=JSON?
Why Use EXPLAIN FORMAT=JSON?
 
Summary tables with flexviews
Summary tables with flexviewsSummary tables with flexviews
Summary tables with flexviews
 
Odv oracle customer_demo
Odv oracle customer_demoOdv oracle customer_demo
Odv oracle customer_demo
 
iCloud keychain
iCloud keychainiCloud keychain
iCloud keychain
 
Highload Perf Tuning
Highload Perf TuningHighload Perf Tuning
Highload Perf Tuning
 
DBA Commands and Concepts That Every Developer Should Know - Part 2
DBA Commands and Concepts That Every Developer Should Know - Part 2DBA Commands and Concepts That Every Developer Should Know - Part 2
DBA Commands and Concepts That Every Developer Should Know - Part 2
 
Moving to the NoSQL side: MySQL JSON functions
 Moving to the NoSQL side: MySQL JSON functions Moving to the NoSQL side: MySQL JSON functions
Moving to the NoSQL side: MySQL JSON functions
 
Oracle performance tuning
Oracle performance tuningOracle performance tuning
Oracle performance tuning
 
New features in Performance Schema 5.7 in action
New features in Performance Schema 5.7 in actionNew features in Performance Schema 5.7 in action
New features in Performance Schema 5.7 in action
 
Performance Schema for MySQL troubleshooting
Performance Schema for MySQL troubleshootingPerformance Schema for MySQL troubleshooting
Performance Schema for MySQL troubleshooting
 

Similar to Dont break the glass

11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01Karam Abuataya
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11gfcamachob
 
Migrate database to Exadata using RMAN duplicate
Migrate database to Exadata using RMAN duplicateMigrate database to Exadata using RMAN duplicate
Migrate database to Exadata using RMAN duplicateUmair Mansoob
 
LVOUG meetup #4 - Case Study 10g to 11g
LVOUG meetup #4 - Case Study 10g to 11gLVOUG meetup #4 - Case Study 10g to 11g
LVOUG meetup #4 - Case Study 10g to 11gMaris Elsins
 
Short Intro to PHP and MySQL
Short Intro to PHP and MySQLShort Intro to PHP and MySQL
Short Intro to PHP and MySQLJussi Pohjolainen
 
Développer avec un Simple Object Mapping Toolkit pour SQL Server
Développer avec un Simple Object Mapping Toolkit pour SQL ServerDévelopper avec un Simple Object Mapping Toolkit pour SQL Server
Développer avec un Simple Object Mapping Toolkit pour SQL ServerDenis Voituron
 
Trouble shooting apachecloudstack
Trouble shooting apachecloudstackTrouble shooting apachecloudstack
Trouble shooting apachecloudstackSailaja Sunil
 
DNN Database Tips & Tricks
DNN Database Tips & TricksDNN Database Tips & Tricks
DNN Database Tips & TricksWill Strohl
 
OpenWorld Sep14 12c for_developers
OpenWorld Sep14 12c for_developersOpenWorld Sep14 12c for_developers
OpenWorld Sep14 12c for_developersConnor McDonald
 
How the world gets its weather
How the world gets its weather How the world gets its weather
How the world gets its weather Ravi Yadav
 
Formal Verification of Web Service Interaction Contracts
Formal Verification of Web Service Interaction ContractsFormal Verification of Web Service Interaction Contracts
Formal Verification of Web Service Interaction ContractsGera Shegalov
 
Proof of Concept with Real Application Testing 12c
Proof of Concept with Real Application Testing 12cProof of Concept with Real Application Testing 12c
Proof of Concept with Real Application Testing 12cLuis Marques
 
4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...
4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...
4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...PROIDEA
 
MySQL 8.0.18 - New Features Summary
MySQL 8.0.18 - New Features SummaryMySQL 8.0.18 - New Features Summary
MySQL 8.0.18 - New Features SummaryOlivier DASINI
 
Oracle 12c Automatic Data Optimization (ADO) - ILM
Oracle 12c Automatic Data Optimization (ADO) - ILMOracle 12c Automatic Data Optimization (ADO) - ILM
Oracle 12c Automatic Data Optimization (ADO) - ILMMonowar Mukul
 
Owasp Indy Q2 2012 Advanced SQLi
Owasp Indy Q2 2012 Advanced SQLiOwasp Indy Q2 2012 Advanced SQLi
Owasp Indy Q2 2012 Advanced SQLiowaspindy
 
smartdc by Ruby
smartdc by Rubysmartdc by Ruby
smartdc by Rubyogom_
 
Optimizing applications and database performance
Optimizing applications and database performanceOptimizing applications and database performance
Optimizing applications and database performanceInam Bukhary
 

Similar to Dont break the glass (20)

11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11g
 
Migrate database to Exadata using RMAN duplicate
Migrate database to Exadata using RMAN duplicateMigrate database to Exadata using RMAN duplicate
Migrate database to Exadata using RMAN duplicate
 
LVOUG meetup #4 - Case Study 10g to 11g
LVOUG meetup #4 - Case Study 10g to 11gLVOUG meetup #4 - Case Study 10g to 11g
LVOUG meetup #4 - Case Study 10g to 11g
 
Short Intro to PHP and MySQL
Short Intro to PHP and MySQLShort Intro to PHP and MySQL
Short Intro to PHP and MySQL
 
Développer avec un Simple Object Mapping Toolkit pour SQL Server
Développer avec un Simple Object Mapping Toolkit pour SQL ServerDévelopper avec un Simple Object Mapping Toolkit pour SQL Server
Développer avec un Simple Object Mapping Toolkit pour SQL Server
 
Trouble shooting apachecloudstack
Trouble shooting apachecloudstackTrouble shooting apachecloudstack
Trouble shooting apachecloudstack
 
DNN Database Tips & Tricks
DNN Database Tips & TricksDNN Database Tips & Tricks
DNN Database Tips & Tricks
 
OpenWorld Sep14 12c for_developers
OpenWorld Sep14 12c for_developersOpenWorld Sep14 12c for_developers
OpenWorld Sep14 12c for_developers
 
How the world gets its weather
How the world gets its weather How the world gets its weather
How the world gets its weather
 
Formal Verification of Web Service Interaction Contracts
Formal Verification of Web Service Interaction ContractsFormal Verification of Web Service Interaction Contracts
Formal Verification of Web Service Interaction Contracts
 
Proof of Concept with Real Application Testing 12c
Proof of Concept with Real Application Testing 12cProof of Concept with Real Application Testing 12c
Proof of Concept with Real Application Testing 12c
 
Results cache
Results cacheResults cache
Results cache
 
4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...
4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...
4Developers 2015: CQRS - Prosta architektura dla nieprostego systemu! - Mateu...
 
MySQL 8.0.18 - New Features Summary
MySQL 8.0.18 - New Features SummaryMySQL 8.0.18 - New Features Summary
MySQL 8.0.18 - New Features Summary
 
Oracle 12c Automatic Data Optimization (ADO) - ILM
Oracle 12c Automatic Data Optimization (ADO) - ILMOracle 12c Automatic Data Optimization (ADO) - ILM
Oracle 12c Automatic Data Optimization (ADO) - ILM
 
Owasp Indy Q2 2012 Advanced SQLi
Owasp Indy Q2 2012 Advanced SQLiOwasp Indy Q2 2012 Advanced SQLi
Owasp Indy Q2 2012 Advanced SQLi
 
smartdc by Ruby
smartdc by Rubysmartdc by Ruby
smartdc by Ruby
 
Osol Pgsql
Osol PgsqlOsol Pgsql
Osol Pgsql
 
Optimizing applications and database performance
Optimizing applications and database performanceOptimizing applications and database performance
Optimizing applications and database performance
 

More from John Kinsella

Removing the Burden of Securing Microservices Through Automation and Visibility
Removing the Burden of Securing Microservices Through Automation and VisibilityRemoving the Burden of Securing Microservices Through Automation and Visibility
Removing the Burden of Securing Microservices Through Automation and VisibilityJohn Kinsella
 
2019 Infosec World Keynote
2019 Infosec World Keynote2019 Infosec World Keynote
2019 Infosec World KeynoteJohn Kinsella
 
An In-depth look at application containers
An In-depth look at application containersAn In-depth look at application containers
An In-depth look at application containersJohn Kinsella
 
Understanding container security
Understanding container securityUnderstanding container security
Understanding container securityJohn Kinsella
 
Docker security configuration
Docker security configurationDocker security configuration
Docker security configurationJohn Kinsella
 
A (fun!) Comparison of Docker Vulnerability Scanners
A (fun!) Comparison of Docker Vulnerability ScannersA (fun!) Comparison of Docker Vulnerability Scanners
A (fun!) Comparison of Docker Vulnerability ScannersJohn Kinsella
 
CloudStack and the HeartBleed vulnerability
CloudStack and the HeartBleed vulnerabilityCloudStack and the HeartBleed vulnerability
CloudStack and the HeartBleed vulnerabilityJohn Kinsella
 
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...John Kinsella
 
What is Cloud Security, and Can I Have Some?
What is Cloud Security, and Can I Have Some?What is Cloud Security, and Can I Have Some?
What is Cloud Security, and Can I Have Some?John Kinsella
 

More from John Kinsella (11)

Removing the Burden of Securing Microservices Through Automation and Visibility
Removing the Burden of Securing Microservices Through Automation and VisibilityRemoving the Burden of Securing Microservices Through Automation and Visibility
Removing the Burden of Securing Microservices Through Automation and Visibility
 
2019 Infosec World Keynote
2019 Infosec World Keynote2019 Infosec World Keynote
2019 Infosec World Keynote
 
An In-depth look at application containers
An In-depth look at application containersAn In-depth look at application containers
An In-depth look at application containers
 
Understanding container security
Understanding container securityUnderstanding container security
Understanding container security
 
Docker security configuration
Docker security configurationDocker security configuration
Docker security configuration
 
A (fun!) Comparison of Docker Vulnerability Scanners
A (fun!) Comparison of Docker Vulnerability ScannersA (fun!) Comparison of Docker Vulnerability Scanners
A (fun!) Comparison of Docker Vulnerability Scanners
 
CloudStack and the HeartBleed vulnerability
CloudStack and the HeartBleed vulnerabilityCloudStack and the HeartBleed vulnerability
CloudStack and the HeartBleed vulnerability
 
CloudStack Secured
CloudStack SecuredCloudStack Secured
CloudStack Secured
 
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
Truly Secure: The Steps a Security Practitioner Took to Build a Secure Public...
 
Securing the Cloud
Securing the CloudSecuring the Cloud
Securing the Cloud
 
What is Cloud Security, and Can I Have Some?
What is Cloud Security, and Can I Have Some?What is Cloud Security, and Can I Have Some?
What is Cloud Security, and Can I Have Some?
 

Recently uploaded

Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 

Recently uploaded (20)

Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 

Dont break the glass

  • 1. Don’t Break the Glass! In case of emergency…
  • 2. Intro • Apache CloudStack committer, PMC member • Security Guy • Founder of Stratosec • …hacker, reverse-engineer • @johnlkinsella, #CCCNA14
  • 3. Goal Provide bread crumbs for you to gain expertise on CloudStack’s data model so you can react quickly in an emergency situation.
  • 4. TOC • What Should Production Look Like? • My Nightmare • My response • How to prepare • How we’re trying to make things better • Homework
  • 5. What Should Production Look Like? • Briefly: – Redundant management servers – Redundant databases – Redundant storage (primary and secondary) – Redundant infrastructure – Data backups – encrypted – Pro tip: store backups outside CloudStack.
  • 6. One Painful February Day… • Made a change in control panel prior to giving customer access • Control panel interpreted this as “Delete the Customer.” • More precisely: “Delete the Customer, VMs and all storage. Now.”
  • 7. Wait, What?? • Took about 15 seconds to realize what happened • Looking in CloudStack Management UI, saw VM shutting down, then being expunged.
  • 9. Emergency Steps • Shut down ACS management server immediately • Fire up your favorite SQL tool (I recommend Squirrel SQL)
  • 10. With your Data Recovery Shrine of Choice Immediately Establish Contact Establish Contact With Data Shrine
  • 11. Know how CloudStack Works • Steps to delete a VM: – Shut down – Destroy – Expunge volumes – Expunge VM
  • 12. The “SQL API” • DB Tables: – vm_instance – volumes – nics – user – account • Common Fields: • account_id • instance_id • created/updated/removed • state • removed BACK UP DB BEFORE MAKING CHANGES!
  • 15. Bonus: changing VM’s IPUPDATE user_ip_address SET allocated=(SELECT allocated FROM (SELECT allocated, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193'; UPDATE user_ip_address SET account_id=(SELECT account_id FROM (SELECT account_id, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193'; UPDATE user_ip_address SET domain_id=(SELECT domain_id FROM (SELECT allocated, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193'; UPDATE user_ip_address SET state='Allocated' WHERE public_ip_address='184.172.14.193'; UPDATE user_ip_address SET network_id=(SELECT network_id FROM (SELECT network_id, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193'; UPDATE nics SET ip4_address='184.172.14.193' WHERE instance_id=1797; UPDATE vm_instance SET private_ip_address='184.172.14.193' WHERE instance_name='i-2-1797- VM';
  • 16. Move VM between accounts UPDATE user_ip_address SET account_id=10 WHERE id=29; UPDATE user_ip_address SET domain_id=8 WHERE id=29; UPDATE user_ip_address SET network_id=217 WHERE id=29; UPDATE vm_network_map SET network_id=217 WHERE vm_id=144; UPDATE vm_instance SET domain_id=8 WHERE id=29; UPDATE op_networks SET nics_count=nics_count+1 WHERE id=217; UPDATE op_networks SET mac_address_seq=mac_address_seq+1 WHERE id=217; UPDATE nics SET broadcast_uri='vlan://230' WHERE id=179; UPDATE nics SET network_id=217 WHERE id=179; UPDATE nics SET isolation_uri='vlan://230' WHERE id=179; UPDATE firewall_rules SET account_id=10 WHERE ip_address_id=29; UPDATE firewall_rules SET domain_id=8 WHERE ip_address_id=29; UPDATE firewall_rules SET network_id=217 WHERE ip_address_id=29;
  • 17. How you can prepare • Create written cloud policy, audit production to match • Test your backups • Become familiar with CloudStack internals • Do you really need frequent data expunge processes?
  • 18. Next: Production Lock • Feature: Allow our customers to ensure that no automations will modify or delete their compute or storage instances – Working for VM – Adding for storage, networks – Need to add to UI
  • 19. Homework • Pick a user VM in your Cloud. Via the database: – Find the VM instance record – Find the VM’s nics in the database – Find the VM’s storage in the database – Try deleting a VM via normal means, then before expunge, restore via SQL

Editor's Notes

  1. (I mean hacker in the developer sense, not TV bad guy)
  2. Kelvin Yang:“noticable transition period”
  3. Know how to do this stuff. In an emergency situation you must be able to react quickly.
  4. Images from:https://www.flickr.com/photos/theogeo/2768774330https://www.flickr.com/photos/raviahuja/9074994473https://www.flickr.com/photos/scoobymoo/76916282https://www.flickr.com/photos/cuppini/6008317955https://www.flickr.com/photos/jeffk/743193940
  5. We could create API calls to help with this, but not sure it’s worth the balance of ACS running and potentially doing more damage.
  6. Making the change in SQL gives you history
  7. This one’s not quite perfect
  8. Icons from http://webtreatsetc.deviantart.com/art/Silver-Button-Social-Media-123487830