OpenStack@eBay About me • 在IT业打混了14年 • 从电信和通讯行业开始职业生涯 • 在2009年进入了互联网行业，从那时起专注于大规模分布式系统的建设 • 是eBay自建云平台Stratus的主要贡献者 • 是eBay新一代基于Openstack云平台的技术lead • 曾经是一名活跃的开源贡献者，从2005年开始活跃于source forge 社区 44 Agenda Why Openstack Openstack @ scale Plan for failure Monitoring, Alerting, Operation Build & Deploy Q&A 55 Openstack journey COMPUTE 12+K Hypervisors 300K Cores 70K VMs NETWORK STORAGE 15 VPCs 4.4 PB Local 9 AZs 1.6 PB Block 3 Regions 52 TB Object OpenStack 2015 Super User Finalist 6 Before openstack •Home grown Cloud : Stratus,includes: –OSP (Baremetal, Virtual Machine) –LBMS –DNS –RAS –… •Scale –99+% eBay site traffic –80,000+ virtual computes –30,000+ physical computes 7 In 2012 Prod Prod QA PCI Secure QA DEV QA DEV 8 DEV One cloud on shared infrastructure 1 2 3 One place to go oAs a service oDashboard oService catalog oService health oSupport One way to do things oOne control plane identity oOne set of public APIs oInter-operable 9 One logical control plane oGlobal topology oMulti-tenant oService ecosystem Principles •Any application Anywhere •Soft cabling •Shared Standardized Infrastructure •Virtualize everything •Automate everything 10 Openstack •Community •Vendor ecosystem •Technology •Adoption 11 Start from Dev Cloud in 2012 •A logical environment defined as a VPC on top of shared infrastructure –Self service VM for developers –Access is same as to their desktops •Implemented as a set of L2 networks (/24) with in a given L3(/20) –No private networks: all developers on same shared networks –No private IP space: traffic is routed within core, no need for floating Ips 12 Openstack is not cloud Monitoring and Alerting Cloud is a Service Metering and Chargback Autoscaling Remediation Config mgmt High availability Infra onboarding Log processing Openstack APIs Metrics CI Network design Capacity planning Upgrades Build Cloud monitoring Scale out Net/info sec SLA Alerting Incident resolution User experience 13 Customer support PLAN FOR FAILURE 14 Patterns for High Availability -global Multiple Geographic Regions for DR PHX SLC LVS PHX01 SLC01 LVS01 PHX02 SLC07 … … … LVS02 … … 2 or More Availability Zones per Region - No sharing, low latency 15 Patterns for High Availability -AZ Global Keystone AZ 1 VIP Service Catalog Mysql Galera Cluster Load Balancer Node nova-api nova-api neutron-api neutron-api cinder-api cinder-api glance-api glance-api Node-1 Pacemaker DHCP-agent Node-N VIP VIP Load Balancer Load Balancer Rabbit (Mem) DHCP-agent Rabbit (Mem) Mysql Rabbit (Disk) 16 Replication Mysql High Availability for your customer Pool Request Region A Openstack Region B Openstack Rabbit Rabbit Starch Starch CMS Biz Rule Falcon 17 Violation Report Monitoring, Alerting, Operation 18 Monitoring evolution Log Source logstash Log Source logstash Log Source logstash ElasticSearch Kibana ElasticSearch Graphite StatsD Zabbix Stackwatch HV, Openstack nodes,processes, network, storage … Stackmetrics 19 Monitoring control plane •Zabbix •Stackmetrics –Including compute capacity monitoring –Including network capacity monitoring •StackWatch –Customer simulator 20