SESSION ID: PS-W02 8 Million Findings in One Year: Fresh Look at the State of Software Security Chris Wysopal Jay Jacobs Co-Founder and CTO Veracode @WeldPond Co-Founder and Chief Data Scientist Cyentia Institute @jayjacobs #RSAC What is this research? • Veracode State of Software Security (SoSS), Vol. 10 • Largest quantitative study of application security findings • Partnered with data scientists at Cyentia Institute #RSAC The Why: • Insights into industry performance, and impact of DevSecOps on fix rates • Provide data for customers to benchmark themselves against their peers • Generate actionable advice for improving application security programs The How: • Formulate questions that might be answerable given the available data • Stand back and use science #RSAC The Data… #RSAC Vol. 1 Vol. 10 software tested software tested 1,591 Over 2,300 Veracode customers 12 months of software scan data: April 1, 2018 – March 31, 2019 Over 85,000 unique pieces of software and 1.4 million individual assessments 85,000+ That’s over a 50-fold increase in sample size! #RSAC The State of Software Security #RSAC Mean Time to Remediation among closed findings • The median fix time remains relatively unchanged from 10 years ago. • However, the tail of everaccruing “security debt” just got a lot longer, causing the mean closed time to stretch out. SANS OWASP Overall Proportion of software applications/products with at least one flaw in the initial scan #RSAC #RSAC Software with high-severity flaws Majority of products/applications are free from high/critical flaws Prevalence of flaw categories in SOSS Volume 1 and 10 #RSAC There is a general increase in web-related categories, likely due to a lot more web applications being written. Less code is being written in C/C++ so buffer overflows, buffer management errors, and numeric errors are way down. Fix rate across all flaws “Fix rate” is the proportion of discovered flaws that are successfully closed or remediated. #RSAC #RSAC Fix Behavior Measuring time to remediate is challenging… • Simple approach is to calculate time for remediated findings • Ignores the still-open (security debt) • But it’s simple and intuitive • Survival analysis studies the time to an event • Accounts for findings that are still open (security debt) • • Team stopped scanning Not closed yet, was still open at last scan #RSAC Time to Failure (example) #RSAC These are “censored” - all we know is they lasted ”at least” this long. Time to Failure (example) #RSAC Observations are lined up so they all start on day 0. Time to Failure (example) #RSAC Line represents best estimate of probability an event hasn’t occurred yet. Flaw persistence curve #RSAC These look at the observed time for only the closed findings. These look at both “closed” and “still-open” findings to estimate median/mean. Flaw persistence curve #RSAC Median Time-to-remediate across flaw categories #RSAC Time-to-remediation across flaw severity scores Surprisingly, flaw severity doesn’t correlate strongly with fix speed Even the Very High severity flaws have a long tail for fix time, taking over 130 days to reach the 75% closed milestone. #RSAC Fixed Quick Fixed Slow Speed and comprehensiveness for flaw categories Selective Thorough #RSAC