Bitdefender’s Software Release Process: Ensuring Reliability

With the increasing frequency of cybersecurity incidents, ensuring reliable software updates has become paramount, especially after the recent industry events following CrowdStrike's product update, which resulted in a widespread outage. We would like to address our software release process for transparency and clarification.

"At Bitdefender, our primary focus remains on ensuring the safety and satisfaction of our customers, nurturing our partnerships, and upholding our reputation for technical excellence," mentioned Mr. Paul Hadji, VP of APAC and Cyber Security Services of Bitdefender. "We can confirm that Bitdefender's internal systems and operations were not affected by the CrowdStrike and Microsoft issues. Our services continue to operate normally without any disruptions."

To provide transparency and reassure our customers, we have released an in-depth look at our software release process, which rigorously tests and validates every update before deployment according to the best practices of the Software Development Lifecycle (SDLC).

1. Development and QA Process

Our commitment to quality and reliability begins with a robust Software Development Lifecycle (SDLC). This process incorporates best practices at every phase to prevent software issues from reaching the final product. Through comprehensive testing and continuous improvement, we maintain the highest standards of performance and functionality. Each update undergoes rigorous manual and automated testing before deployment.

Automated Testing:

Over 90% of potential scenarios are covered by our extensive suite of automated tests. This allows us to efficiently re-run the entire test suite following minor code changes, ensuring consistent quality throughout development. These automated tests are executed daily between releases, maintaining high standards of reliability.

Manual Testing:

While automation is crucial, our skilled QA engineers conduct manual tests to identify issues that automated tests might miss. This dual approach ensures a thorough examination of the software, enhancing overall reliability.

2. Release Process and Best Practices

We use our own products and services internally to test their functionality, identify issues, and ensure they meet our high standards before release. Updates are deployed across our entire company, not just engineering, ensuring thorough real-world testing before wider release.

Sensitive updates, such as major kernel driver updates, undergo extended controlled rollouts (sometimes lasting months) with clear communication in the Release Notes to minimize disruption.

Cloud Releases: Gradual Regional Deployment

For cloud updates, we use a gradual regional deployment process to minimize the potential impact of unforeseen issues by rolling out updates across multiple regions in phases.

We start with a "virtual" region composed of our employees, allowing us to identify and resolve potential issues before they impact customers.

The update is then gradually deployed across different regions over a set timeframe, allowing us to address issues without impacting all users simultaneously.

Endpoint Agents: Staging Updates

For endpoint agent updates, we use a phased rollout strategy. Here’s a breakdown of the stages:

Super Fast Ring – Updates are first deployed internally to all Bitdefender employees for real-world testing and feedback after engineering approval.

Fast Ring – After internal testing, the update is available on the Fast Ring. This program allows customers to test the update in their environments before wider deployment, helping us identify unforeseen issues or compatibility problems.

Updates typically remain on the Fast Ring for a week. For emergency fixes, the Slow Ring can be updated within 24 hours of the Fast Ring release. For critical issues, Slow Ring releases can be delayed or stopped entirely until a new version with resolved issues is released, restarting the release cycle.

Slow Ring – This option prioritizes stability, receiving updates at a later date. By the time an update reaches the Slow Ring, it has been thoroughly tested internally and by Fast Ring users, and any necessary adjustments have been made.

Quickly assign endpoints to the Fast or Slow Ring for managed updates.

You can easily assign endpoints to either the Fast or Slow Ring for controlled updates. Early identification and resolution of issues in the Super Fast Ring and Fast Ring stages have resulted in no rollbacks being needed for Slow Ring updates in 2024.

Summary

We’ve outlined how Bitdefender carefully rolls out updates in stages to ensure smooth operations. Keeping systems secure is a team effort. Here are some best practices to avoid unnecessary disruptions:

• Use Both Rings: Utilize both Fast and Slow Rings.
• Test Early with Fast Ring: Assign a small number of non-critical machines to the Fast Ring to catch compatibility issues before widespread deployment.
• Stay Stable with Slow Ring: Keep most machines on the Slow Ring to benefit from the extra testing by Fast Ring users, reducing the likelihood of unexpected problems.
• Separate Rings, Separate Rules: Establish different policies for Fast and Slow Ring updates for better control over update timing and targeting specific machines.
• Test Like a User: Perform User Acceptance Testing (UAT) where regular end users test the Fast Ring update in the actual production network during regular work hours.
• Review Release Notes: Always review the release notes before deploying updates, especially on the Slow Ring, to understand the specific changes, bug fixes, and known issues.

By following these best practices and our staged release strategy, you can ensure a smooth and reliable update process.

Important Links:

Bitdefender Software Release Strategy

Release Note

Staging updates