Srikanth AnnavarapuDGM – Freyr Digital

Measuring DORA Metrics

Data Collection: Manual vs. Automated

Automated data collection is the best way to avoid inaccurate metric which further leads to incorrect Business decisions. However, automation of every parameter involved in calculating a metric might not be possible due to constraints. In such cases, agree on a manual approach that is reasonable and aligned with Industry standards.

Lead Time:

Measured based on time taken by a feature to move from Business request to production. This can be measured by employing multiple tools at each step.

  • Automate testing & deployments (CI/CD pipelines).
  • Reduce batch sizes (smaller, more frequent changes).
  • Improve code review & approval processes.
  • Monitor bottlenecks (e.g., long QA cycles, manual approvals)

Manual method:

When required tools/plugins are not available for automated calculation of Lead time, a manual approach can be adopted. you can manually calculate Lead Time in Azure DevOps (ADO) or similar systems using raw data extracts. Here’s a step-by-step approach:

Data Extraction

  • Source: ADO Work Item Query or API.

Scope:

  • Filter for Features/User Stories marked as Done (i.e., deployed to Production).

Extract:

  • Created Date(when the work item was requested).
  • Closed/Completed Date(when marked "Done").

Formula for Feature/user story Lead time:

  • Lead Time (per item) = Closed Date - Created Date

Formula for Project-Level Average

  • Avg Lead Time = Sum(Lead Time per item) ÷ Total # of Items

* Note: In cases where planned features are created ahead of time, lead time metric would not be a true reflection and in those cases, Cycle time can be used as an alternative metric.

Deployment Frequency:
  • Count the number of deployments to production over a given period (e.g., per day, week, or month).

Data Extraction:

  • Use CI/CD tools (e.g., GitHub Actions) to log count of deployments.

Formula for Deployment Frequency:

  • Total number of deployments/Number of months

Formula for Project-Level Average:

  • Number of deployments of the project/ Number of months

*Note: As per Industry definition, Hotfix deployments are excluded from count of deployments.

Change Failure Rate:

What Counts as a "Failed Deployment"?

  • Rollbacks (reverting a release due to issues).
  • Hotfixes (urgent patches post-deployment).

Tracking Method:

  • Integrate with CI/CD pipeline deployment failures (e.g., Jenkins, GitHub Actions) to flag failures.
  • Use incident management tools (e.g., ADO, Jira, ServiceNow).

 Formula for Change Failure Rate:

  • Number of failed changes÷ Total number of Changes× 100.

Manual Method:

To calculate number of failed changes, count of support incidents that are categorized as “Software Bug” can be used. This divided by total number of changes released gives us CFR.

Organizations need to have the incidents classified periodically for the metric to be accurate.

Mean Time To Restore (MTTR):

What Counts as "Recovery Time"?

  • Start: When the failure is detected (alert triggered).
  • End: When the system is fully restored (e.g., rollback complete, hotfix deployed).

Tracking Method:

  • Incident management tools (e.g., ADO, Jira).
  • Monitoring systems (e.g., CloudWatch, Prometheus) to log resolution times.

Formula to calculate MTTR:

  • Sum of Restoration Times ÷ Number of Failures

Manual Method:

Sum of Restoration Times can be obtained based on the support incidents that are classified as “Software Bug”. Difference between “Created Date” & “Closed Date” in days can be used as numerator.

Use count of incidents classified as “Software Bug” can be used as denominator.

Connecting ADO, CI/CD, and Monitoring Systems

Collecting these metricsperiodically and optimizing them help Organizations achieve efficiency in the way they releasesoftware.

Having effective tools, plugins and integrating them helps automate most of these tasks and generation of metrics and taking appropriate decisions.

Azure DevOps(ADO) provides required metadata for capturing the raw data for calculating these metrics. DevOps team needs careful evaluation considering the futureBusiness needs.

Visualization Tools and Dashboards

Azure DevOps(ADO) provides dashboardcapabilities with these metadata and widgets to capture some of these metrics. However, ensuring that metadata is updated for every work item is essential to generate accurate metrics.

Alternative tools like Power BI, Grafana can be employed for enhanced visualization and reports for stakeholders.

Challenges and Considerations for Organizations

Data Collection & Usage of Tools

Data Integrity and Tool Limitations

Challenges:

  • Inconsistent Data Sources: DevSecOps pipelines pull data from disparate tools (CI/CD systems, Scanners, incident trackers), leading to mismatched timelines, false positives, or gaps.
  • Manual Reporting Bias: Human-curated metrics (e.g., incident reports) often lack standardization, skewing trends like MTTR (Mean Time to Recover).

Solution:

  • Enforce Traceability: Use immutable metadata (e.g., Azure DevOps’ pipeline IDs, Git commits) to extract data related to code changes.
  • Validate with Cross-Tool Correlation: Combine data from SonarQube (code quality), ADO to flag anomalies, Industry standards to validate metric data
Availability of Tools & Plugins

Challenge:

  • Organizations may lack access to integrated DevSecOps tools (e.g., SAST/DAST scanners, deployment trackers) or struggle with licensing costs.

Solution:

  • Leverage Azure DevOps Marketplace for plugins (e.g., OWASP ZAP, SonarQube).
  • Use open-source alternatives where budgets are constrained.
Configuring ADO for Capturing Metadata

Challenge:

  • ADO’s raw data (builds, releases, work items) requires proper tagging/linking to generate metrics like Lead Time or Deployment Frequency.

Solution:

  • Enforce mandatory fields in work items and periodic updates
  • Use ADO Analytics Views or Power BI connectors to query pipeline metadata.
Effective Updates on Work Item State Changes

Challenge:

  • Manual state updates (e.g., Done → Approved) delay metrics like Cycle Time.

Solution:

  • Automate transitions using ADO webhooks or Azure Functions (e.g., auto-update when PRs merge).
  • Map states to DevSecOps gates (e.g., Security Review state before deployment).
Insufficient Data to Generate Metrics

Challenge:

  • Sparse data (e.g., infrequent deployments, low count of incidents per project) skewing trends.

Solution:

  • Define minimum thresholds (e.g., “Track only if >5 incidents of software bug category”).
Metrics That Are Inconclusive for Decisions

Challenge:

  • Vanity metrics (e.g., “MTTR of 120 days”) don’t drive action.

Solution:

  • Focus on outcome-oriented metrics backed by sufficient data

Organizational Resistance and Misuse

Organizational Resistance

Challenges:

  • Fear of Exposure: Teams may resist metrics that highlight inefficiencies (e.g., high Change Fail Rate due to security rejections).
  • Tool Overload: Introducing new DevSecOps tools, metadata to be captured & plugins (e.g., SAST scanners) can trigger pushback if perceived as disruptive/unnecessary.
  • Misaligned Incentives: Leadership rewarding one metric over other. "Lead Time" alone may misguide

Mitigation Strategies:

  • Frame Metrics as Improvement Tools:
    • Emphasize on how these metrics help Organization in the long run and to compare or measure performance
  • Pilot Metrics:
    • Start with non-critical projects to demonstrate value before org-wide rollout and focus on metrics that offer Business value.

Organization long-term goals Vs Metric derived Results

Challenge:

  • Misalignment: Short-term metric optimization (e.g., improving Deployment Frequency) may conflict with long-term goals (e.g., tech debt reduction, compliance maturity).
  • Vanity Metrics vs. Value Delivery: Teams may prioritize metrics that look good over meaningful outcomes.

Solution:

Cascade Metrics to Strategic Themes:

  • Map metrics (e.g., Lead Time) to business objectives (e.g., "Faster time-to-market for compliance-critical products").

Challenge:

Metric Trap while defining roadmap: Over-focusing on one metric (e.g., MTTR) may ignore systemic issues (e.g., inadequate test automation).

Solution:

Weighted Metric Portfolios:

  • Balance the metrics with Organization’s long-term goals (e.g., Deployment Frequency), stability (Change Fail Rate), over Lead time

Balancing Metrics with Team Culture

The Cultural Risks of Over-Metrics

Challenges:

  • Fear of Measurement: Teams may perceive metrics as surveillance, leading to stress and burnout.
  • Gaming the System: Pressure to hit targets (e.g., Deployment Frequency) can incentivize shortcuts (e.g., skipping security scans).
  • Innovation Suffocation: Excessive focus on metrics may discourage experimentation (e.g., fear of failing deployments affecting Change Fail Rate).

Solution:

  • Psychological Safety First:
    • Emphasize that metrics are diagnostic tools, not performance evaluations.
    • Celebrate "learning from failures" (e.g., post-incident reviews that improve MTTR).
  • Team-Led Metric Design:
    • Involve engineers in selecting metrics (e.g., let them choose between Lead Time or Cycle Time as their priority over Change Failure Rate).

Metrics Vs Team Capability & Training Needs

Capability Gaps Blocking Metrics

Challenge:

Teams lack skills to improve key metrics (e.g., slow Lead Time due to unfamiliarity with security tools)

Solutions:

  • Skill-Based Metric Segmentation:
    • Example: Track Lead Time separately for teams adopting new SAST tools
  • Just-in-Time Training:
    • Automate training triggers (e.g., if Failed Pipeline Rate Due to Security >15%, assign modular training).
  • Mentorship Metrics:
    • Measure % of Pair Programming Sessions to boost knowledge sharing.
Metrics Ignoring Team Growth

Challenge:

  • Overemphasis on output metrics (e.g., Deployment Frequency) undervalues capability building.

Solutions:

  • Balance Output and Growth Metrics:
    • Pair Deployment Frequency with % of Team Contributing to Pipeline Code.
  • Career-Path Metrics:
    • Example: Track % of Engineers Leading Security Reviews to encourage ownership.

References

  • Accelerate by Forsgren, Humble & Kim
  • Azure DevOps Docs
  • DevOps Research and Assessment (DORA) Reports

Share This Page