Cost of quality or lack thereof in software development

A 2022 report by CISQ estimated that the cost of poor-quality software in the US has risen to at least $2.41 trillion, though not in the same proportions as in 2020. The accumulated technical debt (TD) of software has increased to about $1.52 trillion. Those numbers started to grow much quicker with the industry-wide adoption of AI coding assistants.

Researchers at Bilkent University revealed in a 2023 study that AI assistants generated code with a 30.5% error rate, and another 23.2% of the code was only partially correct. The accuracy rates differed across various AI code generators.

So, how to balance the cost of quality?

Definition of Cost of Quality (CoQ)

The Cost of Quality (CoQ) in software development encompasses all costs associated with ensuring that software products meet good quality standards. These costs are divided into two main categories: the costs of conformance and the costs of non-conformance.

Components of Cost of Quality

Conformance costs include:

1. Prevention costs:
   • Training and education
   • Process improvement (Here's how quality should be measured)
   • Tools and technologies
   • Supplier capability evaluations
   • Preventive maintenance
   • Planning and documentation
   • Security training and awareness programs
   • Implementation of security controls and technologies
2. Appraisal costs:
   • Inspection and testing of incoming materials
   • In-process and final inspection/testing
   • Product audits
   • Calibration of testing and inspection equipment
   • Maintenance of test equipment
   • Quality audits
   • Security audits and assessments
   • Penetration testing and vulnerability assessments

Non-conformance costs cover:

1. Internal failure costs:
   • Rework
   • Re-inspection and re-testing
   • Debugging
   • Downtime caused by quality issues
   • Incident response and recovery efforts for security breaches
   • Investigation and remediation of internal security issues
2. External failure costs:
   • Customer complaints and returns processing
   • Warranty claims
   • Product recalls
   • Liability costs
   • Loss of reputation
   • Lost sales
   • Customer support
   • Legal fees and settlements for security breaches
   • Regulatory fines and penalties for security non-compliance
   • Costs associated with bringing products or services into compliance
   • Increased scrutiny and audits from regulatory bodies
   • Loss of certifications or licenses
   • Reputation management

IBM's System Science Institute study has shown that the cost of fixing defects multiplies in the later stages of software development life cycle. Fixing a bug during implementation is six times more expensive than fixing it during design. If the bug is found during testing, it costs fifteen times more to fix. Once the software is in production, fixing a bug can be a hundred times more expensive than if it was caught during the design phase. By comprehensively analysing CoQ, you can identify where to invest in good quality improvements to minimise overall costs.

Don't take this lightly! Orca Security's State of Cloud Security Report 2024 claims that 62% of organisations have severe vulnerabilities in code repositories.

How to calculate quality costs?

Here's the comprehensive formula for calculating the Cost of Quality (COQ). It might seem like a lot, but it will save you time and money in the long run.

Prevention Costs (PC)

• Training

• Process improvements

Appraisal Costs (AC)

- Requirements review

- Code review

- Design review

- Testing

- Inspection and testing of incoming materials

- Product audits

- Calibration of testing and inspection equipment

- Maintenance of test equipment

- Quality audits

- Security audits and assessments

- Penetration testing and vulnerability assessments

Internal Failure Costs (IFC)

- Requirements re-do

- Coding re-do

- Re-inspection and re-testing

- Debugging

- Downtime caused by quality issues

- Failure analysis

- Repair and maintenance

- Design changes

- Quality control administration

External Failure Costs (EFC)

- Customer complaints and returns processing:

- Warranty claims cost

- Product recalls cost

- Liability costs

- Loss of reputation - Reputation management costs

- Lost sales cost

- Customer support:

- Regulatory fines and penalties

- Legal fees and settlements

Example calculation for cost of quality

Let's assume the following for a software development project with different hourly rates for different roles:

• Training: 50 hours at $60/hour
• Requirements review: 40 hours at $70/hour
• Requirements re-do: 30 hours at $80/hour
• Code review: 60 hours at $75/hour
• Coding re-do: 20 hours at $85/hour
• Design review: 30 hours at $70/hour
• Testing: 100 hours at $65/hour
• External failure expenses (assumed): $5,000

Prevention Costs (PC)

PC = Training costs

PC = 50 x $60 = $3000

Appraisal Costs (AC)

AC = Requirements review cost + Code review costs + Design review + Testing costs

AC = (40 x $70) + (60 x $75) + (30 x $70) + (100 x $65) = $15,900

Internal Failure Costs (IFC)

IFC = Requirements re-do costs + Coding re-do costs + Re-inspection costs +Debugging costs

IFC = (30 x $80) + (20 x $85) = $4,100

External Failure Costs (EFC)

(Assumed as a fixed cost)

EFC = $5,000

Total Cost of Quality

COQ = PC + AC + IFC + EFC

COQ = $3,000 + $15,900 + $4,100 + $5,000 = $28,000

Cost of quality for software projects unlocks the ability to assess intriguing insights, such as:

• Measuring how much of your total development budget is allocated to quality assurance.
• Determining the percentage of your development costs that stem from addressing failures.
• Understanding the financial impact of quality-related expenses on your overall sales and maintenance efforts.

This information will help you balance your investments while preventing poor quality.

FAQ Cost of Quality

1. What are the prevention costs in software development?

Prevention costs in cost of quality include a variety of crucial actions.

Training and education

Continuous developer training ensures that developers remain proficient in the latest technologies, frameworks, and languages. Ongoing training helps in ingraining best practices within the development team. Training is paramount – Orca Security's State of Cloud Security Report 2024 showed that 91% of organisations have 10+ year-old vulnerabilities and 46% of them have 20+ year-old vulnerabilities (old enough to be in college!). Without proper training programs, those vulnerabilities will die with your product.

Training in modern development methodologies promotes iterative development, which allows teams to respond swiftly to changing requirements and early detection of defects. F.e. TDD involves writing tests before the actual code, ensuring that each unit of code is tested for correctness from the outset. It also encourages a modular code structure, which simplifies maintenance and scalability making it more cost-effective in the future.

With the advent of AI coding assistants, maintaining control over software quality requires a blend of traditional practices and modern approaches. Ongoing training ensures that developers can critically assess the suggestions made by AI tools, preventing the incorporation of suboptimal or insecure code.

Process improvement

Implementing Robust Development Processes

Robust development processes are the foundation of high-quality software. This involves adopting best practices such as continuous integration (CI), continuous deployment (CD), and comprehensive code reviews. Continuous integration tools like Jenkins and CircleCI automate the process of integrating code changes, running tests, and detecting issues early. The state of CI/CD security is scary. According to Orca Security, 70% of organisations have unencrypted secrets in code repositories.

Peer reviews ensure that multiple eyes scrutinise the code, catching issues that a single developer might miss. Implementing structured code review tools like Gerrit or Crucible can further enhance this process.

Tools and Technologies that Aid in Prevention

Investing in advanced tools and technologies is essential for defect prevention. Static analysis tools such as SonarQube and Coverity scan the source code for potential errors and security vulnerabilities. These tools can detect complex issues like memory leaks, buffer overflows, and SQL injection vulnerabilities, which are often missed during manual reviews.

Code review platforms such as Gerrit or Crucible can be enhanced with AI to flag potential issues, but human oversight remains critical. Ensuring that senior developers review AI-generated code helps in catching nuanced issues that automated tools might overlook.

Automated testing frameworks like Selenium for UI testing and JUnit for unit testing are vital. These frameworks ensure that tests are consistently executed, providing immediate feedback on code changes. Integrating these tools into CI/CD pipelines ensures that tests are run continuously, maintaining high code quality.

Planning and documentation

Comprehensive Requirement Gathering

Engaging stakeholders through detailed user stories, use cases, and acceptance criteria is vital. Techniques such as Joint Application Development (JAD) sessions or workshops can facilitate comprehensive requirement gathering, ensuring that all aspects of the user needs are captured accurately.

Detailed and Maintainable Documentation Practices

Maintaining detailed and up-to-date documentation is essential for preventing defects. Documentation should cover all aspects of the software, including architecture, design patterns, APIs, and user manuals. Practices such as writing comprehensive API documentation using tools like Swagger or Postman can significantly enhance understandability and usability.

By implementing a documentation-as-code approach, where documentation is treated with the same rigor as code, you can ensure accuracy and completeness.

2. What are appraisal costs in software development?

Appraisal costs in software development refer to the expenses associated with evaluating and ensuring the quality of the software product. These costs are incurred to identify and correct defects in the software before it is delivered to the customer or deployed in a production environment. The goal of incurring these costs is to prevent the release of defective software, which can lead to higher costs related to correction and customer dissatisfaction if issues are discovered later.

Here is the appraisal cost structure:

Testing:

• Unit Testing: Testing individual components or modules for correctness.
• Integration Testing: Testing the interaction between integrated modules.
• System Testing: Testing the complete and integrated software product.
• Acceptance Testing: Testing the software in real-world scenarios to ensure it meets the user's needs.

Code Reviews:

• Peer reviews and inspections to detect defects in code, design, or documentation.

Quality Audits:

• Formal evaluations of the software development processes and products to ensure they meet defined standards and procedures.

Tooling:

• Tools and software used for automated testing, static code analysis, and other quality assurance activities.

Prototyping and Simulations:

• Creating prototypes or simulations to evaluate specific features or performance aspects of the software.

User Feedback:

• Costs associated with gathering and analysing feedback from beta testing or early access programs to identify defects and areas for improvement.

Compliance and Certification:

• Ensuring the software complies with industry standards and obtaining necessary certifications.

Appraisal costs are an important part of the overall cost of quality (CoQ) in software development, which also includes prevention costs (activities to prevent defects), internal failure costs (costs incurred to fix defects found before delivery), and external failure costs (costs incurred due to defects found after delivery). By investing in appraisal activities, organisations can reduce the risk and cost of defects slipping through to later stages of the software lifecycle.

3. What are external failure costs?

External failure costs are the expenses incurred when defects in products or services are detected after they have been delivered to customers. These costs can have significant financial and reputational impacts on an organisation. Here is a comprehensive list of expenses:

External Failure Costs

1. Customer Complaints and Returns Processing
   • Handling customer complaints and inquiries about defective products or services.
   • Processing and managing product returns and exchanges.
2. Warranty Claims
   • Costs associated with fulfilling warranty obligations, including repair or replacement of defective products.
   • Administrative costs related to managing warranty claims.
3. Product Recalls
   • Costs involved in recalling defective products from the market.
   • Transportation, handling, and disposal of recalled products.
4. Liability Costs
   • Legal fees and settlements arising from lawsuits related to defective products or services.
   • Compensation paid to customers for damages or injuries caused by product defects.
5. Loss of Reputation
   • Impact on brand image and customer trust due to poor product quality.
   • Negative publicity and its effect on future sales and market position.
6. Lost Sales
   • Loss of current and future sales due to dissatisfied customers switching to competitors.
   • Decrease in market share as a result of diminished customer loyalty.
7. Customer Support
   • Increased demand for customer support services to address issues related to defective products.
   • Additional staffing and resource allocation for customer service operations.
8. Regulatory Fines and Penalties
   • Fines imposed by regulatory bodies for failing to meet product safety or quality standards.
   • Penalties for non-compliance with industry regulations and standards.
9. Legal Fees and Settlements
   • Costs associated with defending against lawsuits from customers or regulatory bodies.
   • Settlements paid to resolve legal disputes related to product defects or non-compliance.
10. Reputation Management
    • Costs of public relations efforts to mitigate negative publicity and restore brand image.
    • Marketing and advertising expenses aimed at rebuilding customer trust and loyalty.

External failures are going to happen, but they might be significant or barely noticeable depending on your initial conformance cost and product quality. External failure costs can be substantial and have long-lasting effects on a company's financial health and market position. Proactively addressing quality issues and ensuring compliance with regulations can help minimise these costs.

4. What are internal failure costs?

Internal failure expenses are expenses incurred when defects are identified and rectified before the products or services reach the customer. These costs are associated with failures detected during the production process and are crucial for maintaining quality control. Here is a comprehensive list of internal failure expenses:

Internal Failure Costs

1. Rework
   • Labor and materials used to correct defects in products or services.
   • Additional processing and testing to ensure the reworked items meet quality standards.
2. Re-inspection and Re-testing
   • Costs associated with inspecting and testing products or services again after rework or repair.
   • Additional quality assurance measures to verify that defects have been corrected.
3. Debugging
   • Costs of identifying and fixing bugs or defects in software or systems during development.
4. Downtime Caused by Quality Issues
   • Production stoppages or slowdowns resulting from quality-related problems.
   • Lost productivity and associated labor costs during downtime.
5. Failure Analysis
   • Costs of investigating and analysing the root causes of defects.
   • Use of specialised equipment and personnel to diagnose problems.
6. Repair and Maintenance
   • Expenses related to maintaining and repairing equipment that failed due to quality issues.
   • Preventive maintenance actions are taken to avoid future failures.
7. Design Changes
   • Costs associated with modifying product or service designs to eliminate defects.
   • Engineering and development efforts to implement design improvements.
8. Quality Control Administration
   • Administrative costs of managing internal quality failures, including documentation and reporting.
   • Meetings and communication related to addressing internal failures.
9. Training for Rework
   • Costs of additional training for employees to improve skills and reduce defects.
   • Specialised training focused on areas where defects frequently occur.
10. Excess Inventory
    • Holding costs for inventory that must be stored longer due to quality issues.
    • Costs of managing and accounting for excess or obsolete inventory resulting from defects.

Minimising internal failure expenses is essential for maintaining efficient operations and delivering high-quality products or services. Effective quality management practices, continuous improvement efforts, and proactive defect prevention strategies can help reduce these expenses.

5. What are the biggest challenges in achieving good quality?

Based on Katalon's State of Software Quality Report 2024 findings, here are the top challenges in achieving good quality in software development:

• Lack of time and skilled resources
• Difficulties in applying test automation
• Frequent requirement changes
• A shortage of skilled resources

Key Challenges in Achieving Software Quality Goals

1. Lack of Time: The most frequently cited challenge is the lack of adequate time to thoroughly develop, test, and refine software.
2. Skilled Resources: The availability of experienced and knowledgeable staff is crucial for maintaining high quality in software development.
3. Test Automation: Issues include the complexity of automation tools, maintenance of automated tests, and integration of automation into the development workflow.
4. Evolving Requirements: Constantly changing requirements can disrupt the development process, making it difficult to ensure that the final product meets quality standards.