Coinbase QA Engineer System Design Questions

Coinbase's testing framework needs to identify flaky tests that may produce inconsistent results. Building such a detection mechanism is integral for ensuring high test reliability.
Problem statement: Design a class called FlakyTestDetector that will track test runs. It must be able to record test run results, particularly monitoring those tests that have varied results across different runs. The class should also provide a method to check if a test is considered flaky after a certain threshold of inconsistency.
- Class Specification:
- Method Signature: def record_run(self, test_case: str, result: bool) -> None
- Method Signature: def is_flaky(self, test_case: str, threshold: int) -> bool
- Method Signature: def get_flaky_tests(self) -> List[str]
Example 1:
Input: detector.record_run('TestCase1', True)
Output: None
Input: detector.record_run('TestCase1', False)
Output: None
Input: detector.is_flaky('TestCase1', 1)
Output: True
Example 2:
Input: detector.record_run('TestCase2', True)
Output: None
Input: detector.get_flaky_tests()
Output: ['TestCase1']Constraints:
- 1 <= test_case.length <= 100
- 1 <= len(runs) <= 1000

As Coinbase implements various test scenarios for its functionalities, aggregating test results is essential for efficient reporting and analysis.
Problem statement: Design a TestResultAggregator class that handles the storage and computation of results from multiple test cases run on their framework. This class must have the ability to add test results, get all results, and compute the percentage of passed tests.
- Class Specification:
- Method Signature: def add_result(self, test_case: str, passed: bool) -> None
- Method Signature: def get_results(self) -> List[Tuple[str, bool]]
- Method Signature: def get_pass_percentage(self) -> float
Example 1:
Input: aggregator.add_result('TestCase1', True)
Output: None
Input: aggregator.get_results()
Output: [('TestCase1', True)]
Input: aggregator.get_pass_percentage()
Output: 100.0
Example 2:
Input: aggregator.add_result('TestCase2', False)
Output: None
Input: aggregator.get_pass_percentage()
Output: 50.0Constraints:
- 1 <= test_case.length <= 100
- 1 <= len(results) <= 1000