OpenAI QA Engineer System Design Questions

Flaky tests can compromise the integrity of continuous integration pipelines, especially during deployments at OpenAI. It is crucial to detect and manage them effectively.
Problem statement: Implement a system that monitors test results over time and identifies which tests are flaky based on a threshold of consecutive failures or inconsistent results.
- class FlakyTestDetector:
- def __init__(self, threshold: int): initializes the detector with a failure threshold.
- def add_result(self, test_name: str, result: bool) -> None: saves a test result.
- def is_flaky(self, test_name: str) -> bool: checks if a test is flaky based on recorded results.Example 1:
Input: detector = FlakyTestDetector(threshold=2)
detector.add_result('testA', True)
detector.add_result('testA', False)
detector.add_result('testA', False)
detector.is_flaky('testA')
Output: True
Explanation: 'testA' has failed twice consecutively.Constraints:
- 1 <= threshold <= 10
- 1 <= len(test_results) <= 1000

Automated testing of AI models creates a range of results that require analysis and aggregation to improve deployment strategies across OpenAI models.
Problem statement: Design a system that can collect and aggregate multiple test results, identifying patterns and generating metrics (like success rates).
- class TestResultAggregator:
- def __init__(self): initializes the aggregator.
- def add_result(self, model_name: str, result: bool) -> None: adds a test result for a specified model.
- def get_success_rate(self, model_name: str) -> float: returns the success rate of tests for a specified model.
- def get_overall_success_rate(self) -> float: returns the overall success rate across all models.Example 1:
Input: aggregator = TestResultAggregator()
aggregator.add_result('gpt-3', True)
aggregator.add_result('gpt-3', False)
aggregator.get_success_rate('gpt-3')
Output: 0.5
Explanation: One success and one failure for gpt-3.Constraints:
- 1 <= len(results) <= 1000
- models are unique strings