IMPACT OF NON-FUNCTIONAL REQUIREMENTS ON SCALABLE SOFTWARE ARCHITECTURE USING MACHINE LEARNING

Abstract

Today's cloud computing systems meet billions of tasks daily and scalable software architecture is a critical factor. Non-functional requirements (NFRs) are requirements that are concerned with the quality of a system in the sense of attributes other than its functional behavior, such as performance, reliability, availability, security and maintainability requirements. The requirements play a significant role in the architectural decisions, but they are often not taken into account during the design process, and thus systems are not effective at load conditions. The Google Cluster Workload Traces dataset provides granular and realistic usage data of resources, task-scheduling and execution of jobs on large, distributed infrastructures, which is now an unprecedented opportunity to study how NFRs are manifested in real-world environments. Although critical, nonfunctional requirements are often hard to quantify and are not clearly analyzed as part of the architectural design process. Architects lack data driven insights and have to rely on heuristics and experience to come up with architectures that are over-provisioned, under-resilient and poorly scalable. There is a notable lack of empirical measurement in the literature of the effect of specific

NFRs on the scalable architectures with realistic production workload data. The number of distributed system failures is more than 60% due to NFRs as indicated by both Bass et al. (2012) and ISO/IEC 25010 standard. The published cluster trace data (2019 version) provided by Google, contains over 2.6 billion events for running tasks over 8 weeks of data, and highlights directly the impact that Workload Patterns, failure counts and resource contention have on the architectural scalability under various NFR constraints. Machine Learning techniques like Random Forest, Gradient Boosting (XGBoost) and LSTM neural networks are used for analyzing the Google Cluster Workload Traces 2019 data. The NFR related signals featured are: task failure rates (reliability), scheduling latency (performance), resource utilization variance (efficiency) and eviction rates (availability). The structures of workloads are identified using clustering algorithms (such as K-Means, DBSCAN) and the relationship between the violation of NFRs and the measure of architectural scalability is measured by regression and classification models. Advanced tools: Python, TensorFlow, scikit-learn, Apache Spark (distributed data processing), and BigQuery (dataset query). This research is expected to produce a scalable NFR impact framework of data, which will enable the specific NFRs to be correlated with measurable results with respect to scalability. The trained ML models will forecast bottlenecks in the proposed architecture before it is deployed to monitor the issues without additional re-design. The findings will provide empirical evidence that NFRs that affect the performance and reliability have the highest architectural impact, a re-usable method to analyse any large scale cluster workload data and guidelines for designing scalable architecture with NFRs in cloud environments.