Abstract
This article describes how a near-impossible project was completed with the help of a modern development tool for the HPE NonStop in terms of time, cost, and quality. It also illustrates how abat+ was able to contribute to the completion of the third largest private construction project in Germany.
Introduction abat+
abat+ is a software company located in Germany that, among other things, is specialized in the development and operation of the PLUS MES on the HPE NonStop and SAP platforms. PLUS is a high availability solution for discrete manufacturing, which controls and optimizes the processes of automobile production as well as other complex variant production. The company also offers system analysis, consulting, development, and 24/7 support, with vast experience for applications on the HPE NonStop.
Problem
At the beginning of 2018, our customer from the automotive industry approached us with the requirement to expand a central identifier, which describes a location or piece of equipment within a factory. Specifically, we needed to change a 4-digit numerical value into a 12-digit alphanumeric, such that the identifier is globally unique. This was a necessity to allow “Factory 56 – The plant of the future” in Sindelfingen to start operating in July 2019. While the change sounds very simple, this was quite far removed from reality.
The affected code base of the MES comprised over 13 M lines of source code spanning the COBOL backend as well as the associated .NET and WinForms clients. The project had a deadline of 18 months and, according to our initial estimate, implementing the changes across the entire source code and database would require an effort of 15 000 workdays. Additionally, the budget available to the project only covered half of the estimate.
To complicate things further, our MES is deployed worldwide in 13 automotive production plants. All other development projects had to continue without disruptions since the launch of the entire series as well as model changes depended upon their completion. This constituted an annual project volume of roughly 20 000 workdays. Meeting the customer requirements was almost impossible. The following problems had to be solved:
- The scope was not clear – Identifying the appropriate code sections which needed to be adjusted required the analysis of the entire MES source code (13 M LOC).
- Limited resources – Due to the limited budget, we were not able to expand our FTE beyond 23 additional staff. There were only three experienced architects available for the project since all other projects had to continue without disruptions. Additionally, there was very little know-how for NonStop and COBOL development on the market, which made even these limited personnel difficult to find.
- Deadline – The project duration was simply too short for the required effort.
- Quality – With the limited NonStop and COBOL know-how on the market, we had to make use of many inexperienced developers, which made quality assurance of the converted software even more essential.
Problem Solution
1. Scope Definition
In order to define the scope, we used the tool ITP Panorama from ITP Software Systeme GmbH. This piece of software made it possible to load the entire COBOL source code into a so-called hypercube. This allowed us to perform the required analysis for the project, which yielded the following result:
- 1678 servers
- 7651 transactions
- 324 modules
- 406 tables
In addition to defining the scope, the analysis result not only provided the quantity structure, but also the insight that the global variable occurred in over 600 variations within the MES and in external interfaces. This further increased the complexity of the project.
2. Insufficient Personnel
We were able to add 20 contractors for the project duration through external service providers. Many of the colleagues were inexperienced in COBOL development and in handling the HPE NonStop. Therefore, it was essential to make the learning curve as steep as possible, allowing the developers to complete their work very easily. Normally the onboarding process for a NonStop COBOL developer within our company requires roughly 6 months, which was not possible due to the time constraint of this project.
3. Project Complexity
To handle complexity of the project, we decided on an agile software development approach, with a focus on short feedback cycles and adjustments to the procedure based on the principle “Inspect and Adapt”. The team was divided in such a way that each experienced developer was complemented by several inexperienced colleagues to ensure short communication cycles and optimized knowledge transfer.
We set up a structured backlog of the MES, which was prioritized according to functional modules and technical requirements. The identified blocks were rolled out over several releases. In this way we were able to guarantee a smooth transition of the changed software when compared to a single massive release. For example, views were created for each existing database table, which provided the rest of the system with a fixed interface. This ensured the downward compatibility of the software despite the changes in the background.
4. Excessive Scope
Manually implementing the changes across the entire MES source code would have necessitated either a much larger development team for the defined deadline or a shift of the project completion date. This required us to think of an alternative solution to save time during development and lead to a core outcome of the project – optimization and automation of the existing NonStop development process.
Development Process Analysis
Up until this point, software development on the HPE NonStop took place using tools such as TEDIT or a text editor. There was no central source code management, which could lead to competing file access and inconsistencies.
Various commercial solutions were evaluated to solve this problem. None of the solutions available on the market were suitable for our project, due to our specific need for a DDL compiler, constraints such as the missing source code management, and synchronization difficulties due to the sheer volume of continuous development of the MES.
A tailor-made solution was required for this project that solves the aforementioned problems and guides the developer through the entire NonStop development process. The goal was to build a tool that exactly covered our requirements.
We completed several measurements to make a valid statement about the time needed to change a single transaction within a function block. This yielded a development time of ~3.5 hours per transaction. Each transaction required 15 steps for development to be completed.
We analyzed for which steps a person was necessary and which steps showed automation potential. The result was that 11 of the 15 identified work steps could be greatly automated. These were primarily activities such as copying files, compiling, and documenting for release management.
Requirement Analysis
The eddy+ Development Team
A workshop was held with a subset of colleagues, where a user story map was created, which lists the work requirements to complete their tasks quickly and with high quality. These requirements were jointly prioritized and implemented by a small agile development team which supplemented the main project. The main features included:
- Guided workflow for converting the function modules
- Automated criticality and complexity evaluation of the fragments
- Automated replacement of code fragments
- Visualization of the automatically changed source snippets
- Automation of manual work steps such as copying, compile commands, etc.
Thanks to an agile development methodology, we were able to release software increments in bi-weekly cycles and quickly react to additional requests and user feedback. The incremental development of our custom-tailored tool continued until all requirements were covered, which achieved a mean conversion time of 1 hour per transaction.
Test Automation
Since our MES is a central and critical component of production, every minute of system downtime would cause significant losses. Therefore, it was essential to focus on zero fault tolerance quality assurance. We had to guarantee each function block performed exactly as before the changes.
To guarantee the highest possible quality, we integrated the ability of defining and executing unit tests for every individual transaction of the COBOL backend.
An algorithm determined the criticality of the relevant code sections with pre-defined test strategies depending on their classification. Test telegrams were created during the source code adaptations, which could be executed for both the original and the newly compiled code objects. As part of the testing methodology, the database changes, as well as the runtime outputs, were automatically compared between the original and altered code objects. If the results of both tests were identical, the processed function module could be completed and scheduled for release. With this approach, we were able to keep the quality of our software extremely high, despite relying on many developers inexperienced in NonStop systems.
Result
After the overwhelmingly positive feedback from our customer and HPE, as well as the wish of all involved developers to continue using a modern development environment, we derived a generalized web-based solution for the HPE NonStop – eddy+.
eddy+ Source Version Comparison
What are the advantages of eddy+?
- Easy to use
We created a new platform that allows any developer to communicate with the NonStop, without needing to know its language or specificities. Just like other modern environments, the developer only needs to focus on WHAT rather than HOW. - Designed as an expandable and modular development environment
Almost every company using a NonStop today, has its own individual macros and custom architecture. As opposed to other solutions, eddy+ offers the possibility to implement this specific logic in encapsulated modules. The core functions remain unchanged, but the logical modules can be replaced depending on the project. - Web-based scalable solution
It does NOT need to be configured for individual people but is done only once! In other solutions, each workstation must be set up, which is often complex, and requires several days to accomplish. - Supports agile development
eddy+ supports development teams with modern practices such as Scrum or Kanban. Prior to eddy+, the use of manual processes and local development was prone to errors. Our solution helps to eliminate them. The NonStop remains master and there is no local storage or development, as such, there is no risk of competing file-access. This makes coordination within and between multiple teams near effortless. - Lastly, it’s designed for HPE
Compared to other auxiliary solutions for HPE, eddy+ uses the native NonStop functions. This means they are not duplicated, but the interaction between the developer and the machine is made intuitively available. With eddy+, there is no need to use a cross-compiler, which means it is deployable for any project, regardless of legacy or new development.
Conclusion
We were able to solve the problems posed by our customer by creating a modern development tool and through the expansive experience of our employees. Therefore, we saved valuable time during code creation, significantly improved development quality by reducing manual steps, and allowed for efficient onboarding by providing an intuitive development experience. Finally, we were able to easily scale the development teams, which allowed us to complete the project on time, in budget, and with extremely high quality.
After the project development finished, the changed software was rolled out globally, with production facilities in China, the United States, Brazil, Germany among many others. In sum 13 plants with over 40 pathways received the changed software without any production downtimes. This was an incredible achievement since we changed a central identifier in the MES, and proof of the high quality that we achieved using our development tool! No end-user noticed the massive change within the software! From their perspective, the system behaved identically to previous versions.
The project resulted in the core of eddy+, which is now available to everyone!
Company Experience
Beginning with the DACOS GmbH, through the takeover by SAP SE, to the founding of abat+, we have amassed over 30 years of software development experience on the HPE NonStop. We managed to complete a project which seemed impossible. If you find yourself in similar circumstances, we will help you find an ideal problem solution. Our full-stack development teams help you in every step of the software life cycle, from personalized consulting all the way through custom solution implementation and support.
Be the first to comment