Performance Management: A Personal History, And Turning A New Page
April 10, 2023 Doug Mewmaw
The term “performance management” is broad, comprehensive, and uniquely personal depending upon the system, situation, and experience of the user defining it. I stumbled into the world of performance management early into my career, without even knowing it. As I reflect back on my career, it is also natural to reflect upon the history of performance management.
I thought this reflection might be interesting to not only the old pros out there but also the new guys just coming into the industry. I have been fortunate to be on an interesting journey, and one that is about to change not only my career, but will re-define the practice of performance management in the IBM i market.
My Early Years Of Performance Management – The 1980s
The mainframe was the first system I worked on professionally. I was a young systems programmer and watched my computer elders with awe. Since I was new in my career, I ate up all information I could get my hands on. I quickly learned the language of performance management.
I learned about Million Instructions per Second, or MIPS, a number used for capacity planning to this day, I learned about Customer Information Control System (CICS), also still used today. I also learned CICS applications were mission critical as key company business operations were performed daily and monthly and are measured in this way.
In the summer of 1984, I learned performance tuning and capacity planning on the mainframe. I don’t remember seeing performance management reporting. This facet of performance management is still to come. I do remember my peers constantly asking the end users how their computer experience felt. This became a key indicator to me for how performance is measured in a non-data centered way. If the end user is frustrated by slow processes, or disfunction, the experience feels terrible, mandating improvements. But where to begin looking? This is what performance management used to look like. There was no easy roadmap for finding the problems in a system.
In 1988, I was asked to lead the systems team as the AS/400 platform was just introduced and that culminated in the IBM i platform on Power Systems that we know today. I attended a systems class where the instructor informed the class that systems programmers were no longer needed with this innovative platform. Two hours into class, all terminals went blank. I watched the instructor struggle for an hour as he troubleshot the culprit. As a systems programmer myself, I couldn’t stop smiling as he talked to the systems support team for help. The need for a performance management tool was evident to me.
That year, I learned CL, work management, and all the procedures for successfully running an AS/400 computer system, which were all still in their infancy.
Lesson Learned: There is always a need for a systems support team. Capacity planning and performance management are complex. The end-user experience matters.
The Middle Years Of Performance Management – The 1990s
During this period, I added IBM’s collection services functionality to my toolkit. This concept, which IBM performance experts discussed for years, was created so that system programmers could see how the AS/400 was always performing. I made queries and created monthly performance graphs. It helped the team understand the level at which the expensive new hardware was operating. This was the first step in an official performance management methodology.
Turn the clock ahead to May 1992. At this point in my career, I left the systems team and moved over to the application development side of the house. During this period, I learned the difference between good and bad code. More importantly, I learned how application rollouts affected the system, both positively and negatively. In terms of understanding performance management, this career move was one of the best I ever made.
My next career move was as the systems manager for all platforms. It was the job I coveted for years. It was during this period that software companies started developing performance management tools. These companies utilized IBM’s performance files and created colorful graphs to show core metrics performance. The industry was finally shown a big picture of performance. During this period, I developed a performance report that correlated system performance (CPU, disk, and memory) with changes in the system – answering the question: What is the impact of adding more memory? This effective performance management question began to shape my vision for what I felt was the logical next phase for state-of-the-art performance management. The concept, “Understanding the impact of Change,” was an effective troubleshooting technique and one I’ve been teaching for years. Even though this was a very tedious process to manage, the feedback I received from peers was that this was a very positive addition to the performance management toolkit.
Lesson Learned: Don’t be afraid to get out of your comfort zone. Learning all facets of performance is critical for becoming an experienced system programmer. Measuring all performance components is essential to understanding the entire system. The systems and the applications should be measured daily, weekly, and monthly.
The Middle Years Of Performance Management – Early 2000s
In the early 2000s, companies went from a process improvement culture to a risk avoidance culture. This culture change made performance management more difficult as employees were hands off when it came to tuning a poorly running system.
During this period, companies, including MSPs, were purchasing performance management tools and providing core metric graphs throughout the industry. During many engagements, I witnessed presentations where the presenter had no idea of what they were talking about. One business partner asked me to change a best practice guideline on a graph so that his customer would be happy with the analysis.
Lesson Learned: Perseverance is a good trait when a difficult culture exists. That is, sometimes the performance message takes a little extra time to get through. Merely having a performance management software tool does not make one an expert in performance management.
More Middle Years Of Performance Management – 2005 To 2018
During this period, I had the pleasure of working for a performance management company. This company was dedicated to helping customers with performance management and capacity planning needs. During this period, I learned how to manage other platforms such as AIX and Linux and helped non-IBM i companies understand how to create structured performance reporting. When I left the company, I had written over 500,000 lines of code, proving that the leap of faith to the application side of the house was well worth it.
Finally, I worked with companies (including MSPs) to ensure all parties understood the performance impact when mission-critical systems were being moved from on-prem to the cloud.
Lesson Learned: There is value in being a customer first. Collaboration with other people in the industry is essential. From a software management tool standpoint, the best ideas often come from people in the trenches. Finally, when a system is moved to the cloud, analysis is still needed to understand system performance. Every cloud vendor operates their server in a different way.
My Recent Years Of Performance Management
After working for corporations since 1983, and realizing the void in experienced performance analysts, I decided to start my own performance management company. Building upon my years of performance management experience I was ready to help others get the most productivity out of their systems. I understood what it took to show intelligent and succinct reports, how to drill into the culprits causing the systems to glitch, and most importantly, how to convey this information to decision makers. Then Covid happened.
While the pandemic contributed to my decision to go to a plan B, I will never regret the choice I made to follow my passion for performance management.
I now find myself in collaboration with two incredibly talented colleagues, and together we started a company called Greymine. We focus on system and applications performance and services. We have created internal programs to expedite our performance management analysis processes. We have never stopped thinking about the industry nor the customer perspective. In the coming weeks, we will be making a major announcement that will add another chapter to the history of performance management.
Lesson Learned: Change is hard. Sometimes a “mistake” in a career is a stepping stone to something bigger. I recognize there is a void in the IT world for a dedicated performance management company. That is about to change. Stay tuned for a big announcement.
Doug Mewmaw is vice president of performance services at Greymine.
This content is sponsored by Greymine.
Relieving The Anxiety About IBM i Performance
Melding System Monitoring And Capacity Planning
Where Did My Faulting Guidelines Go?
Memory Management: It’s Your Fault, Now Fix It
Using i5/OS Performance Adjuster to Better Manage Memory
SSD Performance: Be Careful Before You Buy
Why Are Systems Programmers Always To Blame?
AS/400s Are From Rochester, RS/6000s Are From Austin
System Performance Management Is Like Having Insurance
Congratulations to ITJungle and to Doug Mewmaw for showcasing this very important and promising new statrup company, Greymine.
Doug has followed many of the steps that I did decades before him as an IBM Systems engineer in the IBM mainframe (System/360) and and IBM i (AS/400) environments including directly supporting hundreds of IBM customers worldwide and in authoring many IBM application products (IUPs) that were the basis for founding several software companies, including Apparel Business Systems and Apparel Computer Systems.
Greymine is directly addressing the crucial direct support needed by tens of thousands of IBM customers that were previuusly accomplished by the long vanished IBM Systems Engineers (SEs) , IBM Program Support Representatives (PSRs), IBM Education centers, and IBM Sales Representatives who have all long since disappeared.
Greymine may very well find that by far the biggest immediate opportunities are in the performance tuning of the hundreds of thousnds of IBM programmers who are by far the largest financial cost due to low productivity and lack of effective productivity tools in both the IBM mainframe (System Z) and IBM i environments.