Effective Solutions Through Partnership

Category Archives: IT Modernization

Modernizing IT Part 10 – Modernizing with a Buy or Build

Information Technology, IT Modernization, Rob Klopp, Technology

By Rob Klopp

This repost is quite long, so I will keep my opening comments short and add some new thinking in the next post. Note that I have skipped from my first cio.gov post to the last post. Of course this is intentional. I find that there is a lingering misconception around Buy vs. Build that could stand an open discussion. So here we go…


This article is reposted from CIO.gov; the original post can be found here.

In the last few posts I’ve been bragging about some successes. To close out my term I’ll try to say something more provocative and discuss how the economics of buy versus build are changing. Over the last twenty years in the commercial world, there have been very compelling reasons to buy rather than build. First, a pre-built product carries less risk of a failure in development. Note that there are lots of stories around failed implementations of commercial off-the-shelf (COTS), so this advantage can be overstated.

There is also an economic advantage as a company that has built a COTS package can leverage the software over and over again in the market and amortize the cost of R&D across many sales. In other words, it can be less expensive to buy COTS.

The downside is that a COTS product offers, by definition, a common denominator and not a purpose-built, custom application tailored to your specific business process. This downside is mitigated when your business process is more generic. We see great success in areas like accounting, finance, and human resources where companies tend to solve the business problem in similar ways. However, we see less success in building COTS for government where there is a limited number of customers and little market pressure on prices.

Finally, note that over the last twenty years, COTS has implied the installation and configuration of a shrink-wrapped product where “configuration” meant (almost) “no coding required.”

Let me describe how the playing field is changing.

Before I arrived at the agency, we performed a buy versus build analysis for a new system to build and deliver our postal notices. We evaluated the market and selected a very good commercial product and started building a prototype. In the very last step, we turned up the prototype to scale to the volumes of output required at the SSA and the product failed. It just could not scale. We executed a TechStat, ended the prototype, and parted cordially with the vendor folks. The cost of this exercise was considerable, but we did everything right. We bought rather than built and neither the vendor nor the agency could know that the product would not scale until we tried it on our workload.

That is not the important part of this story. The product we selected, like a great many COTS products today, was built upon an open sourced component freely offered by the Apache Foundation. In the course of our prototype development, my technical staff suggested that they could develop a system using the same open source component that would scale up. The cost of trying this was small and so we agreed to prove the concept. Today, we have developed a proof-of-concept and demonstrated that it will scale. This new engine, which is comprised mostly of off-the-shelf open source code, will be the basis for our next large IT Modernization effort aimed to provide a modern communications capability for the SSA.

Let me ask the reader — is our new communications capability going to be a buy or a build? Most of the capability is off-the-shelf. Note that it is not commercial off-the-shelf, it is open source. Still, did we build?

Consider the original value proposition of a “buy.” It involves less risk, but since the core of our new product is open source tried-and-true, the risk of a development failure is very small. It is less cost, but since the core is free the cost of our development effort was the same or less than the cost of trying to configure the vendor product to our specifications. Furthermore, we do not have to pay for a COTS license. Finally, we have a purpose-built custom result.

This trend, for COTS products to be built from open source components, is significant. Nearly every COTS product uses open source code and then the vendor charges the government a commercial license fee. This is not as egregious as it sounds because in a market system where there are competitive pressures, the value of free open source software will be (maybe imperfectly) passed on to the government.

There is another trend that is important. New COTS products expose their software as components just like open source. We are performing market research examining software from several firms that would want us to code our own custom result using their components to reduce the cost and technical risk of development. This is a very powerful approach, but again I would ask — does using these components represent a buy or a build?

As we plan for IT Modernization, every one of our efforts is likely to include either open source or commercial components throughout. Will these efforts be considered risky “builds” or safe “buys?” The line is just not clear anymore.

So I would suggest that every IT Modernization product must consider leveraging existing code to reduce cost and risk. When the heart of a commercial product is open source, we need to ask if the value add of the commercial wrapper is worth the cost or if we can build a bespoke product from the same components for less cost with the same or only slightly more risk. When there is no market for a commercial offering outside of a few government agencies, and therefore no pressure to reduce prices, we need to consider the ongoing costs of that near-monopolistic relationship.

The buy versus build equation is much more complicated now. We need to rethink. Old school shrink wrapped vendors will push us to always “buy.” New age vendors will offer components and ask us to build a little. We cannot knee-jerk to a “buy” and we must allow this sort of “build” to happen. Once we agree that these builds are OK, we need to consider builds that use open source instead of commercial components. We need to be sure that we are not paying commercial fees for free open source code. Finally, we need to be sure that when we buy, we buy products that compete in a market. If we enter into near monopolistic COTS relationships, we will surely pay the price.

About the Author: Rob Klopp is a CIO with a deep technical background. As the CIO at the Social Security Administration, Rob started and led an IT modernization program that influenced the entire Federal IT space. Before moving East, Rob held technical leadership positions at SAP, EMC, and Teradata; he consulted to several technology companies along the way. Currently, Rob is consulting to the State of California and is again helping to start initiatives to modernize legacy IT systems.

Modernizing Federal IT Part 1 – Catching Up and Jumping Ahead

Information Technology, IT Modernization, Rob Klopp, Technology

By Rob Klopp

This series on IT Modernization was written during my time as CIO at the Social Security Administration. This first post outlines an argument for why modernization is not just a nice-to-have, but is an imperative. Further, I argue that technology is advancing and now is the time to start.

Today, I would add to the argument that the gap between your legacy systems and modern systems is too large to overcome incrementally. Technology is advancing away from your legacy faster than you could ever catch up in conservative increments. Sooner or later you will need to take a bold stance and jump onto modern technologies.

I hope these posts help you to argue for a bold move. If you are on the business side, I hope you can argue against conservative technologists who argue for the status quo. If you are an IT professional, I hope that this series helps you to convince your leadership that the jump is inevitable, that it is not going to get easier, and that the advantages of modernization support the bold move.


This article is reposted from CIO.gov; the original post can be found here.

To start, let us consider the modernization problem from a high level. Figure 1 is designed to outline the problem. The blue curve represents the ability of a hypothetical Federal agency to adopt technology. The green curve represents the ability of a hypothetical Fortune 500 firm to adopt technology. The technology gap between these curves represent the problem confronting a Federal CIO; we believe that we must catch up to the Fortune 500.

Note on the curves that in the late 1970’s and early 1980’s, the Federal space, and especially the SSA, was on a par or even a little ahead of the Fortune 500. In that timeframe, the SSA was aggressive in developing systems and databases to store information. These systems pushed the state of the art. They represented Big Data in the 1980’s.

Unfortunately, the systems of the 1980’s, once developed and stable, became stagnant. Today they represent a significant technical debt. SSA’s core systems are today run on 60 million lines of COBOL and on another 1 million lines of Assembler language. Much of this code was developed long ago, before programmers carefully structured code to reduce the cost of maintenance. The mainframe languages, development, and operating environment are no longer widely taught in our university systems. The Federal staff who developed and maintained these systems are retiring. As a result, the interest payments on this 30-year-old technical debt are compounding and in the next five years we could face a crisis keeping the systems that execute the SSA mission running.

Closing the gap between Government IT and Commercial IT is critically important. However, closing that gap could be the wrong target.

Consider the orange curve… representing not technology adoption curve but technology advancement. You can see there the gap that haunts the CIOs of the Fortune 500. Technology is advancing faster than even the Fortune 500 can adopt it. It is this gap that start-up companies exploit to put the Fortune 500 out of business by jumping straight onto the technology curve.

Start-ups can make this jump because they are not encumbered by technical debt or by a requirement to sustain an existing customer base. They do not attempt to catch-up… they jump. If their jump succeeds, they make companies on an incremental growth path completely irrelevant.

Now consider the red dashed curve. This depicts the ambition of most Federal CIOs: matching the IT prowess of the Fortune 500. Consider the steepness of this dashed line. It is truly ambitious. But if a most-amazing Federal CIO managed to bridge this gap she-or-he would find that their agency has actually lost ground to technology.

Federal agencies are unlikely to go out of business. But the technology gaps represented here could make Federal agencies irrelevant. The most amazing applications today… pick your favorite: Uber or Siri or Facebook… will be outdated in five years. What will the public think of Federal agencies that are still rolling out web services with electronic forms in 2020? Not much, I suspect.

This is the topic I’ll consider in the blog posts to follow… how might the Federal government catch up in IT. I’ll discuss in detail how the SSA is addressing the problem. I’ll also be discussing buy versus build options, techniques to modernize old code, initiatives to develop a modern infrastructure, and jumping onto the technology curve.

I hope you enjoy the series and look forward to your comments.

About the Author: Rob Klopp is a CIO with a deep technical background. As the CIO at the Social Security Administration, Rob started and led an IT modernization program that influenced the entire Federal IT space. Before moving East, Rob held technical leadership positions at SAP, EMC, and Teradata; he consulted to several technology companies along the way. Currently, Rob is consulting to the State of California and is again helping to start initiatives to modernize legacy IT systems.