SENATE ELECTIONS, REAPPORTIONMENT & CONSTITUTIONAL AMENDMENTS
COMMITTEE
Informational Hearing on:
Open Source Software—
Does
State Capitol, Room 4202
Senator Debra Bowen, Chair
SENATOR DEBRA BOWEN: Good morning everyone. I’m Senator Debra Bowen. I would like to welcome you all to today’s meeting of the Senate Elections, Reapportionment & Constitutional Amendments Committee.
The
subject of today’s hearing is open source software. Many corporations now use open
source software systems, including Bank of America, Amazon.com, America Online,
Dream Works, Charles Schwab, IBM, and Merrill-Lynch, just to name a few. In
recent years, even federal and state agencies including, the state of
Today’s hearing will include a discussion of the Department of Defense and the California Air Resources Board’s experiences in moving towards open source computer systems.
My
goal with this hearing is to open the discussion about whether
And
so today, we will talk about the pros and cons and the challenges of going to
an open source model for
Last
year, the National Science Foundation awarded a $7.5 million grant to a team of
researchers from six institutions around the
I generally try to make hearings fairly interactive—that means that I may ask questions as we go along.
I just saw Senator Poochigian here. I don’t whether we will have other members here. I know there are other committees, and there’s a caucus retreat today for some members.
Anyone who has an interest in speaking during the public comments section, I would ask you to please let our sergeant at arms know. I don’t see any arms, but he is our sergeant at arms. He has arms. Maybe that’s it. He will hold them up and show you a signup sheet. We do that not because we wish to violate your privacy, and demand your social security number, or anything like that, but because it helps us know how many people are interested in addressing the body.
Our first panel is seated already. I want to thank you all for coming. Our first panelists include, Andrew Aitken, Founder and Managing Partner of Olliance Group; Michael Evans, the vice president of Corporate Development for Red Hat; Clark Kelso, a figure well-known to anyone who has attended hearings in which I have the gavel, chief information officer for the state of California; and, Anthony Hill, who is the chief technology officer for Golden Gate University.
I’d like to start out with Andrew Aitken. Thank you so much for agreeing to participate with us today. Please proceed.
ANDREW AITKEN: First I’d like to say, thank you for the invitation, Senator. I appreciate this opportunity.
I’ve been asked to provide some high level opening remarks on open source—the industry trends, dynamics, issues of that nature, which I will be happy to do so for a few minutes.
My media person here, Mike Evans, is going to be running my presentation for me. Thank you very much.
So a brief introduction: I’m here representing two organizations today. I am the founder and managing partner of Olliance Group, the leading open source management consulting firm. I’m also on the board of directors of the Open Source Software Institute, which is a nonprofit institution for providing advocacy and information for open source to the government.
So I’d like to start with some high level trends that we’re seeing around open source today. I will get into more details, get into more descriptions, advantages, challenges, and where the industry is going from our perspective.
A little bit more background about us, to put this presentation in context. Our firm, Olliance Group, has been around for four and a half years. We’re comprised of senior executives from firms like, Intel and Microsoft, and other vendors. And to date we’ve completed around 70 open source strategy engagements for most of the large OEMs and ISVs and also a number of smaller startups.
SENATOR BOWEN: I’m going to ask you to stop and tell the
listeners who are out around the state of
MR. AITKEN: Sorry. So we provided strategy consulting to large hardware and software vendors and also, both proprietary and open source startup companies and users—technology consumers.
SENATOR BOWEN: So your OEMs are your original equipment manufacturers.
MR. AITKEN: Yes. Sorry, I sometimes get lost in industry jargon.
SENATOR BOWEN: We all get lost there. But I will get complaints about it.
MR. AITKEN: Okay. So some interesting trends that we’re seeing today: The venture capital industry has gotten very interested in open source. They’ve made a number in the last 24 months. They’ve made more than 40 or 50 investments in open source companies, or companies that are beginning to sell open source solutions.
To date, one of the interesting aspects of that is, our primary client base are proprietary companies who produce proprietary products who want to understand how they can leverage open source in their technologies in their business models. That’s a very strong trend that we’re seeing today.
We’re also seeing a trend, most large technology vendors are beginning to move towards open source in some capacity, either consuming it internally, incorporating it into the products that they produce, or actually producing their own open source software and making it publicly available.
Today, open source is getting beyond infrastructure. Traditionally it’s very robust around the operating system, the web serving layers, application serving, email, technologies of that nature. Today we’re seeing a clear trend beyond that into more of the application layer, and I’ll talk a bit more about some companies that are in that space today.
One interesting item that is going to be very important for the entire open source industry is the newest version of the GNU public license, the GPL license, that governs more than 70 percent of all open source software, the first draft was released last month for public comment, and that process is going to go on throughout the year. And then they expect to complete the newest version of the GPL by Q-1 of next year. It’s a very important process for open source. There are some very interesting twists in the new license such as, how it treats digital rights management, and other technology issues.
The role of government: Government is important for the adoption of most technologies, but it seems to be particularly important, and especially on a global scale, for open source. It is being widely adopted by foreign governments around the globe, both at what we might compare to a federal and a state and a municipal level. There are a number of very, very wide spread efforts to do this. And I can talk in more detail later.
So, I think it’s important to also understand that open source or proprietary is not an either or equation. Today most large enterprises run both proprietary and open source technologies.
Some data points here: Open Office, which is comparable to Microsoft Office, is downloaded more than 800,000 times per month on average, and 60 percent of those downloads are on Windows.
SENATOR BOWEN: So that means you’ve got somebody who is running a Windows operating system who is downloading an Open Office software suite and probably has FireFox or Opera or some…
MR. AITKEN: In this case we’re talking about a Microsoft Office comparable product, so it has presentation, and has an Excel spreadsheet emulator, and it has a Microsoft Office Word offering. I mean, it’s a directly comparable suite. It’s about 90-95 percent compatible. And so they’re downloading this free in open source software and running it on top of Windows.
Postgress is the number two open source database. And approximately 60-65 percent of Postgress, again, freely downloadable, is downloaded and run on top of Windows.
Another interesting trend is that Microsoft is obviously becoming much more aware and involved in the open source community. They have open sourced three of their own very small applications to date. They’ve recently announced a partnership with J-Boss to make their solutions more interoperable.
So again, I would like to point out on this slide, that it’s not an either/or decision.
So these are some of the most cited advantages. Senator, if you have some specific questions, I’ll be happy to address them. But these are some fairly common advantages with open source.
SENATOR BOWEN: Actually, I think there are a couple of things that will be useful to explain in a little more depth. I know when people think open source, some people use the term “freeware,” or “free software,” and then the whole idea of a license for some people might seem strange. So, perhaps you can talk a little bit about open source software and the notion of licensing, and what the license does and why it’s needed.
MR. AITKEN: And I do have a slide on that, if you don’t mind.
SENATOR BOWEN: Sure. If you’re going to get to that, then…
MR. AITKEN: I have just a few more slides and then I’ll get right to that and describe it.
SENATOR BOWEN: Okay.
MR. AITKEN: So this is really one of the key questions—to open source or not to open source? I’d like to address the two key segments: technology vendors and technology consumers.
For technology vendors, it really boils down to two key issues: Are they able to reduce their SG&A (sales, general and administrative costs) by leveraging the dynamics of the open source community model? What this means is, are they able to open source their software to build a community around their software, which then helps drive revenue opportunities for them?
If you look at Oracle’s most recent financial statement, their license revenue did not cover their SG&A costs. So in essence, the people, you are paying for the privilege of an Oracle sales person coming and selling to you, not for the access to that license software—okay? Open source changes that equation, or, hopefully changes that equation.
The other reason for technology vendors to consider open source is, can they reduce their research and development costs and improve their time to market for release of their products by either incorporating open source or by implementing a more open development model?
For technology consumers such as…
SENATOR BOWEN: Can you talk a little bit about what the perceived advantage is in time to deployment? Why would it be any faster or cost any less for engineering?
MR. AITKEN: Sure.
SENATOR BOWEN: I’m going to ask you some questions that I know the answer to, or at least I think I do. By asking, I don’t have to demonstrate to you whether I’ve got it wrong or not, I just get to learn.
MR. AITKEN: Okay. So there is a concept of code reusability. There’s a lot of very, very robust open source code out there today and it allows organizations to take its existing code instead of trying to recreate it themselves. There is also the notion of the community conducts….the community doesn’t necessarily do your research and development for you. Where you can leverage open source for reducing R&D costs is, they tend to do a very good job of finding bugs, producing patches, and fixes, and things of that nature.
If you talk to Martin Mikos, CEO of MySQL, one of the premiere open source database companies, he likes to mention how he has the largest QA department in the world that no other company can compete with, and that’s his community and his customers, who go through the code and find bugs and provide fixes and patches and such.
SENATOR BOWEN: I actually had an email exchange recently with someone about that assertion, that there’s better ability to identify and correct bugs earlier on and it devolved into an argument about two leading internet browsers neither of which is without bugs. The assertion was made that the open source browser has as many bugs as the more traditional browser that probably more people are familiar with. And the assertion was then made that that proves that that’s a nice theory but it doesn’t actually happen that way. Your comment?
MR. AITKEN: I’ll name them. FireFox, I think is the open source browser you may be referring to and I have very, very personal experience with how buggy that can be. The browser that you download and run is very robust, very stable, and offers some wonderful features. The problem with open source, or the benefit of open source, depending upon how you look at it, is there are so many people who have written their own additions, new tools, utilities, and such, around FireFox, that if you want to incorporate those in your browser, they’re not all tested; they’re not all integrated. That’s one of the issues.
One day, I’m not quite sure why, but I went through a list of all the tools and utilities and added about 15 functions to my standard FireFox browser. And it took me about six hours to strip it all out and restart again. So that certainly can be one of the challenges. But over time, those tend to be, in fact, very rapidly, those tend to be addressed and fixed. You’ve probably heard the term “many eyes make for shallow bugs.”
SENATOR BOWEN: No, but it’s a good term.
MR. AITKEN: So for technology consumers I believe there are three primary questions to ask yourself when considering open source: Is there a compelling strategic reason? And I think that’s what we’re perhaps talking about today. I don’t know if it will save the state money to consider open source for their e-voting process. I’m not sure about the technological advantages, but there could certainly be a very strategic reason to do so. But this is what technology consumers need to ask themselves when they’re considering open source.
So briefly, I’d like to talk about some of the business models. Jenny had asked that I provide an understanding of how some companies are making money in open source, and these are some of the various business models and companies that are utilizing them.
So first is services—providing services and support around a particular open source application. J-boss, the leading open source application vendor, is a primary example. Red Hat, also, I’m sure Mike will talk in more detail about that. Compuair is an open source enterprise resource planning provider. It provides a core open source application, but makes their money from providing services around it. Then there are others that operate more on the traditional proprietary model. They provide a product and they provide a license to that product, and they may also provide services, but they generate the bulk of their revenue from a traditional license model. Sugar CRM is a prime example. They’re one of the open source success stories today. And Jasper Soft is a company that provides an open source business intelligence reporting tool.
A couple of others I want to point out briefly: Some people say that open source is more about commoditizing big, inefficient, existing proprietary vendors and that it doesn’t really innovate. I think this company Funambol, which provides open source from mobile applications, is a prime example of how the open source community can innovate, not just follow.
And a really interesting company is called Zimbra, and they’re doing both—they’re innovating, and they’re also commoditizing other vendors. They provide an email solution that competes with other proprietary email solutions. But they have some, frankly, cool technology in there that’s quite innovative.
Proprietary companies—how proprietary companies are making money around open source. Services, traditional consulting such as ourselves, such as large vendors, like, CSC is very obvious. Lost Leader, a company might create an open source application, put it into open source to generate interest, to generate the ability for more customers to see their actual proprietary revenue producing software. It actuates also in the business intelligence space. They provide an open source application that doesn’t generate too much revenue for them, but it drives revenue to their other products.
SENATOR BOWEN: Can you take a moment and tell people what the business intelligence space is?
MR. AITKEN: Sure. It’s reporting tools. So today technology is about generating information and data, well, you need to be able to access that data. You need to be able to extract usable information from it. And so they provide tools that will allow you to search large databases to create reports and make it usable for people like us.
SENATOR BOWEN: So they’re basically figuring out what’s in their own database largely and trying to put together what the information that is useful in business planning or in evaluating performance?
MR. AITKEN: Exactly. For making decisions based on data or information.
SENATOR BOWEN: Well, we use a lot of jargon in this world and I’m sure that I’m missing some of it, but will try to help as we go along.
MR. AITKEN: So there’s another strategy which is very important and it’s called “gifting,” or, “patronage.” IBM is one of the primary users of this strategy. What that means is, they actually will give code to the open source community to seed it to provide code that may not be core or critical to their proprietary offerings but will generate either a large community. In an example, they provided, I believe, $40 million in a number of resources to seed the Eclipse community. The Eclipse open source community which Eclipse is a nonprofit organization that has become the de facto standard for software development tools. And they did it to compete directly with Sun and Microsoft. And within two years it has gained tremendous, tremendous amount of attraction and interest. And so they did this by seeding the community, by giving resources and funds, and they’ve done that on a number of other areas also. I believe they have over 500 engineers contributing to Linux, is that right, Mike? Something like that? And so they’re working to make Linux, the operating system, better.
SENATOR BOWEN: And their return on that investment is, through their?
MR. AITKEN: Through the selling of their hardware. There’s a variety of advantages to them both what I would term as offensive advantages and defensive. So, putting a competitor in a down position, creating a new standard that happens revolve around their products and their solutions and their offerings.
Consumers….so those are examples of companies that consume or utilize open source technology but don’t necessarily produce or provide anything back to the community itself. So Google uses a tremendous amount of open source technology, but doesn’t provide or produce much of it’s own for the community. Oracle is an example; Linux, they base their products….their products run on Linux, and it’s the fastest growing of their products, so they run their products on top of various versions of Unix and Linux, and Linux is the fastest growing product suite they have there—product set.
And the last is, revitalization. So there are some companies there that would take what we might characterize as technologies that have lost their edge, losing their customer base, tired, as it were, and in some cases they believe that by open sourcing it, they may be able to revitalize that product and make it viable again.
Computer Associates is an example. They took a large database and have recently made it open source, formed a new company around it and are hoping that that will revitalize the technology.
To your license question: So, today there is an organization called the OSI that approves open source licenses. There are somewhere around 55 or more open source licenses to date. My favorite one is the one that says, “If you like my software, send me a beer.”
There is a movement afoot to reduce the number of licenses. A number of vendors who had their own open source licenses have retracted them.
SENATOR BOWEN: And you’re talking now about the legal language of the license—“Send me a beer,” as opposed to 40 pages of thou shall do this and thou shall not do that?
MR. AITKEN: Exactly. It doesn’t address the…
SENATOR BOWEN: So having 55 different licenses clearly increases the complexity for someone who is trying to use more than one piece of an open source…
MICHAEL EVANS: I think, Senator, one of your original questions was why you need a license if it’s free ____________ and the simple I give is that, the license is there to guarantee the freedom which is a higher level point than this.
MR. AITKEN: That’s a good point.
SENATOR BOWEN: In other words, you can’t take the open source software and close it?
MICHAEL EVANS: It depends. There are variations of licenses that allow you to do that and some don’t, and that’s the GPL being the one that tends to be the one that doesn’t allow you to do that.
MR. AITKEN: Most open source licenses fall into two categories, either permissive or restrictive (addressing exactly what Mike was saying). The permissive style of open source licenses on the right hand side of this slide, are the ones that allow you to take the open source software and do more with it what you will. More flexibility. The ones on the left, have more requirements to make sure that the open source code that you’re working with is, any derivative works you may create from that original code are also under that same license, so they are required to be made open source themselves.
The one that I want to know briefly is the one on the bottom left hand corner—the SPL. It’s called the sugar public license. It’s one that is becoming more and more popular with commercial open source companies because it allows for protection of your intellectual property. It allows for some branding of your software and provides some other protections that other licenses don’t. But it is not approved by the OSI, and that’s an important distinction.
SENATOR BOWEN: And what’s the practical impact of that? What does that mean…
MR. AITKEN: Of approval? By the OSI, the practical impact is that your software risks….if you publish a license that’s not approved to create your own license, the practical impact is that your software….well, first, you’ll be flamed to death, as they say through various open source license and discuss groups, which actually today impacts your revenue. So, there will be….today the open source community is no longer the geeks and the hackers who were working their second or third jobs at three in the morning. Today the open source community is comprised of developers working at a number of those firms that you mentioned today earlier in your opening remarks. One very pertinent example is, at Wells Fargo the head of their, or the director, of their infrastructure services is also an Apache committer, which means he’s at one of the highest levels of the Apache community, open source community…
SENATOR BOWEN: Can you tell us what the Apache community is?
MR. AITKEN: Sure. The Apache community is a nonprofit organization…
SENATOR BOWEN: When we talk about the layers of service, that’s…
MR. AITKEN: The Apache community is a nonprofit organization that manages a number of different open source projects today—probably 20 or 30 of them today. Apache itself is the most popular open source project that it manages, but there are a whole host of others that are under its umbrella. And Wells Fargo has made the determination that that technology is so important to them that they want to have someone whose job is to be involved with the community to provide their input and their requirements into the community, and hopefully impact the direction of the community, and also to learn and understand where the community is going itself. We are seeing that from more and more organizations.
SENATOR BOWEN: It’s interesting, it’s not all that different than what we see in the telecom and electric utility world, where if you take a look who is on a board of a local chamber of commerce, I challenge you to find more than a handful of local chambers of commerce in California who don’t have someone from either a telephone company or an electric utility serving somewhere in a volunteer position on the board.
MR. AITKEN: It sounds very similar.
SENATOR BOWEN: It’s interesting to see that moving into this area.
MR. AITKEN: So I wanted to conclude with one data point. Our firm recently completed a large research project for NOEM, which we did a lot of quantitative analysis. And one of the data points that we derived from that is that over the last three plus years, open source, and we looked at….this is comprised of…. probably 12 or 15 different open source applications were downloaded from the internet for free more than 450 million times globally. Most of those were downloaded and run on Windows, so I think it’s kind of a big data point.
SENATOR BOWEN: I’m responsible for some of them. I don’t have any family who is still running certain applications.
Let me ask you about the definition of open source and what you use. You know, there’s a running discussion in the open source community about what it means to be open source. What definition are you using? Or does it matter? Maybe we just shouldn’t discuss the head of that pin, I don’t know.
MR. AITKEN: Well, there are a lot of variables, certainly, to that definition whether you’re talking about individuals, organizations that are either commercial or governmental in nature. Personally, to be open source is, from a commercial perspective, means that you’re deriving revenue from open source solutions in some manner. As an individual it means that you’re contributing to the community however you can to the best of your ability. I’m not technical. No open source project would want me to contribute to their code. But I work with users groups and support them and put events together. That’s how I contribute to the community itself. So I think there’s some very wide latitude in that definition.
SENATOR BOWEN: Let me ask a few questions specifically about the process of determining what the appropriate solution is. When you’re working with a client and you’re evaluating whether to use open source software, proprietary software, or a mix, what process do you go through? Specifically, what kinds of things would you consider?
MR. AITKEN: So there is a very defined process to that. And the first thing is, the first question we may ask is, why are you doing this? What are you hoping to achieve? And it goes back to, is it a strategic imperative? Is there a financial imperative? Or do you expect there to be some technology benefit from this? Some of the issues that we look at when a customer gives us that response is, obviously you go through some of the basics, what is their IT organizational infrastructure? But more importantly, and what I think what many people miss is, what is their culture? Culture can be one of the key determining criteria’s in the successful adoption for open source or not.
I can give you a specific example of a retail organization that we helped develop an open source plan. We helped them select what applications they were going to use. And they were much more gung-ho about it than we were. And they wanted to adopt open source as fast they possibly could and we kept trying to put the brakes on that, because they were coming from a pure proprietary environment where the developers were very oriented to a certain technology set and they were enthusiastic about open source, but they really had no idea what that meant. And they deployed and began using a dozen, maybe 15, open source applications and it became a nightmare for them because they weren’t culturally prepared for the differences between open source and proprietary technology.
SENATOR BOWEN: And what do you mean, the cultural differences?
MR. AITKEN: It’s a different way of deploying and developing and managing your own infrastructure. It’s much more participative; it’s less hierarchical, and structural. Certain sets a proprietary technologies operate with very robust user interfaces. So it’s very simple to use. Open source technologies are what is called more command line driven, so you need a more sophisticated engineer to be using those technologies. And if you haven’t identified that, if you haven’t gone through that training, then that can become a significant issue. Additionally, open source might not be as well documented. You might have to provide your own support. And if you don’t examine these issues prior to begin using open source, it can become quite a challenge.
SENATOR BOWEN: You end up instead of paying for the software, you end up paying for someone to help you figure out how to use it.
MR. AITKEN: Yes. And that’s fine to get started with. You do that whether it’s proprietary or open source. But it’s important to identify those challenges; identify what your cultural organization is like to date, and then put a plan in place to address that rather than finding out after the fact.
SENATOR BOWEN: Okay. All right. Well, Clark Kelso has asked if he can have the second slot, since he has other engagements this morning. So, without any further ado, welcome. Glad to have you back again.
CLARK KELSO: Thank you, Madam Chair. Good morning. My
name is Clark Kelso. I’m the chief information officer for the state of
I can state the state’s general policy regarding open source relatively briefly. And this does appear in a memorandum that I released almost a year ago, February 16, 2005, that’s on my website. In essence, we view open source as an alternative that should be evaluated by departments and project owners and managers, and by the states IT professionals, as they are considering the appropriate technology solutions to support a department’s business needs and programs. There is no policy preference that we have established across the board in either project design, or in procurement for either open source, or proprietary solutions. We have instead tended to believe that the architecture of individual IT projects is really best determined by the project owner, and the analysis supporting those decisions typically is what should be appearing in feasibility study reports that are reviewed by the Department of Finance and by LAO and by the Legislature as new projects are moving along.
In considering alternatives, we look at a variety of factors. The most important of which probably includes alignment to the business needs, and the ability to produce value to business programs from a particular solution, security, reliability, performance, maintainability and sustainability over time, development and maintenance risks. We want to make sure that we’re going to not have a failure as we go through some of these projects, technology trends, and, of course, costs. Those are the major factors. There are others that are also considered, but that gives you a fair sense of the considerations that we ask departments to examine in determining an appropriate technology solution.
I think if you look at the history of the executive branch and IT, you can break it down into three or four big chunks. An early chunk, 30, 40 years ago, the state was much more itself, across the board, engaged in its own development activities, typically on mainframes. And the state was, I think, early on, very successful in developing and acquiring very stable platforms to support state programs. Nothing else can explain the fact that we have very stable 25-, 30-, 35-, year old legacy systems that are still serving the state very well. Beginning in the eighties and nineties, with the development of desktop personal computing and then midrange computing, much of which was done at the department level, we began to see an increase in the acquisition of commercial off the shelf products, cot solutions, as well as an increased usage of consultants to assist in the development of custom built applications.
I think today, when I say a commercial, off the shelf, it almost never is just commercial off the shelf, we almost always having to do a cot solution plus…
SENATOR BOWEN: There aren’t too many shelves that stock software for states of 37 million people.
MR. KELSO: Yes. And we, of course, government, does tend to have functions that are not replicated in the private sector. Many cot solutions are developed for really a private sector utilization. I think over time, also, what we’ve seen then, is depending on the department you’ve been in, the amount of attention paid to continuing to develop the ability, and maintain the ability, to do our own development, our own programming. That has varied substantially. Some departments made sure they maintained their workforce. They kept their training up. They kept them near the front edge of developing technologies. Other departments chose to go more the path of, let’s bring in consultants to assist in major development activities.
Part of the issue for departments, of course, has been the state’s HR systems. Something as seemingly mundane as the state’s classification systems for IT workers, which hasn’t been updated in 25 or 30 years itself. It does not recognize that the internet was created. That makes it sometimes difficult for the state to bring in and recruit folks directly out of college who have many of the skill sets that you do need to maintain cost effectively, a good development shop, which I think is one of the important prerequisites, not entirely, but an important aspect of the open source community and culture.
Now, what we have been doing is then several things. We do have several departments that early on made the decision, identified the trend in open source, and turned their systems towards open source. One of the leading departments that has done that is the Air Resources Board. This was a process that they began in the mid to late nineties. They managed to maintain a culture among their IT workforce and the skill sets to do development activities. So today, Air Resources Board, has reported to me that they run about 64 percent of their 88 servers on a Linux operating system. They have about 64 percent of their databases. They have 55 databases they’re managing using open source MySQL. Eighty-nine percent of their 61 web services are Apache based services. A very, I think, significant investment in open source technologies. And they report that they have seen in their operations cost savings that are substantial, exceeding half a million dollars or more. They think open source does provide best overall value to them and their solutions and their business needs. And echoing something that I think you heard before, it seems, although we haven’t quantified it, it seems to them that their development time is a little bit quicker. Now part of the reason for them, I think, that it’s a little quicker is, you are able to avoid, if you’ve got a good development shop and you’re able to pull things down off the web, you are able to avoid procurement cycles. And that, of course, can be a significant delay in the state in bringing up almost anything. That, of course, is a little bit peculiar to the state, but it is an important factor.
SENATOR BOWEN: Let me ask in a little more depth about that. How does procurement work for open source solutions? There is no procurement? You’re basically hiring people in your IT departments at the agency level who…
MR. KELSO: Well, it’s more complex than that, of course. Because the slides you saw from Mr. Aitken’s presentation, I thought it was, at least my understanding of open source community, a pretty fair picture of it. You can see that it’s much more complex than just, you pull it down off the internet; it’s free; you get to use it. So it depends, is my answer. To the extent that we’re going to be pulling down a portion of an open source solution, perhaps mixing it with something else, maybe we actually are going to be going to a vendor who is going to be helping us to develop something in part using open source, you still may have procurements, so it depends really on exactly what is it you’re developing; how small; how complex; can you do it entirely in-house? If you can, this maybe a way of simply avoiding much of the procurement activity. For big sorts of solutions and applications, I suspect there we still have procurements, open source can be a portion of that activity. So the answer isn’t, it depends.
And you can tell that the open source community is itself becoming somewhat more commercialized in a variety of very interesting ways. A development that has occurred as best I can tell over about the past three or four years significantly, the reference to venture capitalists coming in, major IT players like IBM suddenly showing up and saying, “We’re going to play in this space,” that’s fairly recent.
SENATOR BOWEN: And how do you work with departments, or how do departments determine where they’re going to use open source applications? How do we get to 64 percent running on Linux, 88 open source, 55 of the databases, (the web servers are easier because the dominance of Apache and that’s easier to understand) but still, that must mean you have 11 percent of your web servers that are running on something else, doesn’t that require a duplication of expertise and effort? How does that work?
MR. KELSO: I think the answer is, how do you get to that point; how doe they get to that point? It was primarily the leadership and commitment of their IT leadership. Bill Welty among others, who over the years decided that that was, for their department, a good strategic direction to go. And this relates to the things that we are now doing. And it relates to, I think, what Mr. Aitken referred to as that culture in the community that exists.
We did establish last year, a working group across agencies to track open source developments nationwide and within the state. We’re going to be conducting a survey very soon to see who actually has open source, to do some education across the state, because this is something that requires a bit of a retooling if you’re not ready to make the best value of open source solutions. And to create, really, a forum for those who are doing open source. And we have lots of people in different departments who are doing it at various levels, including people who are doing the open office sorts of applications, to give them a place to begin having a community building process so that the departments that are doing this don’t feel so much like they’re alone, and so that we can begin to interact a little more strategically with the open source community.
We also are very far along in discussions with the State Personnel Board, Department of Personnel Administration, and SEIU, to update the state’s IT classification system, to adopt an new testing methodology that would permit us easier access to college campuses, to help our recruiting of people who very likely will be very much in tuned with these types of developments. We have departments doing succession planning to try to see where our gaps are in the workforce. We’re doing statewide enterprise architecture including development of a service oriented architecture that, in a very small way, does some of the same things that the open source community is doing, I think, globally.
SENATOR BOWEN: Well, let me ask you a question.
MR. KELSO: And one final thing before I forget—we actually are looking pretty systematically, I think, as part of consolidation of our server infrastructure and management of our servers at the obvious swap out there. I think something can happen there fairly easily.
SENATOR BOWEN: Let me pick a wild example that I’m sure is
something you’ve never thought about before. Let’s imagine the possibility that
under the Real ID Act that we have to relicense, I think it’s 18 or 22 million
MR. KELSO: Well, and I think this goes back to what I’ve suggested before, but first let me say, that fortunately, I would never be in a position of actually having to make that decision. That’s why we have a Department of Finance that is well staffed to review project details. This would be a big enough project. I’m sure the Legislature itself would have an involvement. But, putting all those caveats to one side, I would be most concerned, I think, about sustainability and risk of a project’s success. Now everything that I think you’ve so far heard suggests that for something like a database development, and I suspect what we would be talking about with a Real ID implementation probably would be built on something that’s internet based, perhaps not, but it may well be. There’s clearly a trend in that direction. I would want to know that something as large as what will be required for Real ID has been implemented successfully somewhere else, either in the public sector or private sector.
SENATOR BOWEN: So you might look at a large fast food company and ask, do they have a way to track huge volumes of transactions with a lot of detail that they’ve implemented using an open source solution?
MR. KELSO: Sure. And I’m routinely getting advice from financial services industries which have similar sorts of database issues. So I’d want to make sure that the risk there is one that the state is comfortable in managing.
SENATOR BOWEN: So in other words, you don’t ever want to be the first, you’d rather be the second, or the third, or the fourth?
MR. KELSO: I think that’s a prudent approach when you’re using someone else’s tax dollars and you’re not just trying to do something with a quick return on investment. I would like to see that it’s been done elsewhere. Now my impression is, and I’m pleased to see with the open source community, that a lot of the major IT companies are beginning to invest there. We’re watching very closely of what other states and other governments are doing.
SENATOR BOWEN: I was just going to ask you about
MR. KELSO: Sure. I appeared on a panel at the Linux Conference last year with Peter Quinn. We were following their implementation with great interest. I mean, all state CIOs are following what is happening there. So we are trying to, through this working group in part, and through my own contacts, see how stable, sustainable, maintainable, open source solutions are. And the big question for me is, do we have a gap in our own capabilities.
SENATOR BOWEN: I should explain, because that question was
very much an insider question so I violated my own rule.
MR. KELSO: That’s why we thought it was worth tracking. I’m not sure it was all of their systems, but I believe someone here on this panel may know it. It certainly was going to be on their desktops, as I recall. Do we have anyone who knows for sure?
SENATOR BOWEN: I think we probably do on a subsequent panel.
MR. KELSO: I’m sorry. Bill.
UNIDENTIFIED: __________
MR. KELSO: It was an open document format across the board.
MR. EVANS: Could I say something just on this topic of the selection of open source that you addressed with Andrew.
SENATOR BOWEN: Please.
MR. EVANS: Because, six years at Red Hat, I’ve seen a range of governments, private enterprises, startups, to big corporations trying to say, “Should we be using open source?” And majoritively now it’s pragmatic versus idealistic decisions. And it’s just the normal stuff of, will the solution and utilization of it work for me short-, mid-, long-term and provide benefits? And then they start looking at the angles like, is there references of something similar? Is it going to be better, faster, cheaper? Is it going to be insecure and reliable? But then the smarter people, to me, are also looking at, now the solutions they’re looking at are likely to be 10, 20, 30, 50, and 100 years, as you described, a 25 year legacy already and starting say, “What will happen over that time period and, if that company that’s selling it goes away, what will I be able to do in 50 years to plug into it, extend it, get other people to work with it?”
SENATOR BOWEN: Let me ask you to identify yourself. We do have people who will be listening to this audio only, and when they start to get voices they’ve never heard before it’s even more confusing than the conversation about computer software already is.
MR. EVANS: Michael Evans from Red Hat.
SENATOR BOWEN: Thank you. And then, let me just ask Mr. Kelso one more question, because I know you’re going to have to leave.
MR. KELSO: I appreciate that.
SENATOR BOWEN: And then, I’m going to hear from Mr. Aitken on his comments, and we will get to Mr. Hill. This is the kind of interchange though, that I wanted to have. It’s much more interesting when we all have some input, than when we listen to 20, five-minute lectures.
Security, one of the things that I’ve heard touted about open source solutions is they are inherently more secure in part because of the shallow bug issue. Though others have said to me, “Well, they’re only more secure because they’re less of a target than some of the proprietary software that is….there are more users, so if you’re going to write a bug or write a virus, write one that’s big instead of one that’s little.” There’s also been the view expressed that open source solutions are inherently less secure because anybody can pull down the code (presumably, this is the code that someone is using) and change it. And I think some basic education about security and open source and where do you....if you are going to run an open source solution in a California agency, do you just Google the websites that have what you want and have everybody just download whatever it is they want, or do you have a more hierarchical kind of choice of deployment so that you know what’s running where?
MR. KELSO: Well, Mr. Welty, who is here in the audience, perhaps can address directly what they’ve done at ARB.
The open source community now is much more organized than simply Googling and grabbing things from here and there. That’s part of one of the benefits of that community, is that there’s a concentration of those solutions so that a lot of people can look at this one thing. It’s not completely decentralized. Mr. Welty, I’m sure, can suggest how they’ve done it. I can tell you from my own experiences as a computer programmer when I was younger, part of the culture of this is, you know, empowerment of the programmer who is able to go out there with the skills to bring in things that others have looked at and put them together in a way that serves your business needs.
SENATOR BOWEN: I think the concern is that, and specific to voting machines, I’ve heard the argument made that well, if you have open source software on a voting machine, anybody can come in and just put different open source….change the code on a particular voting….this presumes, of course, that they have access.
MR. KELSO: Sure. And on the security issue, I have no particular expertise. I’ve heard the same arguments on both sides. It seems to me, those arguments ultimately are inconclusive. I’d like to see something a little more quantitative and empirical. I would say, and I’m sure going to hear some of that, I would say, for me, anyway, the security problem of the day is the fact that we’re all networked. I’m much more worried about the number of connections that state governments networks has to the internet generally, and what are we doing to have defense in depth at all of those connections, knowing that the weakest one, I’m much more worried about that problem than I am about this particular dispute.
SENATOR BOWEN: Okay. Good. Let’s go to Mr. Evans. We haven’t given you a chance, so why don’t you proceed. If you want to start with this question and then do the rest of your presentation, that’s fine. But I think we’re not going to get into the meat of some of what we want to talk about.
MR. EVANS: Okay. So I’m Michael Evans of Red Hat. I just want to finish answering the security question, because the sound bytes that you mentioned are common ones about security.
But two points
that to me have evidence—one, as
SENATOR BOWEN: Can you explain locking down, opening. It inherently doesn’t make sense to some people, so you need to help us.
MR. EVANS: Because the source code is freely available in an open source manner under a GPL license or a BSD license, doesn’t mean that when it’s running on my system you’re able to come in and see my source code. That’s a leap, if you will, and it’s not true. I can lock down things that don’t allow you to see my source code. So that’s maybe a simple way to….although there are definitely a lot of people in the United States and in the world with expertise in this area. This topic has been debated back and forth with different agendas pushing different examples and scenarios and comparisons and coming up with results that often suit them. But we have deep experts in this topic at Red Hat and there’s deep experts in the open source community.
SENATOR BOWEN: A what person?
MR. EVANS: Deep experts.
SENATOR BOWEN: And would you define deep expert for us?
MR. EVANS: Someone who is a deep expert, is someone who fully understands all the elements of securing software and securing open source software.
SENATOR BOWEN: All right. And something that we have none of, by the way, in the Legislature.
MR. EVANS: No. I wouldn’t assume you would want them. Well, I would want them in this organization.
I’ll go ahead then.
SENATOR BOWEN: Please.
MR. EVANS:
It’s a pleasure to be here. I’m Michael Evans from Red Hat. I’m
doing my presentation in Adobe Acrobat because I had challenges with a
proprietary format in Microsoft Office getting translation, which is that
I’ve been six years with Red Hat, watching the evolution of open source over those six years and participating in it. I’m based in the Bay Area of California. And I’m going to talk just a little bit about Red Hat and our experiences in open source, as the company has been around a while.
The
company was founded in 1993; based in
So what do we do, actually? The simplest way I describe it is, we’re a business built on fueling and commercializing open source software. We do focus on infrastructure level software as was described, not application level like voting software, hospital software, accounting software. We are focused more at the infrastructure level. We’re considered one of the leaders…
SENATOR BOWEN: Could you explain….just take a minute again, we are talking here to a group of people who may have no sophistication. They may look at this slide and say, “What does it mean to fuel open source software, and what is the infrastructure level as opposed to the application level?” So if you could help us just with some basic stuff. We only have to do it once.
MR. EVANS: Sure. So fueling in terms of the business model around fueling and commercializing, fueling means helping to educate, propagate, and advance, and publicize that open source. That both the development model and the business model can work, are workable, are interesting, are providing value to the world and to people. Commercializing means, taking various free and open source technologies and packaging them up and offering paid services and options around them. In terms of infrastructure software it tends to apply to operating systems, or databases, or storage, things that run at the bottom level of the computing chain, if you will, that applications like, voting software or accounting software, or hospital management software, would run on top, that’s generally called the application level. And we are heavily focused on the infrastructure side.
We are considered one of the leaders, one of the best known names in open source. And there’s two parts to our model. There’s both an open source development model, which some call the supply chain. How we get the technologies to create that we then commercialize and offer to people to pay us for our services to provide to them. And the business model is how we commercialize that; how we charge for it; how we provide the services to governments, corporations, around the world.
We have a main focus at Red Hat, at least I’m providing things that are lower costs and provide higher value, for a couple of reasons. One, we believe open source allows us to produce better quality technology. And there’s examples of that. It’s the highest performing technology, or the most rich functionality. I’ll talk about some of those examples in a minute. And then lower costs of solutions, we believe allows a much wider use of technology around the world, which we are fans of propagating technology in more places for more people and reducing the silos usage or some of the constraints on financials.
We’re most well known for the Linux operating system, as I said. The Linux operating system has two main elements. There’s a server version that competes with the Unix vendors, with Novelle Netware, and Microsoft Windows versions in the server marketplace. And generally there’s a desktop or client’s side version that runs on desktop machines or devices like handheld devices, that tends to compete more with Microsoft on that side.
The next slide I have is, “Why open source works?” And these are the six elements we attribute to: Due to standards—true open standards which are royalty free and no restrictions; value—people pay for what they need. So there is an element of open source, at least at our level of technologies, where people can use things for free if they want, and then if the need the services they pay for them (Andrew mentioned some of that in his business model discussion); innovation—the unmatched speed of development where we have literally our programmers all around the world plus there’s literally hundreds of thousands of other programmers around the world contributing to some of our different projects like Linux, being a popular one; and the speed of development and the brain power you can tap, and that is unmatched of any one corporation trying to put people in one building or ten buildings; the quality; better software faster; choice, trying to provide choice—no vendor lock in; and full customization ability.
There’s a statement at the bottom I put there called “Software for the next 200 years,” which there is a paper by a gentleman by the name of Dan Bricklin who is from Massachusetts, a fairly famous person in the computer industry, having been the originator of VisiCalc, who wrote about software for the next 200 years, talking about that the software needs to be able to be built upon, modified, changed, adapted to, over time. And you can’t just have a two-year, five-year, ten-year vision of utilization, and if that company that sold you that software is gone in five years, what do you do then?
SENATOR BOWEN: You know, is that really possible? I can remember as a young lawyer, being probably one of the few people who ever went in the word processing room. I actually learned a little bit about how that stuff worked. We had something called a Wang. And we had floppy discs the size of dinner plates.
MR. EVANS: Right. You mean, is it possible that stuff will last for 200 years?
SENATOR BOWEN: Yes.
MR. EVANS: Well, but…
SENATOR BOWEN: That stuff didn’t last for 10.
MR. EVANS:
I just heard
SENATOR BOWEN: You know, government is unique in that. And
one of the things that I’ve learned, we laugh, but my first involvement with IT
was probably in 1994 in the Court Technology Taskforce, which is how I got to
know Clark Kelso. We were looking at providing the infrastructure for the court
system for aiming at the year 2020. And we have….everything we buy lasts
probably
MR. EVANS: And actually, my point is that especially as the internet infrastructure is exploding, you will have infrastructure that is built that is going to last similar to having the plumbing and the electrical and the sewage infrastructures of the world, that will be there for 100 years as the underpinnings. It will eventually be 100 years, but there is stuff that is going to be laid out as the utility infrastructure that is going to be just like when people dig up and find the 100 year old electricity that’s still working and plugging in. So to me, that’s a forethought that….there’s a great article if you haven’t seen this one, and a lot of it is about governments are going to be building their infrastructures around technology and the internet infrastructure.
SENATOR BOWEN: Well, that’s certainly been one of the major trends in the time from when I first created a Budget Subcommittee on Information Technology in 1995, because nobody was looking collectively at what any departments or agencies were doing. We didn’t know what we were spending; what we had. We didn’t even have an inventory.
MR. EVANS: And that analogy of, if you’re building infrastructure or certain technologies for the long haul to be modified, adapted, used….we use the analogy of a proprietary software being like buying a car with a hood welded shut. Open source is just like buying a car today. If you buy a car with a hood welded shut, there’s one dealer who can service it. He or she tells you when you’re able to end of life…
SENATOR BOWEN: I don’t think you have to have the hood welded shut. You just need to have a computer processing but only the dealer has the manual.
MR. EVANS: Yes. There’s validity to that, as well.
SENATOR BOWEN: Okay.
MR. EVANS: There’s another stack here, I’m not going to read through it, that talks about why open source software that goes down the….improved security though, audit ability, people can verify. There’s a transparency about open source software, which is a very common word around Red Hat. We like to believe we’re as transparent as possible in our business, as well as our technology. Cost reduction is an obvious winner. There’s stories of ten times, twenty times, performance increase with a quarter and half the cost that are rampant in terms of utilizing open source software in governments, in businesses, around the world.
As Andrew mentioned, there’s an entire ecosystem building around it. Almost every major vendor and minor vendor and small startups are looking at open source and how do they either work with it, work against it in some cases, or help propagate it? It’s a major impact on the global technology market. And then there’s better choice provided, we believe.
I just want to show one slide, just quickly, we sell free software, which is often a funny way to say what we do when I am at speeches. But this is our recent quarterly. We’re doing pretty good. We’re profitable, and sold $73 million of free software last quarter. So there is a real business to be had.
And this is an interesting data point, as well. There’s a CIO survey generally of corporate CIOs that we did not pay for a sponsor. And in the last two years, Red Hat has come out as the number one highest value provider to the IT industry _______ demonstrates the value of Linux and open source solutions.
There’s all sorts of market data on the expected growth of Linux market share compared to Windows and Unix in the server market. It’s obviously a worldwide force.
This is a story of open source benefits of a survey from Forester Group. Why do you like open source software? Some usual suspects up there—TCO, acquisitions, modifiable code, more choice, more hardware choice, better quality. There’s endless numbers of these surveys out there.
This next point I call “demand and range.” I just thought it would be useful to kind of describe what we see as we look around the world of demand in interest in Linux and open source software. It’s every country and region in the world, and in some places, very extreme. There are some governments that are going beyond what we advocate. We advocate choice, we don’t advocate that you dictate something.
Another interesting dynamic is in academia, where almost every technical school, university in the world is now creating people that come out of school with Linux and open source skills, because it’s a great teaching tool to be able to see the source code and evaluate the source code. And that’s a market force that’s just phenomenal around the world.
The range of customers, it’s everybody in the world. It’s financial markets, teleco, manufacturing, governments. Government market is actually one our largest and fastest growing. I’ve got every cabinet level department in the federal government uses open source. And the bottom one is, the range that open source technology is covering everything from servers, entire networks, and clients. Kind of the entire range of the technology infrastructure. It’s providing, due to the flexibility and allowance for people to modify, it allows solutions to be built by people all around the world.
And just my last slide then is what we advocate, which is, choice and options, free competition, and open standards.
SENATOR BOWEN: Okay, a couple of questions. You mentioned that you are focused on the infrastructure space. Who are the industry leaders in the application space in open source software?
MR. EVANS: Some of them were on Andrew’s slide. There’s Sugar CRM is currently one of the better known ones. Maybe you can help me.
SENATOR BOWEN: Should I toss that question to you?
MR. AITKEN: Sure. And obviously there’s a whole host of different application categories. And one of the points that I wanted to touch on briefly, was how a company like Sugar, has leveraged open source to grow, and I think they’re a good example for commercial open source. So they’re approximately 18 months old. Today they have over 500 paying customers. One of the most interesting data points, I think, is after they launched the 1.0 release of their software, their first real release of their software, they told me the story that they came back into the office the next morning and their software had already been localized in French and German overnight, in less than eight hours. In three weeks it had been localized in 17 different languages. And within, I think, 30 or 40 days, they had 45 partners that wanted to sell their software around the globe. And this was all because of the open source nature of the software itself.
So other categories—I mentioned business intelligence, there’s financial applications out there. There are a variety of different companies.
MR. EVANS: In my mind, it’s fair to state that the application wave of open source is just in its earliest stages, as well. The infrastructure sort of wave has proven things, and has been functioning for eight or nine years.
SENATOR BOWEN: How would you counter the argument, or do you think it’s accurate, that if we moved from proprietary software to open source software with regard to voting, that we will put the proprietary software vendors out of business because they’ll have to give away their product?
MR. EVANS: Well, I believe there’s business models that could be adapted, for one. That the proprietary voting software vendors should be looking at what kind of business model can they adapt to, or shift to? As Andrew described, there are several companies who are doing that—doing dual models or shifting things, and there’s ways to get paid for providing open source solutions.
SENATOR BOWEN: Anything to add?
MR. AITKEN: Yes, absolutely. Today most of the top technology vendors, both hardware and software, are moving towards providing open source based solutions, or driving revenue through open source. And typically it’s either they’re open sourcing some software and deriving services revenue from it—service, support, maintenance, things of that nature. Or, they’re actually providing it under a dual license. And what that means is, they create an open source version that maybe for free under an open source license, and they have another version which is either the same version of that software which they create under a commercial license and charge for. Or, they create two versions of their software, one they give out for free under open source, and the other that has a few more value added features to it, which they charge for. And those are becoming very proven models.
MR. EVANS: And there’s also open source to a limited group of people, as well, some people might have called shared source that Microsoft and others. So there are many ways to skin the cat, as they say.
SENATOR BOWEN: Actually we’ll get into this with the next panel, but voting software is somewhat unique because unlike, for example, drivers license databases or various other applications at the state level, or web services in any number of state agencies that run on Apache, we only use the voting application typically twice every other year. So it’s a sort of an odd beast, and then we don’t use it again. So that kind of intense but occasional use seems to be a place where the model of having someone else be there with the personnel that are required to deal with it, makes a whole lot of sense, so that we don’t have 58 counties having an IT department on staff year-round for something that is going to be used twice every other year.
MR. AITKEN: Do you pay them based on a license or a per usage fee for the software?
SENATOR BOWEN: Every county has their own arrangements, so there’s no standard. And some cities also have their own elections software. So, some cities use county software, county elections capability, and some run their own elections on their own equipment and software. So we don’t have much in the way of standards.
We had no
standards nationally until after
MR. AITKEN: This may be somewhat applicable. There are some companies that produce open source software that run kiosks for large corporations, HR related kiosks, and they charge, I think, on a per usage basis, which might be something similar. So a company produces and sells the software. It’s installed on a kiosk for XYZ Company…
SENATOR BOWEN: That might be a good idea to drive voter turnout, paid by the use of the kiosk.
MR. AITKEN: There you go. To help stimulate the vote. When that kiosk is used, and then there’s a fee for that usage.
SENATOR BOWEN: All right. Let’s go to Anthony Hill. Thank you.
I know we added you to this panel late yesterday afternoon and I really
appreciate you coming. You have gone to open source software at
ANTHONY HILL: Okay. Well, thank you, Senator Bowen, for the invitation, and to the panel for the invitation, to be here and to participate on this committee meeting on open source.
I think as Senator Bowen