Integration: The Cost of Using Someone Else's Library

I don't do much Ruby on Rails development anymore, but Andy lives right next to me at PC-Doctor. He does.

Recently, he's run into an interesting problem. I've seen the problem once before in a completely different context.

Once might be coincidence, but if you see the same problem twice, then it must be a real problem. :)

Ruby on Rails

If you've got a product to develop, it's normally better to use someone else's library whenever possible.

Ruby on Rails makes this easy. They've got a fancy system to download, install, and update small modules that can be put together cleanly and elegantly to create your product.

It's a wonderful system. It works, too.


Andy discovered that it might work a bit too well!

He's got a medium sized web application that uses a bunch of external modules. He wrote it fairly quickly because he was able to pick and choose modules from a variety of sources to solve a lot of his problems.

Unfortunately, he had to upgrade to a newer version of Ruby. That means that he's got to look for problems in each of the modules he installed and find a version that works with the new version of Ruby.

Some module maintainers are faster than others, of course. Not all of the modules are ready for the new version of Ruby.

This is a problem that doesn't scale happily. As the number of modules goes up, the chance of one or more modules not being ready goes up.

As Andy discovered, this means that an application can become painful to update.

I phrased my title as though Andy might not have been doing to the right thing. I'd better be honest here, though. If one of his modules can't possibly be updated, then he's still better off rewriting that module. The alternative is to write all of the modules during application development.

Andy did the right thing. The pain he had while updating was minor compared to the alternative.

Ruby on Rails makes it extremely easy to combine large numbers of modules from different sources. The problem can be duplicated any time you get large numbers of independent developers working together.

Boost

The Boost libraries seem to be suffering from the same problem.

Boost doesn't put a lot of emphasis on stability, either. Changes to libraries are encouraged and frequent. Versions of the library aren't even required to be backwards compatible.

The end result is the same as Andy's problem. One library will change a bit, and that change will have to ripple through a bunch of other libraries. It can take a while to squeeze each contributor enough to get them to update their library enough to put out the next version of Boost. (Boost.Threads was the worst case of this. The developer disappeared with his copyright notice still in the source files!)

It's hard to blame either the release manager or the contributors. They're volunteers with paying jobs, after all.

The end result is still unfortunate. It now takes about a year or so to release a new version of the framework. Some libraries end up languishing unreleased for a long, long time because of this.

Boost has gone through a lot of releases. This makes it really tempting to look at this quantitatively. :)

To the right is a chart showing the number of days between major releases. This is, of course, a silly thing to look at. What defines a major release? There was only 5 days between version 1.12.0 and 1.13.0, for example.

The lower numbers on the chart show the number of libraries that changed with each release. There is a slight upward trend to that as well. Clearly, newer releases contain more new stuff in them than the older releases. Furthermore, not all changes to libraries are the same. Some of the more recent changes are substantial.

Despite all of that, I'm going to claim that the release schedule is slowing down over time. There are many reasons for this, but one of them could well be the same problem that Andy has.

Before a release goes out, there is often a plea on the Boost developers' mailing list for assistance on a few libraries. Those calls for help are proof that the size of Boost is slowing it down. If they have more libraries then they'll have more libraries with trouble.

Early versions of Boost had extremely light weight coupling between the different libraries. More recent versions are significantly more coupled. As developers get familiar with other Boost libraries, they are extremely likely to start using them in newer libraries. It's almost inevitable that the coupling increases over time.

The developers for each library continue to mostly be volunteers who don't always have time to make prompt updates. Getting all updates to all libraries to line up at the same time can't be easy.

Commercial Projects

Both of these examples involve open source projects. Andy isn't building an open source application, but he is relying heavily on open source modules. Boost is entirely open source.


An open source project is going to end up relying on volunteers. It's really hard to manage volunteers! Is it any easier on a large commercial project?

I don't have any direct experience with this. I've never been a part of a big company with hundreds of people all working on the same thing.

Is the problem unique to open source projects? I've got no data, but I'll make some speculations.

Some fraction of open source developers are volunteers with other jobs. This isn't true for commercial projects.

A developer who's spending their free time on a project will have to schedule their time around a higher priority project that's paying them. According to this theory, this dramatically increases the spread in the amount of time required to complete their job.

Conclusions

The problem probably isn't unique to open source projects, but I suspect that it's worse for them.

Ruby on Rails encourages using large numbers of independently developed modules. This model will exacerbate the problem.

I'd love to hear from someone who's got experience with large projects. The problem gets worse with big projects. Too bad I don't know much about what happens with them.

A Theory of Scheduling Low Priority Work

PC-Doctor delivers an enormous number of different products to different customers. Each customer gets a different product, and they get frequent updates to that product as well. Delivering these products requires complex synchronization between dozens of engineers. We've gotten great at scheduling the most important work. Our clients love us for that.

However, the low priority projects get released significantly less reliably. Until recently, I'd assumed that this problem was unique to PC-Doctor. Based on some extremely sketchy evidence from another company, I'm going to release my Theory Of Scheduling Low priOrity Work (TOSLOW).

Let's suppose that we've got a project (L) that is not as important as a set of other projects (H). Here at PC-Doctor, we like to deliver our important projects on time. In order to do that, we often have to drop what we're doing to get something done on a project that needs work now. That means that someone who's in the critical path for completing a project in H will not be able to do any work on L. Things may be somewhat more extreme here at PC-Doctor than they are in a typical company, but I suspect that L will always have trouble causing a delay in H.

Now, it's possible to get work done on L. For example, we could hire someone just to work on a specific project in hopes that it will, in time, start making money. That'd be great. You'd manage to get 100% of a small number of people's time for your low priority project. The trick is that the people working on L do not have any responsibilities that are needed by the high priority projects so they can be scheduled independently of H.

Until discovering TOSLOW, I'd assumed that this would mean that, eventually, the project would reach completion. The people working on L might not be perfect for each task that needs to be accomplished, but they can do each task to some extent. They're devoting all their time and energy on that project, so eventually they'll learn what they need and get to the next step.

If that assumption is correct, then L will get accomplished. Furthermore, it is likely that L can even be scheduled accurately. I've never seen this happen here at PC-Doctor, though.

Here's why. If a project requires interactions with a large number of systems that are being used by the projects in H, then the person working on the low priority project will have to get some resources from the person in charge of each of those systems.

There's a chance that those resources will be obtainable. The low priority project's schedule will be determined by the least obtainable resource. In principle, you'll always get the resource you need eventually. If one person is always the the rate limiting step for H then something should be changed to improve the scheduling of H.

However, even if we can say that L will eventually complete, if we want to schedule L accurately, we will have to be able to predict when we'll get time from each resource being used by H. In order to make this prediction, you'll have to understand the schedule of H. Project L will have to wait until resources being used by H are available. Here at PC-Doctor, this is particularly bad. Engineers working on our main projects tend to work closely together. That means that a large number of them are working on the critical path. In other words, getting an engineer to work on L requires H to be unable to use that engineer. Perfect scheduling is not possible, so this happens frequently. However, this means that L's schedule is coupled to the errors in H's schedule!

It's possible that I'm a project scheduling wimp, but I'm going to claim that, as long as the schedule of L is tightly coupled to errors in an unrelated project's schedule, then you shouldn't even try to schedule L. In the worst case, you should just say that the project will eventually reach completion, but you have no idea when it will be.

TOSLOW can be summarized in this way: If a project is low enough priority that it cannot preempt another project's resources, but it still requires some of those resources, then the error in the low priority project's schedule is going to be large.

An important corrollary to TOSLOW is that low priority projects will always be late. Errors in scheduling almost always cause things to be late rather than early!

Okay... If you're working on a low priority project like the ones that I've described, then I haven't really helped you. All I've done is give your boss a reason to kill your project. How can you avoid the effects of TOSLOW?

Just being aware of the problem will put you ahead of where I was on a low priority project that I've done for PC-Doctor. If you're aware of it, then you can start work on the stuff that requires interaction with higher priority projects immediately. In fact, I'd say that, as long as you've proven to yourself that your project is somewhat possible, you should spend all of your time on the interactions. After all, pretty soon you'll be waiting for resources and can spend some time on the meat of your project.

Recognizing the problem helps in another way as well. If your boss is aware of TOSLOW when the project starts, then you may be able to get your project's priority temporarily raised when it needs some help. This is exactly what the Windows kernel does to avoid thread starvation. (The reason for this is actually to avoid priority inversion, but it's got nice side effects for low priority threads as well.) If a thread doesn't get scheduled for a while, then its priority will get a limited boost. That's what you need to ensure your project doesn't end up waiting indefinitely like this one.

This post was originally published on the PC-Doctor blog.

Developing a New Framework

This post is a bit of a change for me. I'm actually going to write about my work for PC-Doctor! I'm actually a bit embarrassed at how rare that's been.

I want to talk about how to design a brand new framework. It's not something that everyone has to do, and it's not something that anyone does frequently. However, there's very little information on the web about the differences between creating a library and a framework.

I've been working on a framework here at PC-Doctor, and I've worked on a few others in a previous job. I'll admit that I'm assuming that all similarities between these projects will be true for any new framework.

There are three things that I want to see from a new framework project. After I get those, it becomes more difficult to generalize across framework projects. The first framework I developed had all three of these aspects mostly by accident. My current framework has a tighter deadline than the previous one, but it still has all three of these to some extent. I'm a strong believer in all of them.

Goals

All of the frameworks that I've created try to do something new. This makes requirements gathering extremely difficult. If no one understands what the framework will accomplish, you almost have to decide on your own what it'll accomplish. You'd better be right, though!

Management will typically have a short term goal that they want accomplished with the framework. They might even be able to come up with a set of requirements. Don't get trapped by this. A framework must be much more than that. It enables programmers to use a common language to describe a class of application. Learning this language requires a significant time investment. If it only solves a short term goal, programmers won't be able to make that investment. Your framework has to solve a significant problem in order to be worth the investment required.

In another sense, however, requirements gathering for a framework is easy. You can get some details of the requirements wrong without damaging your users' ability to figure out what they want to do with it. After all, they typically write code using the framework. They can just write a bit more code than they should and get things working. Later iterations can use this experience to refine the design, but this won't happen until late in the design of the framework.

Instead of trying to find formal requirements, I prefer to find something that I call "goals". This is closer to a set of use cases than requirements, but they're reformed so that they look like requirements.

After developing a framework in my previous job, I saw how critical these goals were in the framework development. A good set of goals can let you make decisions quickly and accurately about design problems. If the goals are relatively small and simple, then they can be applied uniformly and accurately throughout the life of the project. That means that you're likely to fulfill the goals.

As an example, I'll give you a few of the goals for my current framework:

Goal: Relatively inexperienced developers should be able to use the framework to do somewhat sophisticated things.

This goal has driven a large fraction of my decisions on my current project. In my vision of the future, there will be hundreds of mini-applications using this framework. Having an enormous number of these applications would allow us to do some really amazing things, but that's simply not possible if I have to write them all. In fact, that's not possible if PC-Doctor engineers have to write them all.

This goal is designed to allow us to recruit developers who are more interested in the problems that the framework can solve than the techniques required to write code with it.

If this goal were a requirement, it would state something about the usability of the framework. Perhaps it would say how long a typical programmer would take to develop their first application with it. In its current form, it's almost completely useless to our QA department. That's not what it's for.

Goal: The appearance of the UI elements created with this framework should be directly modifiable by Chris Hill, our art department.

Again, this is me recruiting other people to do my work. Stuff that Chris creates looks about a hundred times better than stuff that I create. Looking good is an important goal for us since we want our product to be fun to use rather than merely possible to use.

This is a better requirement than the previous one. This can be verified directly.

However, it turned out not to be that useful of a goal. While this goal verified some of the design decisions that were made for other reasons, it hasn't been all that useful to me directly. This might be a good thing, actually. If the goals work well together, they shouldn't have to conflict.

Goal: A future iteration of the framework should be portable to a variety of other platforms.

This goal is a good example of a goal that mostly gets ignored. The architecture does indeed support Linux, and a lot of the code should be easily ported to alternate platforms. However, it's hard to pay attention to a goal that isn't needed in the next release. PC-Doctor has some tight deadlines; we don't get to develop frameworks out of sight of our paying customers.

Not all goals have equal importance, and not all goals are actually useful. I don't consider this a failure, yet. Try to have as small a set of goals as possible. The more you have, the more difficult it will be to accomplish all of them simultaneously.

Okay, I've got a few other goals, too, but that gives you the idea. These are extremely high level goals. You could call them requirements, but that would be stretching things a bit. They really aren't that formal.

These goals are extremely important to the project. Choosing the wrong ones can kill the chance of success. Choosing the right ones will make design decisions extremely easy.

Usability

The next thing to worry about is the users of the framework and its usability for those users. I've talked about this before. The things I say in that post are even more valid for a framework than they are for a library. Go ahead and read that. I'll wait.

In my current project, I've got two types of users to think about. The first are mentioned in my goals. These are the developers who will write mini-applications with it. The second group are my coworkers who will help me maintain it.

Unfortunately, this framework has ended up putting the two groups in conflict. I frequently end up adding complexity to the framework in order to reduce the complexity of the API. In fact, I frequently go to great lengths to simplify the API slightly without worrying about adding half a dozen states to the framework's state machine.

It's still too early to tell if this will be a success. However, there are some preliminary indications:

1. Our first client to see the early results of the framework liked it and used it enough to have a lot of feedback for us.

2. Stephen, the product manager in charge of the first product, is currently busy writing a mini-application to test an optical drive. He doesn't complain much anymore. (I need to get him to complain more, actually. He's my only usability testing!)

3. Soumita, the only other programmer to actually dig into the framework so far, complains loudly. While I feel bad for her, making the internals simple wasn't one of my goals. I'm a bit worried now that it should have been, though.

To summarize, the UI of a framework isn't any different from the UI of an application. Use the same techniques to improve it. Above all, take the usability seriously. Frameworks are complicated and require significant investment to begin using them seriously. People will not do that if it's not easy to use.

Tools

You want to make your framework easy to use. You can do that by making a nice, clean API, or you can do it by making tools that allow users to ignore the ugly parts of your API. Both possibilities should be considered.

Boost and XAML are two frameworks that take this principle to opposite extremes. It's worth looking at both.

Boost has a wonderfully clean API. The tools that they've created suck. (Boost.Jam and BoostBook are horrific messes that make me cry.) The framework itself is a joy to use because you don't frequently touch their tools. This is a valid approach to framework design, but it's not the only approach.

Microsoft's XAML, for example, is the complete opposite. XAML is completely unreadable and extraordinarily difficult to use by itself. XAML data files are as readable as object files. However, Microsoft doesn't want you to use it by itself. They created a set of tools that let you completely bypass the XML obfuscation that XAML requires. The tools themselves are clean and easy to use. Again, this is a valid way to approach framework design.

I prefer something in between, though. Make sure there are some tools to help users deal with the worst parts of your framework. At the same time, make the framework itself clean. Solve all aspects of the usability problem using the most effective tool for the problem.

For my current project, I didn't have time to create any tools. However, I did manage to make a lot of the code that users create editable with CSS and XHTML tools. There are a lot of great tools for web development. All I had to do was enable my users to use these. The jury is still out on that decision, but I'm still optimistic.

This originally appeared on PC-Doctor's blog.

The Cost of Complexity

This article is going to have more questions in it than answers. It's about a problem in software development that I'm not sure I've worried about enough. I've certainly thought about it for specific cases, but this is the first time I've tried to think about the problem in general.

My main question revolves around the cost of complexity in software. There is certainly a large cost in making software more complex. Maintenance becomes more difficult. Teaching new employees about the project becomes harder. In the end, you will get fewer engineers who understand a complex project than a simple one.

Unfortunately, almost any non-refactoring work will add to the complexity of a project. However, some changes can have a large effect on the complexity in a short period of time. Adding a new library or technique to the code base, for example, will make it so that the new technology will have to be understood by people working on the project.

What I really want to know is how much can this cost of complexity be mitigated? Besides switching libraries to add, what can be done to decrease the cost? My question is based on the assumption that some complexity is essential. So, given that you're going to add a new library to the code base, for example, what can be done to reduce the cost?

Make it fun?

Some types of complexity are actually fun for programmers to deal with. After all, if you don't like learning new things, you wouldn't last long as one. For example, a new library may be complicated, but it may also do some really nice things. If it's a fun library to use, then is its cost of introduction reduced?

Unfortunately, fun is difficult to predict. Certainly a hot technology is going to generate more interest with engineers. For example, I'd much rather develop a new mobile application using Android than WAP. WAP is probably a lot simpler, but Android is breaking news. WAP is dying. Would I choose Android over WAP just for this? Probably not, but I suspect that Android's problems are reduced a bit because of its novelty.

I looked reasonably hard for some discussion of this. I couldn't find anyone who wanted to admit that this could be a significant factor. Am I all alone here? Somehow, I doubt it. I prefer to believe that everyone ignores this.

Of course, without some measurements, it's just speculation. Hmmm... Maybe you could make some sort of measurement based on fake job ads and resume counts?

Ease of use

No one seems to think of libraries as something with usability considerations. However, an API is really just another interface that happens to be used only by programmers. If a library is easy to use, its cost will be reduced. What can be done to make a new technology easier to use?

Microsoft is pretty good at this, actually. Lots of people are going to complain at that statement because they've got a long ways to go. However, they put a lot effort into making high quality tools and documentation. Heck, they even created a publishing company to create how-to books for their libraries. They put a lot of energy and time into getting things working that aren't related to the API itself. (It's too bad their APIs frequently make me want to kick kittens.)

Libraries that do this well will make themselves cheaper to use. Once a library is chosen, though, what can be done about making it easier?

When I first came to work for PC-Doctor, I was told to create a new web application using Ruby on Rails. None of us knew how to use either Ruby or Rails. It turns out that the online documentation for Rails was pretty close to miserable. (I sure hope they've fixed that!) We bought many copies of some O'Reilly books on the subject, and this helped a lot.

Spend a bit of money to educate engineers. Different people learn in different ways, though. I love to read. Andy, our current Rails expert, is a big fan of some Rails videos that he found. Find the right support for each developer. Buy whatever training is required. The cost of a book is pretty low when compared to a developer who is forced to stumble around the internet looking for answers.

Genuine need

If everyone understands that a library really is needed, then people will be a lot more interested in learning it. This may be related to the fun factor mentioned above.

Complexity that seems like a bad idea afterwards is frustrating. Poorly written code is a great example of unneeded complexity. I've seen first hand what large amounts of horrible, old code can do to someone's moral. Andy, I'm thinking of you...

What did I miss?

Let me know. As I mentioned, I'm way behind in thinking seriously about this subject.

This originally appeared on PC-Doctor's blog.