There are two types of database deployment tools. These are generally categorized automatic and manual. The problem with the automatic kind, is that it can’t always figure out what to do. Here are some examples of things that automatic database migrations can’t figure out, but are fairly easy to code up manually.
1) When adding columns without defaults, what data should be used to fill in the values?
2) When splitting or moving columns, where does the existing data go?
3) When renaming a column, how does the tool detect that it isn’t just a drop and an add?
4) How should the script alter data when adding a constraint that renders it invalid?
5) When changing the data type of a column how should the existing data be converted?
6) What if data needs to be changed outside of or because of a schema change?
These reasons were paraphrased from this article.

Because of these issues, for large complicated databases or databases with a lot of critical data most developers end up choosing explicit manual migrations. There are several tools of this nature out there, but the most widely known is DbDeploy.

DbDeploy itself is a Java program (http://dbdeploy.com/, https://code.google.com/p/dbdeploy/wiki/GettingStarted). It is one of the few database deployment programs that supports both Oracle as well as moving forwards and backwards through the deltas. It is old (the second version came out in early 2007), but well used and respected by the community.

In 2007 DbDeploy.Net 1.0 was released (http://sourceforge.net/projects/dbdeploy-net/). It is called 1.0 even though it comes from the 2.0 version of the Java code. It was released and then kind of sat there on SourceForge because
1) .NET developers weren’t heavily contributing to Open Source in general
2) SourceForge as a code hosting service was becoming less popular.
3) It did was it was designed to do, and no changes are necessary unless someone dreams up a new feature.

Anyway, fast forward to 2009 and DbDeploy 3.0 for Java is released. Also during these years we are starting to see GitHub emerge as the Open Source market leader. Now here is where the problem comes in. Without notifying anyone there is GitHib repository created for the DbDeploy (https://github.com/tackley/dbdeploy) and the DbDeploy.Net projects (https://github.com/brunomlopes/dbdeploy.net). However, there is no mention of it anywhere, so unless you specifically go and look there, you wouldn’t know that.

In 2012, Robert May aka rakker91 ports the Java 3.0 version to .NET, calls it DbDeploy 2 (even though it came from the Java version 3.0) and posts it on CodePlex (a Microsoft open source host). But again, unless you know to look there, nothing.

In 2013 Gregg Jensen makes the first significant outside contribution to DbDeploy.Net 1 in a while. As he does this he notes on the original SourceForge page:
“dbdeploy.NET has been updated by the community on GitHub. New documentation and features have been added at github.com/brunomlopes/dbdeploy.net. I have used dbdeploy.NET for a while, and I like how it works so I contributed there. Gregg”
This is the first breadcrumb left so that someone from the outside world could actually discover something is happening to the project.

Soon thereafter (July 2013) DbDeploy.NET 2 (the new code base) is formally released on Codeplex.

So in short we have two different independently evolving code bases to choose from. This is a problem in itself, because it means there has to be time spent investigating which code base is the correct one to start from. I downloaded both versions and started poking around. Here is what I found out:
1) DbDeploy.Net 1: Unit tests did not run, and it uses a schema that is harder to support with older versions of Oracle (uses an identity column which would need to be a sequence).
2) DbDeploy.Net 2 (from Java 3): Dropped Oracle support. This is something that is often done accidentally in a rewrite, but this was due to a lack of an Oracle database to test against.

I think Oracle is actually one of the most important use cases for DbDeploy.NET for several reasons. The first is that there are a ton of legacy Oracle databases, where as a lot of SQL Server were written or rewritten with Entity Framework, which provides migrations out of the box. Also, there are a multitude of similar tools for SQL Server, but the database deployment tools for Oracle are lacking in many ways.

Anyway, I started tackling the problems in both codebases, and believe it or not it was actually easier for me to add Oracle support to the new program than it was for me to fix the unit tests and schema problems with the old program. So expect check-ins soon.

I don’t know why I have to blog about this. It depresses me and reflects poorly on our entire industry. It is 2014! Don’t we know better?!?

Apparently not. I have found myself *several* times in the past month, having to argue against a big rewrite of many thousands of lines of code. I am amazed and appalled that anyone still thinks this way, after *so* many articles have been written for *so* many years. Two of my favorites are Joel in the year 2000, and this more recent article that references one of my favorite cartoons (thanks to Lance for introducing me to the cartoon).

One thing that is different from when Joel wrote that blog post 14 years ago is that refactoring software is so much easier now. It is now *incredibly* easy, in addition to being much safer.

I don’t want to go through the same arguments again, because so many others have done it for me. Unlike some of the people I have to convince, I am not stupid enough to think that I am the first person faced with this decision, or arrogant enough to dismiss what those hundreds of other people have said. However, I will offer one small piece of advice. Often times the refactoring can happen *while* people are designing what the rewritten version is going to look like. At that point the code will be easy to change and it will be a much simpler process to add the new features.

This blog was cross posted on the Crafting Bytes blog at Refactor vs Rewrite (again)

I just finished forming a new company called Crafting Bytes with Brad Cunningham and Ike Ellis. We wanted to start taking on bigger projects using the techniques that have made us so successful as consultants. Over time we have noticed that we do much better with the projects when we take control of the project management as well as the development rather than simply augmenting the development staff. What is different about our project management style than the project management style of other companies that we work with?

One major difference is that we don’t use scrum. In a sprint developers spend a lot of time estimating the work. Estimating can be important when the estimate is used to determine whether or not the work should take place at all. However, in most cases companies were using the estimate to inform management when the product was supposed to ship, so that they could relay that information to the customer. It would be a better idea to relay the information to the customer *after* the work has been completed. It is much more accurate that way. The other reason managers were requiring estimates was to figure out how much work should be completed this sprint. So in other words managers were requiring estimates for the sole purpose of the project management methodology they were using.

The thing is estimates take a lot of time, and they are rarely accurate. Our thought was let’s forget scrum and just go with a simple Kanban board (from the Lean school of thinking). By doing this we can save ourselves countless hours of trying to figure out how much time things are going to take, and spend more time simply doing them.

OK, so that saves a couple days every sprint, but then what is the purpose of having a project manager at all? I admit that the project managers of many companies are totally unnecessary. You know the type, they spend most of their time polling individual people “are you done yet”. They could easily be replaced by voice recognition software that recognizes the word “yes”. This isn’t really project management, it is instead project reaction. Project *management* would be managing the work of the project, prior to it starting. In short a project managers job is to figure out which work is the most important and which work is so unimportant it doesn’t need to be done at all. Great project managers remove all unnecessary tasks, that is all tasks that don’t lead to working software, and prioritize which features are the most important for the business and the user. In short they control the prioritized list.

Thinking of project management as simply controlling the list of things that need to get done and prioritizing the most important simplifies the job of the project manager and helps the team achieve minimum “time to value” – a vastly underrated metric.

This blog was cross posted on the Crafting Bytes blog at Project Management As A Prioritized List

For the first time in my career I find myself writing contracts. After doing a little research about the traditional types of contracts, they seem to be inherently anti-agile, which goes against my way of thinking. After doing more research I found one type of contract which seems especially suited to my agile beliefs, but I realize that most businesses won’t understand why I write contracts this way. I am writing this blog entry in order to justify the peculiar type of contracts I write.

There are essentially two types of traditional contracts: fixed price and time and materials (also called labor and materials). Let me discuss these one at a time.

Fixed price contracts are where the product and the payment are agreed-upon before starting the project, and the compensation is paid regardless of the actual costs.

Fixed price contracts are risky for both parties. The business can end up paying way more than they should, and the software development firm can end up not getting paid for a significant period of time. For example, if something unforeseen goes wrong, the software developers could be working for months at the end of the project with no pay at all. In the other direction, the software team could convince the business that the project is going to take a long time to complete, either maliciously or unintentionally. Then, during the course of development the software engineers may find something which enables them to finish the project in a couple of weeks (e.g. open source code which does the bulk of the work) and sit on the project until the due date. Software developers have to pad or buffer the estimate to protect themselves against unforeseen circumstances, but if nothing goes wrong the project may complete early, once again causing the business to pay more than it should have.

In addition to the risk of paying too much money, or not getting paid enough, depending on your perspective, fixed price contracts have the potential to be used against either party so that neither party gets what they want. I call this hiding behind the contract. The developers want to get paid, but businesses can insist that they meant something else by the wording in the contract, and refuse to pay until what they were thinking is achieved. On the other hand developers can delivery shoddy software which “meets the requirements”, if no mention was made in the contract of quality, maintainability, stability, etc.

From a business point of view fixed price contracts are also difficult to manage. In fact they are more like gambling than management. In real life managers would never give their teams a contract and tell them to go build the software without allowing for frequent reviews to gauge progress and make adjustments. So if businesses wouldn’t act this way for their own teams why would they allow teams outside of their organization to act this way?

Fixed price implies one more thing which is bad for businesses: fixed goal. Often during the course of creating software, during a review for example, there is a flash of insight on the part of the business, and a resulting decision to move the product in a slightly different direction than originally specified. This direction is often more marketable, more user friendly, or simply meets the needs in a way that the originally specified product couldn’t. But if it takes more time to implement then obviously the developers are going to have a problem with spending more time for the same amount of money. Similarly on the software side, once some parts of the software are completed it can shed light onto other parts of the software, and result in drastic changes to the estimates. Instead in order to obtain a good software estimate, a lot of energy needs to be expended up front that can not be billed. This effort doesn’t result in any work towards the actual software, again a loss for both parties.

Hopefully I have convinced you that fixed priced contracts have a lot of drawbacks.

The second major type of contract is a time and materials type of contract. This type of contract basically states that the business will pay the software developers for the amount of work that they do until the project is completed. Time and materials contracts are only dangerous for the business, because the software developers are guaranteed to get paid for the amount of work they are doing even if it isn’t furthering the project in any way. However, a lot of businesses won’t accept time and materials contracts, for this very reason. Ultimately software development companies who restrict themselves to writing these contracts may lose a lot of business.

As I was contemplating this I remembered reading in Agile Software Development Ecosystems by Jim Highsmith, about another type of contract that fits into the agile philosophy a little better. In the world of rapidly changing requirements, the type of contract that makes the most sense is an iterative one, where each iteration gives the business a chance to give feedback to make the product better, and choose which features to implement next. The problems with the fixed price contract are that they are unchangeable and don’t require or allow for this sort of frequent contract revision with the customer. The problem with time and materials contracts is that they don’t tie the work software developers are doing to particular features in the product or hold them accountable for getting “real” work done.

Highsmith mentions a fixed schedule variable scope contract called a delivered feature contract. This style of contract states as long as there is work to be done, and the business is happy with the work delivered the project continues. There are two important points to this type of contract. The first is that there must be periodic reviews setup, to make sure that the software developers are making the business happy. The second is that there needs to be a prioritized list of features that the business needs developed which is the work to be done. If initially the business thinks that it requires a huge number of features, but as the project progresses they realize that they can ship with a smaller subset, the project will be over as soon as the subset is delivered. At the periodic reviews the company can critique the software delivered thus far, potentially correcting issues early on that would have huge ramifications later in the project. They also get to decide together what to work on next, so that if the business has an idea to change the product after the contract has been signed it can be done.

Delivered feature contracts also protect the software developer by paying for the work done on a feature, not a fixed price per feature. With that said, the most obvious question is how does the business know how to budget for a delivered feature contract, since there is no fixed price? In other words how does it protect the business against the problems of the time and materials contract? If written correctly estimates can be included so that they have some idea of how much money the project will cost. In addition the project can be terminated after any iteration, allowing the company who chose the wrong software development firm to get out early.

This blog entry draws upon the works and talks of many members of the agile movement including but not limited to Jim Highsmith, Robert Martin, Kent Beck and Scott Ambler.

Which is easier? Bringing someone up to speed on the .NET platform or bringing someone up to speed on your business? This is the question that project managers should be asking themselves when converting an old project such as MFC or C++/COM to .NET.

This is not the first release for .NET. Indeed it has been released for five years already (not even counting the large Beta period). The number of assemblies and types to learn is enormous. That is a lot of knowledge to absorb. In fact the longer companies wait to go to .NET, the more learning they have to do.

The productivity gains from .NET are significant, but think of all that time spent just getting back to the level of competency where you are right now. When starting on a major refactoring of the current product, it makes more sense to keep only those developers who have a ton of accumulated business knowledge. Then hire a bunch of expert .NET developers and do a lot of paired programming to get the knowledge to transfer between the groups.

In order to produce agile software there are really two components that have to stay agile: the people and the code (which is ultimately the product).  If either one doesn’t stay agile the other one is doomed.  If the people don’t stay agile then there is no motivation to change the code.  If the code is not agile then it doesn’t matter how much the people want to change it – it is very difficult.  Both are required for an agile software company.

How do we go about making sure that our people stay agile?  The more time they are able to think (and code), the more agile they become.  Developers work best “in the zone.”  It takes about 10-15 minutes of uninterrupted time to get there.  One way to keep interruptions from occurring is to hold fewer meetings.  There are three fundamental practices that should be instilled in all called meetings.

1. Identify the required people and include an agenda in the invite.
2. If those required people are not present within 5 minutes of the meeting’s start time, cancel the meeting.
3. Finally, the person who called the meeting should always summarize the meeting appropriately.  They should also somehow publish the notes and/or action items from the meeting (Sharepoint? in the bug tracking system?).

If that seems like a lot of work, it is.  That’s really the point.  Because you shouldn’t interrupt people’s valuable time without a clear important goal.  If it is worth calling a meeting, it is worth doing it right.  If that serves to deter the number of people that call meetings, then that is a positive side effect.

Another great way to keep people agile is by pairing them together, and encouraging them to discuss possible solutions to complicated problems.  Pairing has a number of benefits involving skills transfer as well as offering two different viewpoints on the problem.

Agile people have versatile skill sets.  They are generalists.  They work hard to learn new skills and apply those skills until they end up with multiple specialties.  Over time of course a skill might be forgotten, but they could pick it up if need be .  One of the best ways to encourage this is by rotating people throughout the department.  Instead of keeping your web developers and your database engineers segregated and fixed, shake them up!  Training and conferences also help give people versatile skill sets.  Basically trained people make less mistakes.

How can we keep the code agile?  Agile code is healthy code.  It is easy to change.  Unit tests are the best way that you can assure that your code is healthy, and refactoring is the best way to assure that your code is easy to change.  When you have good well-trained people working on code making lots of changes, it becomes easier and easier to change over time.  As you have new people or poorly trained people working on the code it tends to get more and more difficult to change or fragile over time.  As people are able to take their time and make sure they do the job right (have only 2-3 things to deal with at any given time) the code tends to improve, and as people are harried by deadlines and release dates they tend to do a poorer job.  Peer reviews can also help balance this effect.  The person peer reviewing the change has nothing on their plate right now except for looking at this code.  If they do a poor job and let lots of stuff go, it might affect the other things they have on their plate later.  If they do an excellent job, it’s not going to take that much longer than the poor review, and there is a positive impact to future tasks.  So they are motivated to make sure they catch items and get them addressed.

I think one of the key things to push for is a lightweight framework that always makes sure that developers are doing the right thing.  Any rule worth stating is worth enforcing.  Also, program managers should be setting aside time to make sure that their code base is agile enough to support the changes that are bound to take place.  To keep the code agile you need a person or in the case of a ton of code maybe even a group of people that go around looking for ways to eliminate code and make it simpler.  This person(s) can spend their time trolling through the code looking for inefficiencies and areas to make common.  Over time this will pay off in a sub-linear rather than exponential increase in complexity.

I obviously try and convert software projects to good practices when I can.  Things like peer reviews, unit tests, acceptance tests, refactoring, etc. are all practices that make a big difference in the long term health of your project.

One of the biggest areas of pushback I see is that the project is already underway, and they don’t want to change practices or go back and fix things that are broken right now.  Instead they will “fix everything in the next release.”  This type of short sighted thinking is far too common in our industry, and it is not just the fault of the project managers. Developers have a responsibility to make the project managers see the light.

I liken this mode of thinking to sailing a ship filled with cargo with a hole in the bottom of the boat.  It really doesn’t matter to me if you choose to bail out the water in this release or the next.  But plug the freakin’ hole! Please just stop any more water from coming in!  Releases are rarely delivered on time, functionality often changes mid project, and all the while you are accruing more and more technical debt.  If left alone long enough this technical debt is enough to sink the ship.  I am not asking you to take the time right now to go back and fix everything, but you *do* need to change your practices so that you stop accumulating water.  Eventually the project is going to release, and the ship is going to have to sail again.  That is when you are going to notice whether or not you took a little time to fix the hole.  If you didn’t, then getting that next release out is going to be really, really difficult.  The code is going to be difficult to change, and you are stuck bailing out an entire boat filled with water before you can even start to load the next cargo.  To make matters worse maybe some good people leave or move on to other things after the release, because they know that they are surrounded by water.  Change is always hard, but often good practices take less time than bad practices.  By fixing the hole now, you aren’t really slowing down the ship, and the ship may even live to sail another voyage.

There are numerous agile methodologies: Scrum, XP, Lean, Crystal, DSDM, etc.  And they all agree on 4 fundamental things.  They prefer:

• individuals and interactions to process and tools
• working software to documentation
• customer collaboration to contract negotiation
• responding to change to following a plan.

While there is value in the ones on the right, the ones on the left are more important.

It is especially hard to be agile at an FDA regulated company.  Sometimes I think they prefer the ones on the right.  In fact in a couple of cases I am sure of it.  I know they value documentation over working software, because documentation has yet to harm anyone (other than perhaps the person writing it .  I also think that they prefer processes to people, and following a plan rather than responding to change for the same reasons.  And although the software won’t kill anyone, it is not likely to be usable by anyone either .

Despite the differences in opinion there are ways to make it work.  Often this involves defining a plan that allows for change and using easier and lighter weight methods of documenting – like pictures and videos.

The ultimate goal of a good employee should be to put themselves out of a job.  It seems counterintuitive, but it is true.  I mean every single employee from the janitor to the CEO.  The truly great CEOs understand this.  But even the janitors – if they could find away to make the floors perpetually clean, think of what an accomplishment that would be!  The only feasible way they could do this is through some form of automation.

We as developers should strive for the same thing.  You should strive to do such a great job, to write such good code, that does exactly what it is supposed to do, is so easily understood, easy to maintain and add features, that the people that hired you to do the job in the first place won’t need you anymore.  I say this knowing that there is no way they will let you go.  Instead what will happen is that you will be promoted and given new responsibilities to automate.

As a manager you also should be striving to make things run themselves.  One of the best ways to do this is schedule time to develop automation.

Those of you who know me understand how strongly I feel about this subject.  There is just too much work to do manually.  The more software products you develop the more you have to maintain.  But the goal is obviously to build as many profitable products as possible, and they obviously need to be maintained to some degree.  So where does that leave you?  If you had to leave part of your staff on every product that you have ever developed you would end up with huge bloated teams where communication is a nightmare.  So automation is really the only answer.  This is a huge part of agility, making it easier to do common tasks.

So like the Robert Heinlein short story about the man who was too lazy too fail – anywhere that you see an opportunity for automation, take it.  I think a fundamental part of the responsibility of any member of a software team is questioning why you do something.  Part of this is asking yourself if there is an easier way than the way being suggested, one that requires less overall work.

Often that easier way will be automating some repetitive task.