Purchasing Enterprise Software

The Technical Evaluation Process

So, your friends on the business “side” of the house have some new requirements? You were just put in charge of determining how to best meet these requirements and have decided that the tools you have in-house aren’t going to cut it and building the solution yourself just doesn’t make sense (I’ll save that topic for another post.) What do you do? I’ll help you walk through a process that is meant to save you time and money by reducing the risk involved in your decision and ultimately picking the right solution.

What Comes First?
First, you need to do some research on who the top vendors are in the space. To help you determine who these vendors are, you should check out what the analysts have to say about this. Garner and Forrester come to mind, and if they have an evaluation already done that addresses the software you are looking into, then you are in great shape. If not, this is going to be a much more difficult process than expected. So, let’s just assume this work has already been done.

Depending on how much time you have available to build a solution for the business, you’ll want to choose the top 3 to 10 vendors in the space to evaluate. I’d recommend taking a look at vendors that are well established (aka the “big boys”) as well as some of the smaller vendors. Most likely you will get a much different approach and vision on how your problem can be solved by looking at it from different sides. Chances are, the big boys are going to have a much bigger vision of the space, but will tend to be less innovative and the smaller vendors will have more leading edge features, but have a narrower scope to the solution. It is important to see both sides of this as your requirements may fit one approach better than the other.

Find Out What the Vendors Offer
The traditional approach here would be to issue an RFP (request for proposal) from all of the vendors you’ve selected from the above step. This would be wrong. Unless you have a long delivery cycle you will probably just be wasting everyone’s time. To build an RFP you will have to go out to the different stakeholders in your company to find out what their requirements are and write them all down as a list of features and functions they want to see. This is a very structured process and will eat up countless hours of your time. At the end of this process you will have a document or spreadsheet with hundreds of questions to send out to the vendors. Those vendors will then put in the minimal effort to answer “yes” to each of your questions. They will do this because they know that to answer “no” or be unable to answer a question at this stage will immediately eliminate them. From the vendor’s perspective the goal of an RFP is to get to the next stage.

So what do you do? Instead of building up a list of features and functions, have your business users describe the current problem and their ideal solution in their own words. Then, you should do the same based on your understanding of the technical environment that the solution will operate in. If the solution requires single sign-on, portal integration, etc. this is where your input comes into play. You shouldn’t feel compelled to rewrite the business’s requirements or ideal solution. This should be as unfiltered as possible so that the vendors have as much understanding about the requirements as possible.

Then, take the compiled problem statement and idealized solution that you’ve just written and send it off to the vendors asking for a written response from them outlining how they would propose to solve the problem and what the solution would look like. Ask them to include which products and services would be required in the solution and ask for them to describe how they’ve solved this problem at other clients of theirs. Don’t ask for them to scope the cost of software or hours to implement because you won’t get a straight answer and frankly, it is not possible for them to estimate this at this point.

If you don’t get a good answer back from the vendors or they are too vague, then push them to be somewhat specific about their vision. Also be sure to mention to them that if they can’t get more specific they won’t be selected to participate in the next round. You should use the written responses as your first method of filtering the vendors. Those that paint a solution that seems to meet your requirements and mention some of their products or packaged solutions should be moved to the next round. Those that don’t seem to understand what you are trying to do (after you’ve helped them by clarifying any questions they may have had) or are so vague you have no idea what they are proposing or have not put any effort into the response (by giving you some marketing boilerplate response) should not be included in the next round.

The Demo
The next round should be a series of phone calls to your down-selected vendors (if you started with 10, you have perhaps 4-6 in the next round) to schedule demos with them. At this point, all you are looking for are high-level, generic, demos of their solution to give you an idea of how they going about solving the problem you presented to them. This is meant to be generic because this will give you the opportunity to see how they view the general problem space and allows them to show you what their vision of a solution might look like, but generically. This is important because by seeing everything the vendor offers in this space might open up other possibilities you hadn’t realized. It also gives you the opportunity to see what they feel their differentiators are.

If the vendor is smart, they will ask to do some additional discovery before the demonstration. If they ask for this, it shows that they intend to take some interest in you as a client and want to invest additional time in you. Depending on how much time you have, you can invite them in for some in-house meetings to do this discovery where you can be more detailed in describing your requirements. A phone discovery session is also fine as it allows them to ask some questions to ensure that the demo they will be giving you hits the mark. Any vendor that doesn’t do this I would be suspicious of. Either they are so confident that they can meet your requirements in the first demo that they don’t need to ask questions, or they aren’t interested enough in you to go through the effort.

For the actual demo itself, I would recommend spending between one and two hours per vendor. This ensures they have enough time to show you all the features and functions of their product and gives you adequate time to ask all of the questions that come to mind. I also strongly recommend you see the demonstration in-person. Remote demos are very difficult to pull off and often encourage folks to get distracted. As for who should attend the demo, I would recommend that the key members of the decision committee attend, but no more. You can invite others to the later evaluation rounds. After seeing all of the demos you should now down-select your vendors to 2 to 3 and no more. This is because you will be asking the remaining vendors to spend a lot more time on your solution which in turn involves a bigger commitment from you and your organization.

The Custom Demo
At this stage, a lot of organizations would jump right into the Proof of Concept. I don’t recommend that unless the solution you are looking for is so unique and so highly customized that you don’t think the vendors will be able to mock something up on their own. Other reasons why you probably don’t want to do a PoC:
- You will need to be 100% available to that vendor for the entire duration. Typical PoCs last a week, which means you will need to set aside two to three weeks, just to run the PoC.
- You will need to have additional staff on stand-by to support each of the vendors during the PoC. This might mean System Administrations, DBAs, Network Administrators, and your Subject Matter Experts need to be available, on a moment’s notice, for those 2-3 weeks.
- You will need to invest heavily in preparation of the PoC, far more than for a custom demo. You will need to fully build out the use-case as well as define the decision criteria.
- Is your facility setup to have guests on site and accesing your systems for weeks at a time?

A custom demonstration only requires a fraction of the above requirements, but should have nearly the same result. The vendors should provide a glimpse of what your solution would look like using their software and tooling. But, to get here you will need to invest more of your time detailing the requirements as well as providing the vendor access to your business users so that they can do additional discovery. The result of this should be a custom tailored demonstration that the vendor feels will give you a feel for what the solution looks like.

Once you’ve viewed all of the custom demonstrations you may be ready to make a decision. However, if it is still too close to call, this is when you should broaden the audience and bring the vendors back in to present their solutions to the larger audience. This audience should be comprised of only relevant business and technical users of the proposed system.

In Parallel
While the evaluation is going on and you are in the demonstration phase, you should ask the vendors to provide a list of references, use cases, and white papers that describe their solution. This should give you an idea of how their software and services can be deployed under ideal circumstances. Remember, they are not going to print and share the projects that went bad.

When you get to the custom demonstration phase, ask the vendor to setup reference calls with a couple of their customers. This will give you the opportunity to get an unfiltered description of how they implemented the vendor’s software solution as well as find out how responsive the vendor’s support organization is. Again, be aware that anyone the vendor has you speak to is most likely the “best case” scenario, but even still this will give you a different perspective.

Additionally, you should be asking the vendor what their product roadmap looks like for the solution they are proposing. Will they give you access to their product management team to do a call on this? How open are they about sharing this type of information? The more access they grant and the more open they are the better.

The Decision
Ok, so if there isn’t a clear “winner” from the custom demo you will have to start factoring in the non-feature/function areas of the solution to make your selection. What does the future hold for the vendor? Is one more financially stable than the other? Is one more willing to put features in future versions of the product for you? Does one want to make you a reference customer and is offering to go the extra mile to ensure your success? Which product roadmap syncs up with your future direction? Which vendor can help you address other areas in your enterprise down the road? These are all factors that you should weigh when making the decision. At the end of the day, if the decision is too-close-to-call based on how the solutions look then I recommend you go with the vendor you would rather have a relationship with over the next 5 years because ultimately you will end up leaning on them at times and if you have a good relationship your project will most likely be more successful.

Your Enterprise Software Implementation is Going to FAIL!

And here’s why…

(Note: This is not intended to be a profile of any one customer I’ve worked with. Nor are the examples here verbatim of what I’ve seen. These are just general patterns I’ve encountered.)

I’ve been in enterprise software for 7 years now and although I’ve seen a number of really cool success stories using a number of different software solutions, it is the failures that seem to stick in my mind. Perhaps it is because I like to learn from failures to help my customers succeed where others could not that I tend to cling those memories of disaster. Often times it is not possible to pinpoint the source of failure, although most folks try to do so. Failure usually occurs for a number of reasons, many of which can be detected right at the beginning of the implementation. In fact, projects that fail are usually setup to do so (although not intentionally) from the start.

In the spirit of redneck comedy (or “blue collar”, although I think that term is more offensive, but I digress) you know your project is going to fail when…

If you are trying to implement more than one piece of enterprise software that isn’t “pre-integrated” or doesn’t have out of the box integration…

you are in for a world of hurt. The pain increases exponentially with each additional product in the mix. I’m not saying that you shouldn’t implement multiple pieces of software to form some sort of composite application. I’m saying that if you try and do this on the first release you are taking on a lot more than you can probably chew. Why not break the project into several iterations? Breaking down the implementation by doing a short release cycle with one product first, then adding a second component in the next release followed by incremental enhancements until everything is integrated is much more pragmatic. I know this sounds obvious and I can’t tell you how many heads bob up and down when I mention this approach. However, very few organizations actually do this. Why? The thinking goes that if you try and integrate everything at once then all of the bugs will be discovered early on in the process so you’ll then have plenty of time to fix them along the way. It never happens this way because the complexity is non-linear. For example, the complexity of combining 3 applications isn’t just 3X, it is really 9X (man-hours, cost, etc) because application A and application B are twice as complex on their own and integrating them adds two more levels of complexity. Add a third product into the mix and you add one more product complexity plus two more integration points When you get to that level of project complexity (and the long timelines and huge cost associated with it) you’ll find that either you have the worlds most patient project sponsors, or the project gets cancelled.

If you choose to “go it alone” because you have “plenty of smart people” on your team to implement the software…

you are fooling yourself and setting your team up for failure. Because of this attitude you’ve only given your team one option; to succeed. When they don’t this will cause major backlash with finger-pointing going in every direction (mostly at the software vendor) and will cause some of your best people to leave your organization out of shame or frustration. As a manager, how can you have possibly gotten this far in enterprise IT without spending money on software implementation beyond your fixed labor costs? Hey, I get it, enterprise software I hard, really hard. That is why most software vendors have consulting services and have extensive partner networks. We want you to succeed and recognize the fact that you are going to need help, if just on your first implementation. Please take it. If you don’t, I’m really going to want to say, “I told you so,” but that would be in poor taste. Chances are, you’ve taken the training and read the documentation (more on that below) but in my experience that simply isn’t enough. Asking for help isn’t a sign of weakness, it is a sign of maturity.

If as part of the sales cycle you’ve only seen me, the only product expert you are going to meet (because you aren’t using professional services or an SI), twice, and the contract is already signed…

you are already weeks behind the knowledge curve. Generally, IT organizations are curious to learn as much about a piece of software as they possibly can before they have to go off and implement it. They will also try and take as much free knowledge as they can get their hands on. The smart organizations will figure out that I (and my peers) am a free resource to use and abuse until the sales cycle is complete. This means product demos, deep dives, workshops, sessions of stump the chump, and even proof of concepts (the latter being a blog post for the future) are all fair game to learn as much about the product, and my experience with implementations of it as, as possible. If the natural curiosity to learn is not there, or you are too busy to deal with this now (see below), or any of the many reasons why you choose to remain in the dark are keeping you from getting a brain-dump, you are in trouble.

If during the two meetings we had together (see above) you spend most of the time “fighting fires” or dealing with “production issues”…

you probably don’t have your ship in any sort of order and adding another piece of complicated software to the picture is only going to make things worse. What makes you think you can maintain a new application if the old one(s) keep breaking? Adding one more layer of duct-tape is only going to cause more stress to the already overtaxed system that you and your poor staff will have to deal with. Chances are, you’ll be fighting these existing issues all through the implementation of the new software, which means your team won’t be as focused on the new implementation. Today’s botched implementation is next week’s “production issue”, thus creating a downward cycle that will be very difficult to get out of.

If, after having explained to you the best practices for implementation and choose to go with a fully custom, non-standard, and non-recommended implementation on your first project…

then I hope you have deep pockets; because to actually get that beast into production you are going to end up paying a whole lot more than you initially budgeted for with a lot less delivered than was promised. Did you run that use-case by anyone with extensive product experience? Did anyone warn you not to make things so complicated on the first project? Do you have cotton in your ears or do you think you just have “really smart people”(see above)? The very first time you implement any software the KISS principle should apply. Keep It Simple, Stupid. What is wrong with getting that first one out the door, even if it has limited functionality that you can improve upon later? Won’t the quick win on your first implementation buy you more leeway down the road when you want to do more complicated things? I know I’m being rhetorical here, but I see this far too often and it just doesn’t make sense to me. I’d opt for a guaranteed quick (if simple) win over a potential disaster any day.

If you’ve asked me where the documentation is and actually intend to read it. Or, if you’ve asked me how to setup a comprehensive test framework to run the software through its paces before you start implementing…

you are on the track to success. Software vendors don’t staff teams of people and spend thousands of man-hours writing documentation just for giggles. Everyone knows engineers hate writing docs, so why do software vendors go through all of this effort? So that their customers actually read them to gain a better understanding of how the system actually works, what the best practices are, and what to avoid. It is amazing how many support tickets get opened that turn into documentation questions as opposed to real product bugs. By actually RTFM you will be in much better shape to not only implement the solution, but also so ask the right questions when problems arise.

Additionally, if you have the foresight to plan for QA on your implementation before promoting it to production (you do have a sand-box, test, and staging environments, right? Sigh.) you are probably going down the right path. Thinking about QA doesn’t actually mean you are doing it right though. Did you spend 90% of the implementation time doing development and are saving the last 10% for QA? How firm is that go-live date? Well, either your “firm” date is going to slip or you are going to release one buggy application. Ideally, you are doing test-driven development. But short of that you should plan on spending at least 1/3rd of the project in QA and performance testing (you do know how to performance test, right?) before promoting to production. If not, well, those bugs aren’t going to fix themselves.

There are a number of other reasons why it implementations fail, but I’ve already ranted for long enough. Keep in mind that any one reason is probably not enough to bring a project down, but when you start combining several of them together the snowball of failure really starts to grow and that in turn has the same chance of success as, well you know…