Purchasing Enterprise Software

The Technical Evaluation Process

So, your friends on the business “side” of the house have some new requirements? You were just put in charge of determining how to best meet these requirements and have decided that the tools you have in-house aren’t going to cut it and building the solution yourself just doesn’t make sense (I’ll save that topic for another post.) What do you do? I’ll help you walk through a process that is meant to save you time and money by reducing the risk involved in your decision and ultimately picking the right solution.

What Comes First?
First, you need to do some research on who the top vendors are in the space. To help you determine who these vendors are, you should check out what the analysts have to say about this. Garner and Forrester come to mind, and if they have an evaluation already done that addresses the software you are looking into, then you are in great shape. If not, this is going to be a much more difficult process than expected. So, let’s just assume this work has already been done.

Depending on how much time you have available to build a solution for the business, you’ll want to choose the top 3 to 10 vendors in the space to evaluate. I’d recommend taking a look at vendors that are well established (aka the “big boys”) as well as some of the smaller vendors. Most likely you will get a much different approach and vision on how your problem can be solved by looking at it from different sides. Chances are, the big boys are going to have a much bigger vision of the space, but will tend to be less innovative and the smaller vendors will have more leading edge features, but have a narrower scope to the solution. It is important to see both sides of this as your requirements may fit one approach better than the other.

Find Out What the Vendors Offer
The traditional approach here would be to issue an RFP (request for proposal) from all of the vendors you’ve selected from the above step. This would be wrong. Unless you have a long delivery cycle you will probably just be wasting everyone’s time. To build an RFP you will have to go out to the different stakeholders in your company to find out what their requirements are and write them all down as a list of features and functions they want to see. This is a very structured process and will eat up countless hours of your time. At the end of this process you will have a document or spreadsheet with hundreds of questions to send out to the vendors. Those vendors will then put in the minimal effort to answer “yes” to each of your questions. They will do this because they know that to answer “no” or be unable to answer a question at this stage will immediately eliminate them. From the vendor’s perspective the goal of an RFP is to get to the next stage.

So what do you do? Instead of building up a list of features and functions, have your business users describe the current problem and their ideal solution in their own words. Then, you should do the same based on your understanding of the technical environment that the solution will operate in. If the solution requires single sign-on, portal integration, etc. this is where your input comes into play. You shouldn’t feel compelled to rewrite the business’s requirements or ideal solution. This should be as unfiltered as possible so that the vendors have as much understanding about the requirements as possible.

Then, take the compiled problem statement and idealized solution that you’ve just written and send it off to the vendors asking for a written response from them outlining how they would propose to solve the problem and what the solution would look like. Ask them to include which products and services would be required in the solution and ask for them to describe how they’ve solved this problem at other clients of theirs. Don’t ask for them to scope the cost of software or hours to implement because you won’t get a straight answer and frankly, it is not possible for them to estimate this at this point.

If you don’t get a good answer back from the vendors or they are too vague, then push them to be somewhat specific about their vision. Also be sure to mention to them that if they can’t get more specific they won’t be selected to participate in the next round. You should use the written responses as your first method of filtering the vendors. Those that paint a solution that seems to meet your requirements and mention some of their products or packaged solutions should be moved to the next round. Those that don’t seem to understand what you are trying to do (after you’ve helped them by clarifying any questions they may have had) or are so vague you have no idea what they are proposing or have not put any effort into the response (by giving you some marketing boilerplate response) should not be included in the next round.

The Demo
The next round should be a series of phone calls to your down-selected vendors (if you started with 10, you have perhaps 4-6 in the next round) to schedule demos with them. At this point, all you are looking for are high-level, generic, demos of their solution to give you an idea of how they going about solving the problem you presented to them. This is meant to be generic because this will give you the opportunity to see how they view the general problem space and allows them to show you what their vision of a solution might look like, but generically. This is important because by seeing everything the vendor offers in this space might open up other possibilities you hadn’t realized. It also gives you the opportunity to see what they feel their differentiators are.

If the vendor is smart, they will ask to do some additional discovery before the demonstration. If they ask for this, it shows that they intend to take some interest in you as a client and want to invest additional time in you. Depending on how much time you have, you can invite them in for some in-house meetings to do this discovery where you can be more detailed in describing your requirements. A phone discovery session is also fine as it allows them to ask some questions to ensure that the demo they will be giving you hits the mark. Any vendor that doesn’t do this I would be suspicious of. Either they are so confident that they can meet your requirements in the first demo that they don’t need to ask questions, or they aren’t interested enough in you to go through the effort.

For the actual demo itself, I would recommend spending between one and two hours per vendor. This ensures they have enough time to show you all the features and functions of their product and gives you adequate time to ask all of the questions that come to mind. I also strongly recommend you see the demonstration in-person. Remote demos are very difficult to pull off and often encourage folks to get distracted. As for who should attend the demo, I would recommend that the key members of the decision committee attend, but no more. You can invite others to the later evaluation rounds. After seeing all of the demos you should now down-select your vendors to 2 to 3 and no more. This is because you will be asking the remaining vendors to spend a lot more time on your solution which in turn involves a bigger commitment from you and your organization.

The Custom Demo
At this stage, a lot of organizations would jump right into the Proof of Concept. I don’t recommend that unless the solution you are looking for is so unique and so highly customized that you don’t think the vendors will be able to mock something up on their own. Other reasons why you probably don’t want to do a PoC:
- You will need to be 100% available to that vendor for the entire duration. Typical PoCs last a week, which means you will need to set aside two to three weeks, just to run the PoC.
- You will need to have additional staff on stand-by to support each of the vendors during the PoC. This might mean System Administrations, DBAs, Network Administrators, and your Subject Matter Experts need to be available, on a moment’s notice, for those 2-3 weeks.
- You will need to invest heavily in preparation of the PoC, far more than for a custom demo. You will need to fully build out the use-case as well as define the decision criteria.
- Is your facility setup to have guests on site and accesing your systems for weeks at a time?

A custom demonstration only requires a fraction of the above requirements, but should have nearly the same result. The vendors should provide a glimpse of what your solution would look like using their software and tooling. But, to get here you will need to invest more of your time detailing the requirements as well as providing the vendor access to your business users so that they can do additional discovery. The result of this should be a custom tailored demonstration that the vendor feels will give you a feel for what the solution looks like.

Once you’ve viewed all of the custom demonstrations you may be ready to make a decision. However, if it is still too close to call, this is when you should broaden the audience and bring the vendors back in to present their solutions to the larger audience. This audience should be comprised of only relevant business and technical users of the proposed system.

In Parallel
While the evaluation is going on and you are in the demonstration phase, you should ask the vendors to provide a list of references, use cases, and white papers that describe their solution. This should give you an idea of how their software and services can be deployed under ideal circumstances. Remember, they are not going to print and share the projects that went bad.

When you get to the custom demonstration phase, ask the vendor to setup reference calls with a couple of their customers. This will give you the opportunity to get an unfiltered description of how they implemented the vendor’s software solution as well as find out how responsive the vendor’s support organization is. Again, be aware that anyone the vendor has you speak to is most likely the “best case” scenario, but even still this will give you a different perspective.

Additionally, you should be asking the vendor what their product roadmap looks like for the solution they are proposing. Will they give you access to their product management team to do a call on this? How open are they about sharing this type of information? The more access they grant and the more open they are the better.

The Decision
Ok, so if there isn’t a clear “winner” from the custom demo you will have to start factoring in the non-feature/function areas of the solution to make your selection. What does the future hold for the vendor? Is one more financially stable than the other? Is one more willing to put features in future versions of the product for you? Does one want to make you a reference customer and is offering to go the extra mile to ensure your success? Which product roadmap syncs up with your future direction? Which vendor can help you address other areas in your enterprise down the road? These are all factors that you should weigh when making the decision. At the end of the day, if the decision is too-close-to-call based on how the solutions look then I recommend you go with the vendor you would rather have a relationship with over the next 5 years because ultimately you will end up leaning on them at times and if you have a good relationship your project will most likely be more successful.