A Patch in time

Software engineering is a notoriously difficult discipline. You get the customers requirements, then write the code, only to find that they forgot to tell you something. You modify the code, then they ask for another change. You add that modification, but it doesn't quite work correctly - they gave you the incorrect sequence of events, or you made a spelling mistake. Each iteration of change brings you closer to the desired result, but then there is another problem caused by an external factor - a change in hardware, an updated driver that's not compatible, a new legal or regulatory condition.

So for as long as there has been software, there have been modifications to software. In the early days of mainframes, these were added directly to the code on the machine by the programmers. As PCs became more common, and more software was being produced, it became impossible for the programmer to make the changes and most users wouldn't know how to. So the concept of the patch came about - a means for the programmer to make a modification to code and then issue it to the end user, without having to completely re-install the software.

The biggest example of this of course is Microsoft. Back in the 90s, they developed a process of patching and slowly began to automate this to ensure that as far as possible, the end user PC would be protected against software faults of whatever type. This works well for the average user, but within a business, it can be a bit of a PITA. Imagine having 500 PCs all suddenly accessing the Internet to download the latest patches - it would cause a lot of problems with bandwidth. Now imagine the problem with 5,000 or 50,000 PCs.

Now we don't have quite that number, but it's still an area that we need to control. We looked at the Microsoft options for automated patching, but weren't completely happy. About 3 years ago, we found a software product that actually manages the patching process and does it very well - and not just for the OS, but for lots of other software products as well. It downloads a single copy of each of the patches and places them in a repository at each one of our sites, then automatically scans and updates the machines in a controlled way. We've found that it works very well indeed - and it saves us a great deal of time.

Unfortunately, it's not possible to add SAP into this process. It appears that they are developing a process for managing patches thru Solution Manager, but as our system integrator people didn't install it it, we've got a rather laborious manual process instead. We are at the early stage of setting up a Solution Manager system and I hope that we will be able to use it eventually, but it has to be fitted in amongst other tasks.

Now when the consultants first arrived nearly 3 years ago, they told us that they would have the first system (DEV) set-up and working within a week of the original project launch date ready for the BPOs (Business Process Owners) to start working on the sandbox client. They also said that the other 2 systems (QAS, PRD) would then be completed within another 2 weeks. Unfortunately, the initial project launch meeting was delayed by 2 weeks and they didn't start work on installing the DEV system for a further 3 weeks. This wasn't completed straight away and it was a further 6 weeks before it was ready for use. The other 2 systems weren't even started at that stage and in fact, neither was ready until about a month before our original go live date, some 9 months after the project launch.

At the time of the first delay, we didn't even have data in the QAS system, let alone the PRD system. As it had taken so long, I raised the question of establishing if there was a need for patching the software, but they just brushed the question aside. I did some research on the topic, and found out about the transaction SPAM (Support Package Application Manager I believe) - a rather unfortunate name for adding patches!

When I first ran the transaction, I wasn't really sure of what I was looking at - there seemed to be a number of green items in the list of applied patches, but rather a lot of yellow ones as well. After further research, I discovered a description of the process to add these items. It wasn't entirely accurate. but it was sufficient for me to turn most of the yellow items to green and I was rather pleased with myself. Unfortunately, the same process had to be carried out in each system - a lengthy process. But after several weeks, I had finally sorted all of the items out, including a update to the SPAM transaction itself and a kernel update as well.

It later turned out that the items in the list were just a series of patches that had been loaded when the systems were installed, but the guy hadn't completed the process. I then found that there were a lot more that hadn't been applied that weren't in the list. Unfortunately, I haven't found a way to work out what patches are missing apart from looking at the patch number and searching on the SWDC (Soft Ware Download Center) of the SAP portal for the ones with the next numbers.

In the early part of last year, I started to download these patches - I then hit another snag. After a certain date, all new patches have to be confirmed before they can be downloaded. This has to be done within a Solution Manager system - as we don't have one, a bit of a problem. We have a way around this, thanks to SAP support, but this was the point at which I realised we would have to have a SolMan system.

As if that was enough of a stumbling block, a few months later, SAP introduced a new system of "Maintenance Certificates". These have to be downloaded in the same way as a normal license, but they only last for 3 months. If you don't have a valid maintenance certificate, you cannot apply a patch. It turns out that none of the consultants were aware of this, and I've since found that there are some SAP personnel that didn't know about it.

Despite all this, I started to get the patches applied to the systems. The DEV system was completed without issue right up to the last patch - but that then caused a problem. It turns out that the consultants had made a configuration change but neglected to advise anyone. Adding the patch caused a serious problem with a couple of processes. They said that they would fix it and tell me what they had done, but I heard nothing more. In the mean time, they asked that I not patch the other two systems.

Now that was about 10 months ago. I sat down recently and worked out how many more new patches have been issued and it is a heck of a lot. I know that some of them will only take about 10 - 20 minutes to apply, but others will take 2 -3 hours or more. I did test with one on the QAS system - it caused an issue. However, thanks to a response from the forum on the SAP Developers Network, I now know what is required to fix most issues that occur during the patching.

The problem is that I think we are now going to have to follow a very tedious process for many months to come. Each patch will have to be applied to the QAS system, wait to see if breaks anything and then work out what is needed to fix it before we try on the production system, This has to be fitted in amongst other jobs and the PRD system can only be done at weekends. As you can imagine, this goes down well with the family!

The estimate is that there is about 80 - 100 hours of work just to get the QAS and PRD systems to the same level as DEV. Probably 6-8 weekends if we are lucky and work evey weekend - however that will not happen, so it's more likely to take over 4 months. After that, then with a bit of good fortune, we may have our SolMan system running and we can see if there is a less labor intensive process.

Looking over the parapet

When I first started as a junior manager, one of the more senior people made a comment to me - when you are working, every so often you need to lift your head up over the parapet to see what's going on. It's very easy to get tied up in the daily work and the various issues involved in that and forget to take a look at the bigger picture. So I thought that I would take a break from writing about the daily problems that we face on our SAP project and a take a more helicopter view. (Of course there is a chance that I could get my head shot off!)

Prior to 2006, each of the separate business units was operating its own systems - and there were a lot of those. Of course, this lead to problems - systems didn't integrate well (if at all), data was not transferable, they were awkward to manage and maintain and required considerable administration. Although the staff didn't see this, the were also costly to operate - in most cases, we had people employed just to manipulate data, without adding any value to the work that they did or benefit to the company.

The senior managers knew that this was an issue - some of the departmental heads also realized this (although most didn't and probably couldn't have cared less anyway). It was fairly clear that the situation had to change and that there was a prime case for an ERP system. The main arguments were straight forward - one time data entry instead of multiples, consolidated data processing, cost savings from reduced labor costs, and more agile response to market trends and customer demands. It was also thought that better and more informed decisions would be available throughout the business, streamling the decison making process. Eventually, the choice was made to go with SAP.

Over the past few years, I've read a number of books, white papers and various other materials - many say the same thing, that implementing SAP is a Business Change project not an IT project. I would totally agree, but would add that this is true for a lot of what we would normally refer to as an "IT Project". For example, we installed a new telephone system, an IT project, right? Wrong! - a business project as it changed the way that a lot of people worked. We've also installed a new WAN link and that must be an IT Project. No - again, it changes the way that things are done throughout the business so it is most definitely a business change project. In so many cases, there is an "IT project" but that is a smaller part of a larger business change project.

When the consultants organized the very first meeting, they gave a PowerPoint presentation which I still have. They made this very point about the SAP project being a Business Change project and there it is on slide 17, bullet point 3. Unfortunately, they didn't emphasize this and a lot of people still saw it as an "IT Project".

The problem is that so many people automatically assume that they are not involved in an "IT project" as it will be too technical, and immediately switch off. I suspect that happened to a number of the members of our project team. That did change eventually, and I think that most of the project team now accepts that it is not just IT. However, that is not the case for all managers and certainly not all staff. Far too many still see the project as purely an "IT Project" and treat it as such, and I think that this is an area where we have not done very well.

There is no question that we have tried to get everyone involved in the project - we've allocated resources, supplied relevant information, and provided the means for everyone to play with the Sandbox system so that they can learn about it. People are encourage to use the training materials to learn and make constructive criticism. Considerable efforts were made to ensure that everyone was involved - but depsite this, there are still senior managers, key users and other staff memebers that really don't know what they are supposed to be doing. No matter what we say, they still see it as an "IT project" and therefore nothing to do with them.

So what to do? I wish that I could answer that. Clearly it has to come from thetop, and the CEO needs to make sure that all senior managers are on board. It could be argued that we could have done a better job of communicating with staff. I won't accept that we haven't tried, but I woulkd have to acknowledge that we haven't gotten thru to everyone. I think that this will have an impact - without everyone at least trying, it will take longer to get everything working as it should do.

Unfortunately, I suspect that this will have an impact on our future use of SAP and will mean that we won't get some of the payback that we should - or at least, not as soon as we should do. But then, we are dealing with real people and this is a common issue for many projects and not just endemic to our company or SAP. People are creatures of habit and hate change - and there are those that will be atagonistic to any proposed transformation, no matter what the benefits. But then - that's life!

Just keep rollin' on

I've not had time to post in the last couple of weeks. I wish I could say that everything is going well and life is calming down. I wish I could say that.

In fact the last couple of weeks have been pretty chaotic, with various problems. With the amount of time that has been taken up by the SAP project, a lot of other tasks have been pushed to the back, and I had hoped that now we are live we might be able to start getting some of these jobs done. In fact, the SAP work seems to be taking over all of the time that we have, and at the moment, we seem to be still fire fighting rather than being proactive.

Among the issues we have at the present are a large number of items that we have been waiting for the consultants to address for ages now. The system has suffered a number of crashes and some of the ABAP dumps have been collected and sent off for analysis. In one specific case, SAP support identified that the code had been changed and that they were not responsible for dealing with problems caused by personalised code.

I can understand that, but the reality is that we haven't changed a thing - none of our staff would know how to. I have a rough idea, but I wouldn't want to try to change anything at this stage. It's obvious that one of the consultants has made this change - the problem is, we don't know who, they won't admit to it, and they have kept no records of who changed what or why. There have been several other crashes that I have not yet sent off to SAP - I intend to do this on a regular basis as it seems it is the only way to identify what has been changed.

Here is a classic tale - I went into the sales office earlier in the week to look at a problem with one of the staff. Whilst she was talking to a customer, I was looking around the office and saw one of her colleagues hard at work. The person concerned had an invoice printed off on her desk, and was working in Excel. While I watched, she added a logo, some lines and started to put numbers on the page - then I realised that she was recreating the invoice in the spreadsheet. I had to ask what she was doing - it turned out that the customer needed the invoice but that it was coming out with the wrong amounts / values. The only way around this was to create a invoice thru Excel.

Back at the beginning of the project, we gave the consultants a list of the output documents that we needed along with samples and designs. They had 3 people working on this over an 8 month period and we still don't have all that we need. We have several that should give us similar information, but filtered and sorted in a slightly different way - but in fact give us completely different information. It also has to be said that we've found that we cannot rely on many of these documents as they just don't produce data that is even close to being correct.

Now to be fair, we do also have a problem of our own creation. We uploaded values for the sales prices last year just before we went live. However, we now find out that some of the prices uploaded were in fact incorrect - one of the sales staff responsible for dealing with certain key customers had given us the wrong data and we actually uploaded the incorrect values. That has been made worse as we now need to change those prices, and he should have given us the data for the beginning of the year. Instead we finally got the files on Monday - but with part of the data missing. He had gone through the data and removed all of the key index information so we now have no way to tie the new prices to the correct products.

I've also experienced a problem with our internal Project Manager - he has worked incredibly hard over the last couple of years. However, he has also picked up bad habits from the consultants. He has access to the profile "SAP_ALL" which gives him the permission to whatever he wishes. Now I have no problem with that in the development system, but not in the production system.

He was shown by the consultants how to unlock the production system and instead of making changes in the development system and then transporting those according to the correct SAP procedure, he has made the changes directly in the production system. This lead to a major problem as he had created a new transaction, but I couldn't assign to a role, as it didn't exist in the master system. On top of that, he regularly creates new SQL scripts and tests them in the production system and it causes problems for everyone else as the script is poorly written and ties up system resources. He just won't accept that what he is doing is wrong.

One other thing that I would note - in the past couple of years, I have had almost no contact with SAP apart the Support desk. Since Jan 1, I have 3 separate phone calls from different personnel at SAP. They seem really eager to sell me more services!

Happy new year...

...to everyone - I hope that you had a great Christmas. I must admit that I have been taking it real easy over the last few weeks, but now at the start of a new year, it's time to start work again.

On this occasion, there is a topic that I particularly want to cover. I would suggest that it is actually relevant to a lot of projects, not just SAP. It will make the difference between a bad implementation and a good one - and between a good one and a great one. And the topic is...

Training!

I'm a great believer in training, and not just a brief run thru of "press button A, press button B" etc. Staff that are properly trained can identify when something is wrong and can fix a problem before it gets out of hand. Because they make fewer mistakes, they require less support which leaves us free to do the things that help make the technology work for the staff and the business. I believe that in the long term, money spent on training is an investment and pays off many times over.

Unfortunately, some of the senior management don't quite see it that way. Their view is that staff don't need to be "IT trained", just know enough to do their job. I can see an argument for not spending time on training someone on aspects that they will never work on, but I really feel that in the case of our project, we really needed to get more staff trained earlier and in more depth - doing so would have made them feel part of the process, and would have given them a sense of ownership that is partly missing.

However, there is also one particular area of training that I want to identify, and that is for the IT staff themselves. When we started, not one of the team had previously worked with SAP - we had some basic SQL skills and between us had worked with a number of products. To begin with, we were completely reliant upon the consultant firm for everything. They insisted in installing the software, and gave us some instructions in doing some of the tasks. They provided some material which they said was SAP standard training manuals as well as some other items that they created themselves - mostly powerpoint slides with a couple of screen shots and no explanation of what was being done or why.

Over the last couple of years, my team have worked hard on trying to learn the product. We had a go at doing an installation ourselves, and when it didn't work, we found a number of online resources that pointed us in the right direction. We also bought a ton of SAPpress books that provided some real good information and help. We found out how to perform upgrades and carried those out without any guidance. We also had some basic knowledge of problem solving and managed to identify some key items which meant that we were able to manage the systems ourselves without having to keep bothering other people.

But all thru that, I felt that we could have done better with a better knowledege of key areas. So I decided a while back to take a look at the SAP training courses. These tend to be a tad more expensive than some others, but not outrageously so - and there are a number of training centers about. However, I had heard a few comments from a couple of people that had attended one of these and what they said didn't make me feel too comfortable about it.

I decided that I would book myself on one course, just to test it out - I felt it would be worth $1500 for a 2 day course just to see for myself the real situation. I decided to go to an actual SAP training center rather than one of the others that were cheaper, as I though it would work out better, even tho' it was almost the other side of the country.

On the first morning, the course started with the trainer doing quite a bit of "housekeeping" - almost the first 40 minutes were about various aspects of the training arrangements, the course, the health & safety etc. I must admit that I began to feel that perhaps the training was just going to be "Death by Powerpoint". However, very quickly after that we got onto the actual material and then it became obvious that the course was well worth the time effort and money.

The trainer was an SAP employee of many years standing and very knowledgeable. He used some slides to illustrate process, then performed the task using an SAP system, before then giving everyone on the course the chance to do the work for themselves. Although he had quite a strict agenda to complete, there was opportunity for the trainees to ask questions - in almost every case he had the answer immediately. I gave him a couple of questions that he couldn't quite answer, but he then used the break periods to research the issue and then immediately after, he would cover the question in detail.

There weren't too many people on the course which was good from my point of view as it meant that we had plenty of support from the trainer. I also found it interesting talking to the others - they were all from completely different industries and backgrounds so there wasn't a lot of cross over, but even so, it was good to discuss how we all did things.

When I got back, I was able to use the new knowledge almost immediately. I spent sometime with a couple of the guys and went thru' some of the material with them - together we found an answer to a problem that had been bugging us for some while. I felt that there was no question, the course had proven to be a winner and I thought that it would be useful to get one of the others on a course as well.

The one that I chose for the second experiment was a full 5 days course - and there were a few comments from people about the cost. On the morning of the first day, the guy I sent phoned me at midday to say just how impressed he was with it and that he had already begun to learn stuff that answered many of the questions we had. After that, he called me virtually every evening to tell me that he had found out yet more things that we needed to know. When he got back, we discussed some of the issues, and it became very clear that these training courses would prove to be absolutely vital to us.

We now have several other courses booked up at the same center - I'm not going to take a chance on any other. The courses are spread out over a period of months so that we aren't trying to cram too much into a short period, and I'm trying to spread the courses amongst the staff so theat everyone gets a chance. I've even suggested that some of the other staff could go on a course - I've found one that would suit someone from the production team and they sound keen to go.

It could be argued, that we should have done some of this training last year or the year before, and that's possibly valid. But I would say that in my case, and that of my staff, we agreed that we found the courses so valuable because we already had a certain level of experience. If I had attended my course 2 years ago, it would probably not have been of the same benefit as I wouldn't have been able to tie the material in to the work that I have to do.

Ultimately, I suppose that it comes down to the individuals - some will obviously get more out than others. But having had a couple of the courses now, our team are definitely impressed with them and keen to do others to build on what we have learnt. It's important to justify the cost, and it may not always prove to be so beneficial. But for me, there is no question - money well spent and with the plans for the future, the company should benefit in both the short and long term.

It's the process stupid!

One thing that everyone says about SAP - the processes can be long winded. However, you have to understand the reason behind that. The software was developed for use by very big companies and to meet the requirements of very restrictive legislation. As a result, many of the business processes seem to require more work than you would find in other software packages.

As I work primarily on the admin side, it's worth mentioning that the process of applying user access is particularly long winded. But there is very good reason for this - when you have 10,000 plus users, it is absolutely crucial to give all of these people the appropriate permissions. But working out what is correct can take a long time. And it has to be said that this has caused some friction amongst our project team.

Our basic AD permissions have been set-up over many years. They are not perfect by any means, but generally people have the necessary access and are kept out of those areas that they should not see. If we have to make changes, it usually only takes a few minutes and it is a pretty straight forward process that can be carried out by anyone in the IT team at any time during the day.

Unfortunately SAP is not so simple. A few months ago, I found out about Central User Administration (CUA), and we wll be looking to implement this at some stage, but at present we don't use it as it was not set-up at the beginning of the project. User admin requires details to be maintained in each system which is awkward and time consuming. The actual access permission is based upon a series of roles - permissions for transactions and authorisation objects are added to the role, and then user is placed in the role. A series of "transports" allows the roles to be copied between systems, so these should be the same in each of the systems.

However, the process is also designed to allow the change(s) to be made and then for these changes to be checked - first in the development system, then the test system, before it finally arrives in the production system. This is to ensure that any such changes are appropriate and don't do something that they shouldn't do. It makes a lot of sense and for the larger companies, I can see that this would be absolutely essential. For us, it is a very tedious process and we are not doing it as we should.

Unfortunately, even tho' the project team have been told about the correct process (repeatedly), they just don't get it. I've noted many times that I have been asked to allow a particular person to have a particular access permission - when I apply it, I've indicated that this then allows someone else within that role the same permission. We could have people able to do specific tasks that we would rather they didn't. After the first few months, I set-up a process to try to get these changes authorised properly. It sort of works, but even now people will try to bypass the process.

One other thing that I've noted - when we started, we set-up the roles based upon the job role descriptions that we use within the business. This was the advice from the consultants and I agreed that it made the most sense. However, having been doing it for some time now, I feel that we might need to look at this again as we have a lot of overlap in what various people do - the number of staff is limited, so most people actually do 2 or 3 "jobs". There is a good argument for changing some of the permissions so that instead of being applied to a "job role", we might have a "process role".

For example, we have sale clerks with various transactions, but there are a couple of these transactions that might also be requested for other people not in sales for various valid reasons - they could be used by accounts, by shipping, by purchasing, by production, as well as a couple of others. This is in fact, 15 roles in total. At the moment, these permissions have to be added to each of those roles, and after testing, they have the same permissions - potentially, we could have just 2 roles with those permissions, and then apply people to them in order to do the work and it would work just the same. Obviously, more work to set-up to begin with, but going forward, possibly much better as it would require less admin work.

Oh well, another day, another dollar!

Almost lost for words

I don't like to hear people swear. You hear it too often - so called "celebrities", sports or music stars using language that would make a drill sergeant blush. Often, the argument is that they are just passionate about the specific view they are putting forward, but I feel that in reality, it shows a lack of intellect. They swear because they don't know how to convey a point of view. But despite my views, this last few days, I have done more than my fair share of swearing.

Let's go back to the beginning. Some 2 1/2 years ago, the consultants started putting in the SAP system. All they did was install the basic software and addons - they didn't apply any of the patches or updates. At the end of last year, they made a comment that it was for us to perform these updates. It took me a while to get the information that I needed, but in the early part of this year, I managed to carry out a process, to apply the patches that they had downloaded, but not actually put in place.

Subsequently, I went onto the SAP Service Market Place Download Center and found that there were a great many other patches that they had not downloaded. I started work on downloading these, but after I reached a certain point, a message appeared to say that all patches released after a certain date had to be confirmed through SAP Solution Manage before they could actually be downloaded. As they had not installed a SAP Solution Manager system, this caused some problems - however, I was able to get around this eventually. As SAP now require Solution Manager to be used, I felt that we should get it installed - we have a copy of the software and the existing license allows us to use it, so I saw no problem.

I got a copy of the relevant SAPpress book and downloaded some material on the product and its use - it became clear that Solution Manager offers a lot of other functionality that we would find very useful and from conversations, it seems that SAP are going to increase the requirements for running the product. So we've spent a little time in between other jobs, getting the software up and running.

Having finished this over the weekend before last, I decided on Monday to take a look thru the product. It runs thru the SAPGUI and looks much like the main ERP but with a few different menu items. I glanced at a few of these, not really going too deep into the various areas - but then I saw something that caught my eye. When I opened part of the menu item for Solution Manager, I saw the phrase "ASAP Focus".

Those of you that have been following me for a while will know that I had a brief exchange with Michael Doane (http://sapsearchlight.blogspot.com/2009/08/sap-implementation-projects-still-crazy.html), a consultant with many years experience of working with SAP. He described a methodology to carry out an SAP standard implementation called "ASAP" which he advised should be used by all system integrators (consultants). I had not heard of this and it appeared our consultants didn't use it. Michael was a bit frustrated (and I could see why) as this is supposed to be used by everyone.

And here it was - a full description of the processes, documents to assist, presentations, step by step guides, glossary, checklists, issue registers, and a complete roadmap for the whole implementation project. Basically everything that we could have used and would have found useful - instead we had to develop our own systems. The more that I looked at this, the more frustrated I became, and of course, I then started to really let rip with the bad language.

So now we are starting to look at getting the Soution Manager installed properly. We will have to get someone in that knows about it, and get one of our staff trained up. It's just a pity that it has taken so long to find out about the product. It would be interesting to see what the senior management from the consultants has to say on why they didn't use it, but I dare say that I will not be allowed to ask the question.

Belated happy thanksgiving everyone.

Where does the time go?

I've just realised that it is nearly a month since my last update and about 7 weeks to the end of the year. Where does the time go?

We are still struggling with the SAP system, and now the consultants have all gone so we are very much on our own. We have a list of items that they still haven't gotten working, and our CEO spent over a hour on the phone to their director to try to get them to accept responsibility to fix these items, some of which have been on the issues list for 2 years.

I managed to resolve part of the problem with the invoicing - we now get the invoice run every day. At a meeting earlier in the week, our project manager said that this had totally fixed the problem, but after the meeting, he admitted that about half of the invoices are still not coming out right for one reason or another. Our finance manager did express some concern that without invoices going out, we won't have money coming in, and this could cause serious issues - we could potentially run out of liquid cash by the end of 1st quarter next year if the situation doesn't change.

In addition, it appears that there are a lot of jobs that haven't been invoiced at all - one of the guys did a quick check and it appears that this is actually an internal problem. People are completing a task, but it's not getting marked as complete, so the system won't process the job. Unfortunately, we can't just force it thru as there is no easy way to tell if the work has been done or not - it generally takes a couple of hours to confirm, by searching thru various paperwork.

There is still an issue with purchasing - jobs get re-scheduled, but the due date for items is not being adjusted correctly and several times, we found ouselves waiting for goods, or items were being delivered for something that has been put back acouple of months. It's more of an annoyance than a serious problem, but as part of the overall project doesn't make it look good.

The system is still not giving us many of the reports that we need. Various people were told to use different transactions, variants of specific reports etc. but many of these don't really seem to give us what we really need. Some of them seem to provide vey large amounts of data that is actually of no value what so ever. I had hoped to be able to spend some more on this during the project, but just never seemed to have the time.

As a result, I've booked one of my staff onto an SAP training course - it's way across the country at one of their actual training centers, and we want to see if the courses actually help; if they do, we may book some more. We do need to get better information about using the system, and we don't want to have to keep paying huge amounts every time we want to make minor changes. The courses are expensive, but if we get our people better trained, then it will prove worthwhile - having invested so much, we have to make the best of it.

Have to go, as we're off to visit my wife's parents - they're going on a cruise next week. Never thought that I would be envious of them!

Oh boy!

I definitely spoke too soon... the question is where to start?

On Monday, there was a meeting about some of the deliveries. It appears that there is a big problem with some that have been sent out on the wrong dates. After considerable analysis, it appears that one of the processes used to check availability of products is re-setting any date that has been entered for delivery, and making it due 7 days after the inquiry. So a load of orders have been rushed thru production and despatched - but in a number of cases, they have arrived well before the actual date the customer wants.

In some instances, they have hung onto the products, but have asked that we accept delayed payment, and to keep them happy, we have had to agree to that. In a couple of cases, the product has been refused and we will have to re-deliver later on (in some cases, next year). Meanwhile, production are upset because they have had to do rush items and it has caused some scheduling issues.

To fix the immediate problem, the Sales Manager sat down with all orders and has gone back over each item and re-entered the required date. However, after his first attempt, one of his staff had gone back in to check the availability using the same process and once more, it had reset dates. As you might imagine, he was pretty pissed about it. They are now checking dates on all outstanding orders each day to try make sure that it doesn't happen again.

We also have a major issue with some stuff that was marked for export. The guy responsible asked for the items to be put on the system, but unfortunately, the overseas client had already placed the order - although the system is supposed to flag duplicate purchase order numbers from the customer, it still isn't doing so as I mentioned last week.

The goods were made twice and as they are non-standard items, we are not going to be able to sell to anyone else. It's cost us about $10,000 not including shipping. To make sure it doesn't happen again, they have developed a manual system of checking each order - at one point they asked if I could create an Access database to monitor the orders!

Finance are not doing well either. A consultant had set up a process for running invoicing automatically - but so far, it has only produced about 1 invoice in 10. The staff are trying to get the rest processed asap, but it seems to take ages just to do one order. They are concerned that if we don't get invoices out, we get no incoming money. I know that we have managed to maintain a reasonable level of cash flow previously, but that could change in a few more weeks if we are not careful.

A big problem has occured at one of our other sites. They were struggling anyway - people were taking too long to process anything at all. I spoke to the GM on site and he was getting really quite anxious - he did indicate that they might have to start turning away orders as they just couldn't get the work done.

This has been made worse as he has now lost all but two of his office based sales staff - they just upped and quit in the middle of the week, saying that they'd had enough. We've been asked to send staff to him to help out, and it is causing some major issues with arrangements. I did also hear that a couple of others have also indicated their intention to leave - this could get really serious.

One good thing tho' - a decision was finally made about the CRM system. After 2 years of work, it's been agreed that the SAP CRM doesn't meet our needs. The way that it runs is just too slow, and the sales people think that the interface is really nasty (it doesn't show what they need on the main screen). It also doesn't produce the information that our Sales Director said was a fundametal requirement.

We have had 4 different consultants workng on the CRM since the beginning - and basically, we have nothing to show for the time that they spent on the processes. The SAPGUI part runs OK, but they said that it had to be used through a web browser - and that was where the problems were. The speed of loading was really poor, up to a minute to move between screens at times.

Over the last 2 years, they have made all sorts of excuses - not enough memory, issues with SQL settings, AV products, the only thing they didn't blame was the phase of the moon! A while ago, they indicated that they wanted to get some more people in to work on the problems, but that they would want more money to do so. The CEO was not impressed - so he has made the decison to forget about the CRM, and the sales staff are happy with that.

The consultant company sent us some documents recently about arranging for support after the go live process. We are alreading paying for SAP support (close to $100,000 pa) and they want us to pay money to them as well. They won't actually come up with a price, but just give loads of .ppt slide with the "benefits" for having them as a support contact. From what I have seen, it could double our support costs - but there would be no reduction in our staff to compensate.

Oh well, tomorrow is another day!

Did I speak too soon?

The first couple of days this days this week went OK - pretty much like last week. But then we hit some problems.

The issue with the documents printing out extra pages is still going on, although we have one of the consultants back in to help us. He's been looking at the specifics, but hasn't found anything so far. He originally suggested that we had faulty settings on the print server, but after making us double check it all, he had to admit that there is nothing wrong there.

On Wednesday, one of the staff in sales was putting orders on the system and suddenly realised that the prices were coming out incorrectly - she only spotted this as she has been doing the job for close to 15 years and can tell if a price is close or not, which these weren't. They spent the next 2 days checking this out but so far, no-one can see where it's going wrong. All we know is that the prices loaded were correct.

We also had a problem with the production plan on Thursday - for some reason several orders were entered twice (we haven't yet found out why) and this caused the production schedule to change to deal with an unanticipated demand. The system is supposed to detect if a customer's reference number is entered twice, but it didn't flag this up as it is supposed to do.

I've been busy with changes to user roles - I'm still getting just under a dozen a day. It's starting to get tedious, but one of the consultants said that this is quite common. He indicated that one project he was on had changes being made to the user roles every day 3-4 months after their go-live. I sure hope that's not the case with us as I have lots of other work that I need to be doing. Most of this is stuff that has been delayed because of the SAP implementation and we need to be getting on with catching up.

I've also heard some complaints from senior managers. They want specific reports daily / weekly and what they are getting is not what they want. We were told by the consultants that all of this was standard and they gave us transaction codes for reports - but it turns out that some of these don't actually give us what we asked for. One in particular is a report that indicates sales invoiced, but the figures that it is producing make no sense at all. We also had a labor report that seems to indicate that efficiency levels are about a quarter of what they were a few months ago.

On the plus side, people are starting to get to grips with the program. I don't get so many people locking themselves out because they entered the wrong password. They are still slow at using it though, but that's probably because they are still finding their way around; it's going to take a few months for them to pick up some speed. We had a few complaints about speed on a couple of occasions, but we were able to identify that someone else was running a report or other process that was tying up resources - hopefully, we have now put a stop to that.

In particular, we found some workflow items that had been set-up by a consultant that were failing every time they ran. We had run into this before and with the SAP support had found out how to correct this; we were able to do the same this time.

In addition, we had some feedback from the SAP Earlywatch check. They basically indicate that our systems have more than enough processing power / memory to run the program. This contradicts what the director of the consultants tried to tell us some months back (he insisted that we didn't have enough memory). They also made some suggestion about settings which we have now implemented, and some others about security permissions which we can't just yet.

My staff and I now have a basic routine in place to check on the system performance although I think we need a bit more experience to be able to understand the results better. Unfortunately, we have nothing to judge the results against - I have no way of knowing if we are doing a good job or not, or if the results are good, bad or indifferent.

Oh well, such is life. I suppose that these are mostly the types of teething problems that everyone gets with a new product.

Week 1

Well, the first 7 days have gone by, and so far things are not looking too bad. If it carries on like this, then I for one will be quite pleased.

First thing on Monday morning, I was in very early. I'd made a point of getting as many people as possible to logon in last week to make sure that they knew their username / password and that they could actually get access to the production system. However, I suspected that there would be a few phones calls and I wasn't dissappointed! I had about 5 people had managed to forget their password and completely lock themselves out of the system - I was able to talk them thru getting back on. This was fewer than expected, so I was quite pleased.

Later, I had a few questions about access permissions - although we had spent a lot of time working these out, it was clear that there would still be issues after we had gone live. Some of the decisions proved to be a bit optimistic and it was found that we needed to give some of the lower level staff access to more than had been originally agreed. Again, I expected this, so it wasn't that big a deal for me - but it was for some of the managers, as they want everything done imediately and with SAP that is not always possible.

We've developed a reasonably satisfactory process for getting these changes authorised, but people still try to bypass the process and get very irate when I send their requests back to them with the instructions. However, they will learn eventually. I am getting more than I expected, about 8 -10 a day. Hopefully that will tail off in the next week, but I'm not convinced it will as I'm sure that they didn't check all of the variants prior to go-live.

We had one major discussion - a few weeks ago, there was a request for a change which I said was a bad move as it could cause a potential problem. That request was then reversed a few days later as our Operations guy found out for himself just how bad it could be. But then a week later, he changed his mind again - we needed this to allow the project team to push things thru. Then there was huge problem caused by one of these guys using the permissions inappropriately, so it was taken off - and then added back on again a week later. We've had an SAP Earlywatch report and they identified that this was major issue and that we should remove this permission - so I had to sit in a room and explain to the CEO why the project team had been given it in the first place!

There's one rather odd problem with the Sales documents - for some reason, they are printing an extra blank page in between the normal pages, so a 3 page page document comes out as 6 pages. We've got one of the consultants coming in ths week and it's been highlighted that they should look at this - I suspect that it's a formatting issue, but I haven't had the chance to work with some of these documents so I'm reluctant to play around with it at this stage.

The SAP production system is working well. So far, no complaints about speed. It's clear that some staff didn't get enough training as they are taking a lot longer to carry out a function than they should - again, I suspected that this would be the case. I felt that many had not had sufficient actual hands on work, but it was difficult to fit this in. I'm sure that they will pick up speed, they just need to concentrate on using the product.

One thing that I'm bothered about, is that some staff are still using the old legacy systems. They said that they were just referring to old data, but why would they need to? The same data is in the new system, and is just as available. I think it is a bit of a comfort blanket, and as such, I feel that it is probably OK to leave them alone for the present. However, I don't feel that we should let them use these older systems for too much longer as the data will quickly get out of step.

Anyway, so far, so good. There have not been the more serious problems that I dreaded, so this weekend I've had the chance to relax and take things easier. We are not out of the woods yet, but if it continues as it all did this week, then I for one will be quite relieved and very happy.

Here we go

Well, that's it. No more prevaricating, no more delays. Tomorrow is the day - SAP go live day. No further hold ups, no excuses. It doesn't matter if we are ready or not, here we go.

The last few weeks have been crazy - the amount of work, mind numbing. Everything else went out of the window apart from the SAP project. All calls from suppliers and cold callers were turned away - all meetings cancelled apart from the project team meetings. And the last couple of weekends, we have all worked our butts off, to do the best we can to make sure we are ready.

There have been a few issues with the data load - it started OK, but then one set failed and it took several hours to find the problem. It turned out that one of the consultants had made some changes in the Development system which had been transported to the Test system but not the Production system. As a result, the data load worked in one, but fouled up in the other. OK that got fixed, but we uncovered some more items like that which delayed the process each time. We started to get a bit worried - in the end, all of the static data is done, we just have a few items left to do, but they are live data items so can be loaded over the next few days. W also found that some pricing information was way out - a change had been made to some data, and no-one knew about it. A minor issue, but it held things up becase we had to find out why.

There was also an issue with some of the product information - the consultants had been asked to make a minor change to allow for a particular piece of data we need to allow us to report on in a particular way - they had done some work, but it was wrong and we only uncovered this when the production manager ran a report. It hasn't been fixed yet, but they say that they'll have a solution in the next 2 days.

There was also a bit of friction between finance and sales - the sales people had cleaned up their data, but left a load of customer information in as they wanted to be able to track marketing effectiveness. The problem was that when finance started to look at it, they found some companies that no longer existed due to M & A activity, plus some that have unfortunately closed due to the economy. I can see the problem from both sides and normally I don't think it would be that big a deal, but everyone is now getting very tense, so smaller issues get blown up out of proportion.

The consultants got a guy to do a check of the hardware and configuration to make sure it's all OK. He sent thru some suggestions for parameter changes which I did on Friday. I checked and all was fine - but first thing Saturday, I got a call to tell me the system had fallen over. I quickly restarted and it seemed all right, but thought I should go in to be safe - a good job too as it fell over again. I looked at the log and realised that it was referring to a problem with one of the parameter changes, the value they had given me was totally out. Having put it back to the way it was, everything carried on ticking over.

The project team have been doing a lot of data check and testing of processes, far more than the consultants said we needed, but not as much as I think we need. I'm not convinced that the guys have been able to test all of the different scenarios or variants - but there's no time left to do any more.

My eldest was invited to a party for a friends birthday - her father was throwing a barbecue, so after I finished yesterday, I dropped by. It was really busy and it was nice just to sit with a beer with the sun going down and not have to think about too much. However, I got talking to to guy that used to work for a company that installed SAP a few years ago. We compared stories and although it was an earlier version, and different consultants, it was a strangely familiar tale. I am glad that it's not just us - it may sound crazy, but it helps when you look at something that's wrong and you can feel that other people have been here before.

Any way, there we are. In total, we have spent just over 3 years on this project, just under 30 months of that with SAP. I haven't seen the final costs and I know that the FD is holding back a couple of invoices and he says he won't pay them to make sure that we don't get left without access to proper consultant support (although I suspect we might be better off going elsewhere). However, the project cost seems to be a little under $1.5 million for the consultants and software - we think that with all the work our people have put in, we have spent another $750,000. Has it been worth it? Only time will tell.

Going to get a good nights sleep - early start tomorrow. I'm not sure how I really feel about it, it almost seems to be unreal. I really don't know what to expect - I'm hoping it will go well, but just a bit afraid that it won't. Still it's too late to worry about it know, just got to get on and deal with it the best we can. I think that the coffee machine is going to be working overtime this week.

I'll try to post more over the next few weeks so that you can see how things turn out.

Data loading

As you might guess, we are in the process of loading data into the production system, ready for the final tests before going live. The last couple of weeks have been frantic - we are just over half way there.

There have been a few issues - we had a big problem because the consultants hadn't transported a configuration change. The data wouldn't load and we had to wait for the consultant to process the change before we could carry on. There was another open issue for which they finally came up with an answer only for us to find that we had to modify the data load file in order to deal with this new item.

There has also been an issue with one of the tests - it's producing the wrong output figures. We are not sure why as in the tests in the test system, it worked fine. We are still checking this one out.

The guys at one of our other sites has also hit a snag. They wanted an output document for inventroy control - long term, they want a barcode on the paper so that we can use a scanner. Unfortunately, the document that they specified is producing completely wrong data; again we're not sure why.

In addition, there has been a problem emailing paperwork to customers / suppliers. It worked on the test system, but for some reason not yet discovered, it doesn't work on the production system.

Everyone is putting in a lot of overtime - we are all getting very tired and starting to make stupid mistakes. I think everyone needs to take a vacation after this.

The final stretch?

A couple of weeks ago, I had some contact with a guy that has been in consulting for a long time – he was kind enough to say some goods things about my writing, so I’ve decided to return the favor. Catch his blog here: http://sapsearchlight.blogspot.com/

One thing that I did find of interest; he referred to a process of project management that is supposed to be used by the consultants – ASAP (AcceleratedSAP I believe it stands for) also known more recently as Focus ASAP / ASAP Focus depending on where you get your info. He said “Outside of IBM and Accenture, all certified partners MUST adhere to SAP’s ASAP methodology, sometimes referred to (recently) as Focus ASAP. Most of these partners add some of their secret sauce to the core SAP methodology. I am willing to bet that if you look at this firm’s proposal of services to you that they make a big deal about their methodology.”

I did actually look thru all of the paperwork from these consultants – nowhere does it a make a mention of this. I asked around our project team to see if anyone had heard it cited, and the general answer was a definite “No” – apart from one person who remembered reading a reference to “Focus ASAP” in one of the SAPpress books that we bought a ways back. So I then thought I’d ask their Project Manager – unfortunately, he’s “not available” at the moment (I don’t know why) and we are not sure when we will next see him. I approached one of the other consulting staff and asked the question – the response was along the lines of “Oh that was used about 10 years ago, but no-one uses that anymore, it’s a really horrible system”. Interesting?

-----------------

As you may imagine, things are hotting up. For many people the project is starting to get real and you can sense the panic growing. I’ve had numerous project members and departmental heads speak to me and even a couple of the board expressing their concerns. We have no formal contingency plans other than those that I and my IT staff have discussed. We are committed to go-live and nothing will stop or delay that now.

A major problem was uncovered in the last week that has created a lot of bad feeling. As I’ve indicated, we trade across borders so use multiple currencies. We suddenly found that all of the customers had a default currency set and that this overrode any attempt to set the right currency. The initial suggestion was that the data was wrong – a quick look at the files showed that wasn’t the case. So we were blamed for loading it incorrectly, but that was quickly disproven.

Eventually, it turned out that the consultants had set the appropriate configuration in the test system, but not in the production system – thank goodness we found it before we started issuing invoices. They blamed us first for not raising it as an issue – however we did, over a year ago and could prove this by referring to the list of issues. They then said that the various departments hadn’t tested the process correctly – they had, but obviously not in the production system. No matter what we showed, they still tried to maintain it wasn’t their problem. And I’m betting that after we go live, all of the problems we then find, they will say the same thing, no matter what the problem.

In the past few weeks, I’ve been doing some travelling to the other sites (partly why I’ve not been adding to the blog) and I’ve heard of some other stories of SAP implementations going on at the moment. What is really surprising is that they are being marketed as successful, and yet from what the people on the ground are saying, they are anything but.

In one case, its being run by one of the Big 3, the others are smaller operations with smaller consultancies. Not one has come in on time, not one on budget, not even close. In each case, it seems highly improbable that the organization concerned will ever see the ROI that they wanted or were promised. At the biggest one, they went live last year and they were expecting some significant reductions in staff because of efficiencies, but instead they have had to take more staff on in order to manually process items because the system isn’t working right. And in that particular case, they’ve outsourced the data center which they expected would save them nearly $1 million a year – the local paper has found that the outsourced costs have increased their bill, not decreased it.

Well enough for now – I expect that over the next few weeks, I’m going to be very busy, so I may not have the time to keep you updated on progress. However, I may try to post a few short items if I can just so that you see how things pan out. What ever happens, many thanks to you all for reading this and for the great comments - they have been really helpful and I hope that I can return the favor one day.

I've got a bad feeling....

Things at work have started to get very tense - people are now starting to realise that the go-live date is getting closer. There are still so many items that have yet to be fixed up, and despite all the effort, it seems that many of these will not resolved in time. So the decison has been made that we will go ahead whatever the cost and live with any issues.

No doubt, many of you that read this will have heard similar stories to the ones that I have heard - about projects that suffer problems. I see it almost like a slow motion train wreck, and there is nothing that I can do.

The PM from the consultants was in the other day - he was supposed to be spending time with the finance people to go thru the closing of period / month / year. When he discovered that they didn't have the right transaction code, he didn't try to resolve the problem or ask me to deal with it - he just logged on as himself and added SAP_ALL to their user accounts. For those that don't know, SAP_ALL is a profile that when it is added to a user account gives that user full administrative access to the whole system.

I found out about this a few days later when I was making yet more changes - to my horror I found that some 25 accounts had been modified to give them the same access level. I've been told that this is needed because they no longer have time to waste and need to get these processes working and that we will fix it all up about a month after go-live. Call me cynical, but I suspect that won't happen and I'd give 50 bucks that they still have the same access level at the start of next year.

I also spotted something earlier - we have yet another new consultant on site to address a couple of specific issues. I wasn't told about this until after he had been there 3 days. I queried how he had been doing any work and was told that he was using the account of one of the other consultants - in addition he has created duplicates of the accounts our project team members use, and he has been using these to test things; but as they haven't been added to specific roles, he just added SAP_ALL.

In addition, it was highlighted that another consultant has been modifying some items - he has made a mess of it and now that item doesn't work at all. They've gotten someone else in to address the issue, but it appears that it will take 3-4 weeks to fix.

There was a review meeting earlier - as we went around the table, it was identified that one of the senior managers wasn't there as he is away on holiday. A comment was made that it was unacceptable that this key person should be away at such a crucial time. It was pointed out the person concerned is on his honeymoon - it was booked up 18 months ago before the date of the go-live had been put back so many times. In addition, the person concerned has had very little training - I think I am the only one that has spent time with him at all (altho I might be wrong). Certainly, he has actually had very little input into the project.

What happened then was astonishing - I tried to point out that this was a bit unfair and one of the directors started shouting and threw his notepad across the table at me (I suppose that I should be greatful it wasn't his laptop - no doubt I would have had to repair it). The rest of the meeting was extremely uncomfortable as you might guess.

I could post more, but just don't feel that it's worth while. I suppose that when it goes belly up, then they will be demanding that I wave my magic wand and fix it all in 10 minutes. I made a point of reading Rudyard Kiplings "If" - I think that it has a lot of resonance for those of us in IT, even tho it was written some 100 years ago.

http://www.kipling.org.uk/poems_if.htm

Busy, busy, busy....

I was going to write something last week but just couldn't make the time. It's been pretty hectic this week as well.

Part of the problem is that we have to do the SAP work at the same time as our normal tasks. We have to maintain hardware, patch software, keep the AV running, check backups, hold people's hands when their printers don't work etc. In addition, we have a 5 year development plan that gets updated every 6 months or so and we try to make sure that we stay on track. Over the last couple of weeks, there have been some changes to buildings and we were moving hardware around, doing some cable patching etc. This is straight forward if it is planned out - however, some of our managers think forward planning is deciding what we will have for brunch!

The SAP data loading has started - I've made the point to people several times that now this has started, we can't really delay the go-live date. Of course if necessary we could, but it would cause a large number of problems. However, we want to concentrate their minds and make sure that they understand the importance of the process. Despite this, some still haven't gotten with the program - we loaded some price information only to be told a day later that they wanted to make some changes!

Now I did the intial tests of the loading process way back and setup the scripts to import data. One of my staff was put in charge of the data assembly and he's done an excellent job. He's also tested the script in the test client to make sure that it should all run OK. So when we started the load process in the production system, we expected it to go well - and for the most part it did. Unfortunately tho' we hit a couple of snags.

For those that don't know SAP, you have 3 systems, Development, Test and Production. In those you have "clients" - effectively instances that can be run separately. One of these is designated the master client and all changes are made in there - then a system of "transports" move the configuration changes so that all systems are the same. It turns out that several of the consultants have been making changes to the configuration, but then haven't transported it to the production client correctly. Worse, they didn't even put the changes in the master client, so any transports will over write what they've done. We've had to go thru and sort these out as we find them, which is slowing down the load process - so lots of late nights and weekend working.

On top of that there are still a large number of outstanding issues, and one of these is to do with the finances. Apparently the consultant for that area told the project manager that he was intending to deal with it after go-live. I wish I'd been there to hear the rest of the conversation!

We still haven't gotten the labeling issue totally resolved and the PM wants one of the consultants back on site to deal with this. They've complained that they wanted to deal with this a few weeks ago, but couldn't as we were dealing with other matters. They can be a real PITA at times - we have to wait what seems an eternity for them to get back to us, but when they finally do, they expect us to leap into instant action at a single word.

I'm still waiting to find out what they intend to do with the CRM - it still runs very slowly and there are numerous parts to it that don't meet our needs. The sales director has indicated that if they don't get it fixed soon, he will make the decision to toss the project. As far as he is concerned, he would rather that we use the existing product as that works. We can't link it to SAP, but that doesn't bother him.

Several of the project team have started going back over the training with their staff - I've been suggesting that they should do this for months as most staff have forgotten everything they were told. There were a couple of sessions last week - of 19 staff in training, 3 could remember their password, the others all had to be reset! However, once they went thru the basics again, they stated to recall some of the steps. Unfortunately, I don't know if we are going to be able to get everyone back in the training room. We'll just have to wait and see.

Onward and upward!

And it's all gone quiet

I haven’t posted for a couple of weeks – well, that’s mainly because not a lot has been happening. So I thought that I would go back in time a bit, and cover some of the main issues from the early days in a bit more detail.

We had a launch meeting over 2 years ago now (unbelievable how the time has flown) and it had been agreed that at that meeting, the consultants would discuss the various steps of the project, and it was expected that we would come away with a clear idea of what had to be done, who was to do what jobs and have a timetable that we would all work to. Well, that WAS the plan.

I still have my notes from that meeting – one of my first points was that the agenda that they put up on the screen was not the one agreed with the project manager. In fact, it only had 4 bullet points on it – nothing else at all. When you consider that this was for a meeting that was expected to last all day, I felt that it was a bit light on detail.

I also made a couple of other comments after about 30 and 45 minutes – “when is he going to get down to it?” You see, the meeting was being managed by the director from the consultants, and he was doing most of the talking. That first hour, he did a LOT of talking – but at the end, I had no notes as he had not covered a single item that was actually relevant to the project plan. I could see that a couple of our senior managers were getting restless and the CEO left twice to take phone calls.

We broke for coffee at 11:10 – my real notes by that time were on 5 lines although I had some real nice drawings, none of which had anything to do with the project. During the break, our CEO and the consultants’ director got together and I’m told the CEO told him to move the pace up a bit. But it has to be said that by lunch, we still hadn’t really any better idea of what the project plan was. My favorite part was when the guy turned to the CEO and suggested that we ought to have a few bottles of champagne to get the project off to a good start – of course I had to speak up and suggest that perhaps champagne might be more appropriate once the software was actually running. Funny that he and I have never got on since.

After lunch, we were back to this guy talking on and on without actually making any real positive comments that would have been of any value. Later, just before the afternoon coffee break, he allowed their project manager to say a few words. This guy did actually discuss the timetable – but took no more than 20 minutes. Essentially, he stated that there were to be 6 key milestones (there were more, but he only referred to the 6) and he gave the dates for those, one of which was the meeting we were at (10 days late already) and the last one was for the go-live date.

By the end of the day, all we really had was a very basic timetable showing the start and finish with the “blueprint” phase, acceptance of this, the data cleansing, data loading and cutover to the live system. The idea of data cleansing had been briefly covered, but in very limited detail. I had suggested that we should start this as soon as possible as I knew we had a lot to do – this was brushed aside and in fact it didn’t start for a further 4 months. I should highlight that we are now some 18 months on and today it was found that one particular set of data still hasn’t been corrected and won’t load in its current state even though it has been sent back to the relevant department 5 times.

It was a further 2 months before most of the project team even really began to know what was expected of them; none of their responsibilities had been discussed and nothing was down in writing. I know that I can be a bit anal, but for me this so important. People have to know and understand what is expected of them – most won’t make notes, so I feel that writing it down so that they can refer to it is a must.

If I were to ever be involved in something like this again, I know that my priority would be to make sure that the launch meeting set the tempo for the project – that within 30 minutes, everyone would know exactly what was expected of them as individuals and of the team as a whole. We would have clear definition of responsibilities written down, with due dates and standards expected.

I think it a shame that the experience that I have gained from this project will never be used to make another project more effective – that is partly why I’ve been writing this blog, as I hope that other people might come across it and find it of value. The skills that I have gained have come at a high price in terms of workload and frustration and it should be possible to help other people avoid this. Certainly from what I have seen, the consultants have learnt little from it and are most likely to repeat the same errors, over and again.

Oh well, back to the grindstone.

The seconds tick away…

I’ve indicated that the company has a number of sites in this country as well as several more overseas. The original plan was that we would implement the SAP program and once it was proven to be working, it would then be rolled out to the other sites. There was a slight change to this in that we bought out a company in another country and for a while, it was required that they would be part of the project; that was cancelled, so we are back to just the original sites.

It should be highlighted that each site sells different products – yes there is a commonality, but the difference is sufficient that they use diverse processes. Now in the past, this has caused an issue – it’s been difficult to get any agreement on standardization - this is why we wanted a single ERP system.

When the project was set-up, the consultants suggested that we create a project team made up of managers to represent the key areas within the business. It was decided that our director of operations would head the team and act as project manager from our side; he would liaise with the project manager supplied from the consultants.

So far, so good - however, I queried one thing; most of the project team were from the one site, and there was only one representative for the other 3 sites. The explanation for this was that as we would all use the same processes, the one person plus a single consultant was required due to a specific area which they dealt with and that the other sites didn’t. I understand the idea of this, but I felt that this was not right.

Although we need to try to get some consistency, the nature of the differences in the way that each site works is fundamental and many of the business processes are then affected by this. I was concerned that we would see the system set-up to work one way only and that the other sites would then struggle to use a system that could not handle their particular needs.

After the blueprint phase, I was even more convinced that the one person would not be able to deal with all of the issues on his own. As we moved through the next steps, it became obvious that he was struggling to keep up. About 16 / 17 months ago, it was decided that he needed help and he has had 4 more people helping him, 2 on a temporary basis and 2 permanently. Despite this, I still felt uneasy – they were still concentrating only on certain key steps and ignoring other modules on the basis that the project team would cover those items.

Now over the past 8 / 9 months, the guys and gals from the project team have worked extremely hard to train people from the other sites. During this phase, there have been a number of questions raised by the people being trained about the way that they are to work in future. Some of these questions are not really relevant – we have been able to simplify some processes and things that they used to do are no longer necessary. But unfortunately that’s not so in each case and there are a number of key items that are absolutely required that still have yet to be set-up properly.

And now it gets worse. Our project manager went to one of the other sites 2 days ago to meet with the managers from those other sites. He showed them a PowerPoint Presentation and then ran through a demo of SAP using some data that they would recognize so that they could see that it worked. He told yesterday that the presentation went well and that these managers are now really happy with it and they all feel very positive. The presentation lasted a total of less than two hours.

I queried if these managers had actually used the software themselves – the answer was no. He sort of said that they wouldn’t be the ones to use it, so it wasn’t important that they knew the software, but that they should understand what it could do.

I have to disagree – the people concerned are the site director, who needs to be able to get sales data, the production manager, who needs to know how to schedule and manage the production, the sales support manager who will have a major task in ensuring that products are produced, packed and shipped on time, and one of the senior managers that is involved in ensuring key accounts are dealt with correctly. I know that of these guys, 2 have never logged on to SAP, the others have logged on, but neither has used it in the last 6 months.

Although he has said he is confident that they are happy with things now, that is not what I am hearing from other conversations. I really am concerned that these senior managers are now completely out of the loop – and worse, because they don’t understand, when the staff have problems, these managers will not know what to do or how to find a resolution.

At this stage, we are committed to a go-live date – the CEO has said that we will not delay any further under any circumstances. If it is not ready, it is not ready but we will just have to make do with it in the condition that it is in. (I actually believe that there is now some political pressure from somewhere being applied to force the project forward whatever the cost.) Whatever the truth, I do see that sometimes it is necessary to force the situation, but I am troubled that there are too many pitfalls and things may get ugly. Time to clean up that old fall-out shelter!

Thru a glass darkly

It occurred to me that when writing these posts, I've fallen in the same trap as a lot of people; I make assumptions that the readers will know certain things about SAP that I don't need to elaborate on. So I thought that I would clarify a few issues.

SAP is an acronym - Systems, Applications & Processing. The title was originally in German, but it works the same in English. It is both the name of the company (SAP GmbH or SAP Inc etc.) and it is also the name of the software product that the company produces. Normally, it is fairly clear which of the two is being referred to, but just occasionally, it can be a bit confusing.

The company is a global player - they operate in almost every market area of the world. Previously, due to the size, complexity and cost of implementing their product, they concentrated on selling to the larger organisations. However, some years ago, they realised that there were a lot more smaller businesses out there than larger ones and so changed their focus somewhat - they now try to sell to all sizes and type of organisations.

The software is a massive product, with many modules covering different areas within a business. It is intended to be a true Enterprise Resource Planning (ERP) product; that is, it allows data from all areas of a business to be handled and processed in order to allow a seamless transition of data and therefore much greater accuracy of processing and transparency of data for analysis. This has previously been of real importance to the larger operations, but is becoming equally important to the smaller ones that wish to compete on a equal footing.

So far, so good. However, now it starts to get a bit more complex. You see, SAP (Inc) don't actually do most of the selling of SAP (product) . Yes they have a massive marketing budget and teams of specialists that help in the selling process, but the majority of the selling is really done by other companies; and these are the ones that also (generally) do the implementation of the product. These can be smaller companies operating in a single country or larger one that operate across a global region - it could also be one the big boys such as the major consultancies. This actually also includes people such HP & IBM who are surprisingly active in this area.

Now there are people out there that will tell you that SAP (the product) can operate "out of the box". I know what they mean, but in fact that is a very simplistic statement and in reality is simply not true in the way that say Microsoft Office would work out of the box. From my experience, I can say with utter certainty that the software installation is not a just case of putting a CD / DVD in the drive (they actually supply a large box full of over 100 discs!). In order to carry out even the most basic of installations successfully, you need to know certain things about the product and your business before you start. Even then, there are a number of factors that could prevent an installation taking place at all.

In order to use the product effectively, it also needs to be set-up - SAP (the company) refer to this as "best practice". What they actually mean is that over the years, they have developed a series of modifications and changes to meet the needs of the larger businesses that they have installed the product in. They have a wide range of these based upon sector and process type; they can select which of these to install on a "mix and match" method. This requires a good knowledge of the product AND of the business - and I would suggest that this one area is responsible for many of the problems that occur during implementation.

Now it gets even murkier. In some of the consultancies, the people that work for them are employees, but many are only on short term contracts - say 18 months to 2 years. These consultancies will hire people for the specific project based upon need and skill. Unfortunately, many of the people hired are of varying quality - and in some cases, they end up working in a module area that they are not particularly skilled in. I've also seen that the consultancy can hire in a person who is in turn a private contractor, and these generally get paid by the day or week. I also understand that in some cases, the larger consultancy contracts out part of the work to a smaller firm. And of course, the smaller firm can then hire in the actual people, who might even be employed by yet another business.

You see, some of these consultancies specialize in specific areas - HR, Accounts, or groups of modules such as those that make up the manufacturing process. The concept is fairly sound - you hire in the expertise in the area that you have the need. This allows you to act as a consultant even if you don't have the required specialization.

But with all of these different people, using different employment methods and structures of reporting, you find communication problems even in the relatively simple projects. From experience, many of the individuals like to work in particular way and use specific methods or practices - in some cases, these clearly don't meet the specific requirements, and I've seen that they often go against the defined goals.

Now, for a lot of people, this will seem a strange way of working - but you have to understand that this is the SAP business model. They are NOT in the business of selling software - they actually want to sell the knowledge and experience of the consultants, either as business consultants, project management, training, education services - basically all of the additional items that are perhaps a bit harder to quantify.

Don't get me wrong, when the software is set-up properly, it seems to work well (if perhaps a bit long winded). However, I would suggest that the way that the product is sold and then managed is almost designed to lead to issues that will then require the hiring of additional outside expertise. The problem then of course is that often people get locked in to the product - no-one (especially in senior management) wants to admit that they might have made a mistake, so they act like the gambler who keeps on playing in the hope that the one big win will cover all his losses.

Our project is one of the less expensive ones - we've spent nearly $2 million so far after 2 years and based upon some of the comments and observations, I expect that we will see an annual expenditure of another $100k to $150k for the next 5 years in addition to any contracted support cost (which was hiked from the beginning of the year). I still have a copy of the original documents - they stated categorically that the total cost over the first 5 years would be $1.06 million and this covered all installation costs, support and consultancy.

In reality, we will probably have spent around 3 times as much by the time that we are finished. I did some basic numbers and to get the return that they suggested, we would have to continue to use the product for around 18-20 years, and that doesn't allow for any subsequent changes or projects. I have to be honest, I cannot see that we will ever actually get a return on our investment - it's just going be a big old money pit.

It's a lovely day...

....for sitting in the sun, maybe move into the shade a bit later, and share some bread & wine with the family and just generally enjoy a very pleasant weekend. Maybe a barbecue on Sunday?

However, part of the time I've spent doing some SAP related work. I discovered a minor issue on one of the servers - not enough to cause a big problem, but I want to try to eliminate any of these little errors before we go live - I think that we'll have enough to do once we achieve liftoff.

Now the actual SAP global support people are pretty good for the most part - the method of reporting errors is a bit of a pain at times, but once you get someone on the case, they've shown that they can get things done. So far, they've managed to fix almost all of the issues that I've raised - the biggest problem I see is trying to identify the correct function area for the problem to be listed under and they can be a bit sharp with you if you enter the wrong data.

On this occasion, it's actually been passed through 3 different support people - as each one has looked at it, they've suggested that the problem is caused by something different. The last one has asked me to carry out yet another set of upgrades. As has been indicated, my staff and I have had very little real training and the instructions that we received were so vague that they proved to be of little value. I have however been given the email address of a guy in Germany that is prepared to help - he writes / speaks pretty good English (a good job as Ich verstehe nur ein bission Deutche!)

The information that he provided for how to download and carry out the upgrades was not perfect - however, after reading and checking the information carefully, I have been able to use it to work out the correct procedure. I've also been able to create my own training document on how to do this specific task so any of my guys could do it if I'm not available. For me it's important that we don't rely on any single person's knowledge - it's too risky and I've seen companies that suffered because they were stuck when a key employee left them.

So on Friday, I managed to get the test system upgraded to the latest relase of the kernel. I actually am quite surprised that it went so well - prevously, the work was done by one of the basis people from the consultants and we had all sorts of problems both during and after the work was done. On this occasion, the update worked so well, that I just couldn't believe it. I literally spent about an hour trying to find a problem before I was prepared to think that maybe it had completed successfully. I then carried out the upgrade on the other systems after everyone had finished for the day and by late on Friday, it was all done. Subsequently, I carried out a system transport of the use role profiles. Again, it all went quite well, so we can start Monday with a freshly patched system.

Now this may not seem like a big deal - for many people, patching and updates are a pretty simple process and one that most users don't see too much of. Within the company, we run an automated patching solution for all of the PCs and servers that takes care of the OS, the productivity suite, and most of the other software products we use and that has saved time and money over the years. It also ensures that as far as possible, we are protected against malware - an ounce of prevention is better than a pound of cure.

But for me, the ability to be able to this work without having to rely on the outside consultants is a major step forward. I don't think that I can understate just how much better that makes me feel - stupid I know, but having been thru such a long process with so many problems and so many failed areas, this is a real achievement even if it is such a small success. Now this doesn't mean that finally all is well - I wish! But at this point in time, I'm prepared to take anything positive that I can find and use.

So now, the work is done - time to relax and feel good about having finally achieved something positive. Now if we could just get the rest of it working the same way.....

-------------------------

BTW, I see that I have a couple more followers - welcome to you all. I hope that you enjoy reading my comments, and find them of some value. I will keep on writing as things happen; I think that we are now pretty committed to going live later this year (but more on that next time) and I will continue to detail what happens after the big event. So keep watching...

Another 2 cents worth..

Uhhmmm...

We had another meeting yesterday - the director from the consultants arrived with a new guy in tow. Introduced as our new CRM consultant - however, we did some investigation and it turns out that he last did CRM some 5 years ago with version 4 of the product. Since then, he has acted as PM for most of the project he worked. So he seems like a really sensible choice (not).

During the meeting, the director from the consultants said that the new guy would be spending about 10-12 days on site to get an idea of what our requirements were. Excuse me.... what have they been doing for the last 2 years? Un - believable.

Now we were promised at the previous meeting I referred to, they would carry out a demonstration using a customer's system to show that the product works. (That was actually supposed to have been done about a week ago.) However, that still hasn't happened yet - and in addition, they said that they are now setting up a test system for us to see, and they want to make sure that it is OK before they do the demo so it will be another couple of weeks. Color me crazy, but I would have expected that to happen before we bought the damn product, not two years into the project and certainly not a couple of months before we are due to go live.

I have to say that I think the director from the consultants is so full of BS that he could fertilize the Sahara. If he told me that the sun rose in the East, I would want to get up at midnight to check it out for myself, before I would actually believe him. The funniest bit was his insistence that we are on target - we are now actually well over a year behind, but that doesn't seem to be an issue for him. He also tried to say that it's within budget - but not according to their documents, which I still have a copy of. We are in fact over spent by 3 times the original figure they quoted for a 5 year write off - and we had identified at the time that their figure were incorrect as they didn't include several key expenditures.

When I left the meeting, I just wanted to explode.....

The same consultants are also working on another project and occasionally, they can't get to us because they are on site with the other company. They started this project way before ours - not sure just how much longer it's been running. This other company went live back in the new year - I tried to contact them, but their IT manager wouldn't discuss the project with me (they had been told not to I think from the sound of it). Since then, one of our sales staff spoke to one of theirs - and they got the woman's number for me to call.

She's fairly new in the job as it appears they have lost quite a few sales staff because of the project - and she made it clear that she doesn't want to stay long either. She made some comments that no-one really seems to know how the system is supposed to work and they get all sorts of problems during the day which take ages to resolve (she referred to one problem that took 3 days to fix and they lost a huge sale because of this). Their sales people are really fed up with all the wasted time - and it also appears that they lost a couple of IT staff as well which has made the problem much worse.

I've also had a contact with another company (different consultants) - they have only implemented part of the SAP system so far - sales, purchasing and accounts. One of our people went to see them and talked about how they were getting on - they seemed to be doing OK, so it shows that it can happen. But they have also indicated that they are still getting issues - it appears that they had a problem with getting the invoicing correct. I'm not sure exactly what their problem was, but it sounds as if it was applying the wrong pricing. This is something that can happen to any company and I would suggest that most systems have done that at one stage or another, so it would be unfair to highlight this as if it was just SAP.

But even they said that the process had been painful and they were not entirely happy that they were getting all of the benefits that they expected. Questions have been raised about the cost of the project and it seems unlikely they will realise any actual savings for their investment.

Someone made a comment that bad news travels faster than good news - true and I accept that many big project fail somewhere along the line and I'm sure that there are some SAP projects that are models of success. But for me this has been by far the worst project I have ever been involved in by a long way - I've worked with some really astute business people in the past, and I know that they would have demanded heads to roll for a failure a bad as ours has been. I really can see the ax swinging at some stage - and I'm concerned that my guys and I will suffer as a result.

Oh well, tomorrow is another day...