We don't know, what we don't know

I'm going to go back to the beginning of our SAP implementation project. That was over 4 years ago now, and at times it seems like it was only a very short while ago - at other times, it seems like we have been doing this forever.

When the original plans were put forward, it was made very clear to everyone that this was not a normal IT project. Although there were some technical issues that only IT were getting involved in, the majority of the work was actually to be done by staff from across the business, and would involve people at all levels, in all departments. We knew that it would take time, and altho' the SI promised otherwise, I was fairly sure that it would take a lot longer than planned.

An ERP implementation (of any flavor) is always going to be a much bigger task than most others - it requires a lot more work because so much of the product integrates data and processes, and it is important to make sure that what happens in one area, doesn't impact on another. There are many things to consider, and even for those businesses that have a clear understanding of what they do and how they do it, the implementation project is going throw a few curve balls into the mix.

Added to that, most ERP products (including SAP) will have specific ways of doing things, and it is important to learn how to do these correctly. With most software programs, it's possible to buy a book, work with a demo verison and learn all you need - OK, somtimes, you also need to undertake specialist training, but with an ERP product, this is just not practical due to the size and complexity of the program.

For that reason, it's necessary to buy in the expertise required. Some companies will do this by hiring extra staff that have had previous experience, sometimes on a permanent basis, but more often as contractors. A more common approach is to use a System Intergrator (SI) which will be a organisation that has prior experience of working in the specific area with the particular product. The really big companies will use one of the big consultancy firms - for the SME type business like us, it's more usual to make use of the medium size or smaller SI.

The idea is pretty simple - the SI will supply people (consultants) at a set rate that have the required expertise. These people will then help us implement the software by providing technical expertise, passing their knowledge onto our staff, and guide us thru the various steps. This should reduce wasted time, enable us to get up to speed quickly, and ensure a succesful project. Well, a good idea in theory!

The problem is of course, that when learning new stuff, you don't know just what you need to know. Certainly if you have been involved in projects, you will have an idea of good project management practices - if you have worked with larger companies, you will understand the need for good communications and dissemination of information. You may even know how to manage project teams and stakeholder requirements. But in the case of SAP, if you have not worked with it previously, it is unlikely that you will know much about the practices required by that product, which is why people buy in those consultants.

That of course presumes that the consultants do know the stuff that we don't know and are willing and able to pass that knowledge on to our staff. Unfortunately, as we have seen that is not always the case. As you might imagine, after the issues that we have had to deal with over the last few weeks, there was a meeting to identify what happened and how to prevent it happening again. At the meeting, it was pretty much agreed that the problem was caused by the consultants not following proper procedure - the procedure which they are in fact supposed to be teaching us.

Of course, the SI dispute this - their people are doing their job and trying to help us by getting the work done as quickly as possible. Their arguement is that it is down to us not following correct procedures and incorrect testing of the work.

I will say that I would not totally disagree - I don't think we always do all of the testing that we should. We also should have the settings locked down much more than they are specifically to protect us from this sort of thing happening. But the problem is that if their people don't set and follow correct procedures, how can our staff know what is or is not the correct way to do something? What's worse is that if people learn bad practice, it is generally much harder then to change the way that they work.

Well enough for now. There are going to be some meetings in the next few weeks, and these will have an impact on what we do next. I am actually looking forward to it, altho' I suspect that I may not feel so pleased afterwards!

Spoke too soon...

I did say last time that we had fixed the issues caused by some bad config changes, and that we should be back to normal. Unfortunately, I jumped the gun a bit - we were still having some issues right up to the beginning of the week.

To be fair, these were mostly smaller issues - someone found that they couldn't post a purchase order because one of the items on the list had some mis-matched data. There was a similar issue when the guys in the warehouse were trying to assemble some stock to a delivery. Late on Friday, there were a couple of invoices that went wrong - and all of this was from data problems caused by the consultant loading config changes directly into the Production client.

We've locked the client against config changes now, but in reality, that won't stop it happening again. The consultants have the ability to unlock the client, and there's not a whole lot that I can do to prevent this. I have detailed what we need to do, which is to limit what access permissions they have, but whenever I make that point, the consultants start complaining about being "unable to do their work".

I will say that quite some time ago, I had the chance to speak to one of the consultants that we had on site for only a few days, and from what she said, it is clear that they generally don't get a lot of support or management when they are working on a customer site. She was quite new to the company, and to SAP, and it was clear that she felt a little out of her depth. She was being asked questions by people from our project team, and all she could do was refer it to one of her colleagues - she just didn't have the experience to handle many of the issues herself.

This is an item that has been raised before, and I know that a couple of the more experienced guys that are SAP Mentors that follow this blog, have also raised questions about the skill levels of many consultants. A given person may have a valid certification, but without at least some practical knowledge, that person might not be much more use than one of our own staff.

For me tho' the main issue seems to be that most of the people that have worked on our project don't seem to know much about basic good practice. I understand that time is money, and they want to get the job done as quickly as possible, but cutting corners is usually going to cause problems. Someone else made the comment that you can do it quick, do it cheap or do it right - but not all three, and we have seen that many times.

This is not made any better by not having any one checking to make sure that these people are following the right procedures. It seems that the SI wants to make that our job - OK, I can see why that is their preference, but at the beginning, we would have no idea what that good practice might be. We know more about it now, but in many ways, it's just too late. They have worked to fairly poor practices, and as a result, we also now have some fairly poor practice in place, and it's not easy to change that.

Well, we have a few new things being developed now - I won't be able to write about them for a little while, but I think that we have some interesting times ahead.

It's broken...

Last week, I made a comment that “the consultants were still working on config changes”, but didn’t go into any details. I can go back over that now…

So I made the point that the consultants were still making some config changes, and doing so directly in the Production system. Normally, all changes should be done in the Development system, checked, then transported to the Test system, before being checked again – and only when it is proven that the changes won’t cause any functional issues, do they then get transported to the Production system.

Now there are certain changes that have to be made directly in a system as they cannot be transported. However, as far as I am aware, SAP best practice still highlights that the correct procedure is to make the config change in the Development system and then once it has been tested making the same change directly in the Test system. It then gets tested again before the change is made directly into the Production system.

The reason for this is simple – if you have a live system, any change can have undesirable effects. If your business is reliant upon a system, you want that system to work all the time – you do not want it going wrong, or working incorrectly. If you follow the correct procedure, you should be able to ensure that any change made will not turn around and bite you in the ass.

But of course there are those people that choose not to follow procedures because either they don’t know any better, or because they think that they do know better. (Dare I suggest that some simply do not care?) In our case, I don’t believe that it is malicious, but I feel that the people concerned have never had to work to the appropriate disciplines.

On Monday morning, people found that they were not able to post anything in the production system. As I arrived at work, it became clear that we had a crisis on our hands. The CEO and FD were already there, discussing the problems and trying to get answers on what had happened and why things were not working.

After investigation, it became apparent that one of the consultants had really messed up – the config changes added to the system were physically preventing any new data from being added and quite a lot of the older data from being processed. I got everyone out of the system in case a restart of the server might fix the issue, but that was no good. After many hours of discussion with the people concerned, it became obvious that the only way to fix the issue was to reverse all of the changes.

That work has been going on all week. In that time, no-one has been able to do very much in SAP – there were a few jobs, some reports could be run, but not very much. Fortunately, the factory was able to continue working for several days as we had some info left over from the previous week, but that dried up after about 3 days. At one stage, the directors were even considering closing down production and sending people home and putting plans in place to do this.

However, by midday yesterday, all of the config changes that caused the problem have now been undone, and people were getting down to running the stuff that they had not been able to do. A couple of the staff from Sales will be working on Saturday and they hope to catch up with what they have missed – Finance have managed to organise 2 billing payment runs, and they are staying on for a couple of hours to get the invoices in the mail.

Meanwhile, the Production manager has started to run one of their scheduling jobs, and the Production Supervisor will try to run another later tonight. I doubt that we will be back to normal by Monday, but we should be more or less OK by Tuesday lunch time – evening at the latest.

So of course, everyone is pretty angry about this – but so far, no word of apology from the SI. The CEO has told them in no uncertain terms that any attempt to bill us for the work will be rejected. I think that we ought to get some sort of compensation, but based upon their previous mess-ups, I think that is highly unlikely to happen.

So we have managed to escape a major disaster  –  yes, things could have gotten a lot worse, and other people have suffered far worse things that we have. But I do feel that we should have not been exposed in that way to begin with, and I do get frustrated that we are suffering because of other people’s behaviour with apparently no recourse to compensation.

Not so happy new year

The new year has come and gone; so our overseas site must have gone live, right? Unfortunately, no.

Although we technically stopped work for Christmas, I was getting emails and phone calls right thru the holidays. I started to feel a little concerned as it looked increasingly like they were struggling with some basic stuff; and on New Years eve, I got a call to tell me that the go-live was postponed until the end of January.

Despite the fact that I had asked many times if the data had all been loaded (and been told that it was), the actual situation is that there are still a number of items of master data that have yet to be loaded. Little things like a load of customer and supplier information, bank details, a lot of product information and the open activity etc.

On top of that, the consultants were still working on config changes (more on that next time) up until Christmas eve - but then stopped work until the first week of January, and so there's a substantial number of items outstanding that haven't even been looked at.

I was asked to go over to the site along with the VP of operations - we had a really bad flight over, then problems at the airport with a lost bag. This delayed us getting to site by several hours, which was made worse by the cab driver taking the wrong route.

By the time we got to the site, he was not happy at all. The consultants were watching some video on a laptop and howling with laughter which really did not sit well with him. When he spoke to the consultants, he made the point that he had been told that everything would be ready, and the delay was down to them entirely. He also said that he did not see why we should continue to pay their company if they were not able to do the work that they themselves had agreed to.

Subsequently, he had a conversation with the director of the SI - who didn't apologise for any of the issues or the delay, but suggested that our VP should apologise for being too demanding! To be honest, I stayed out of it as much as I could - but the conversation we had at the hotel that night showed that our VP is not impressed, and I think that he will not leave this point alone until he gets satisfaction.

So, we spent a week on site, going thru the outstanding items, and trying to get these covered off. Surprisingly, we did manage to get a lot done while I was there, and altho there are many areas still not finished, we could be on track and be ready to go-live in about another 3 weeks.

I will say that I am still a bit concerned at the moment. The consultants have the staff at the site doing some training - by entering data into the live system. Now I accept that it is live data (from real orders etc.) but this should be done within the test system, not the production system. But hey, what do I know about anything.

Some of the results they get are not quite right - well that's to be expected if it's not been tested as it should. They have had to go thru some of the data to make amendments, and they had to make a couple of other config changes that had been put into the test system, but not the development or production system. Before you ask, they still haven't put them in the dev system.

So here we are again - waiting for work to be done that I was told categorically had all been finished. I am planning to go back over again in a couple of weeks, but possibly that may be sooner. I'll try to keep you updated.