Spoke too soon...

I did say last time that we had fixed the issues caused by some bad config changes, and that we should be back to normal. Unfortunately, I jumped the gun a bit - we were still having some issues right up to the beginning of the week.

To be fair, these were mostly smaller issues - someone found that they couldn't post a purchase order because one of the items on the list had some mis-matched data. There was a similar issue when the guys in the warehouse were trying to assemble some stock to a delivery. Late on Friday, there were a couple of invoices that went wrong - and all of this was from data problems caused by the consultant loading config changes directly into the Production client.

We've locked the client against config changes now, but in reality, that won't stop it happening again. The consultants have the ability to unlock the client, and there's not a whole lot that I can do to prevent this. I have detailed what we need to do, which is to limit what access permissions they have, but whenever I make that point, the consultants start complaining about being "unable to do their work".

I will say that quite some time ago, I had the chance to speak to one of the consultants that we had on site for only a few days, and from what she said, it is clear that they generally don't get a lot of support or management when they are working on a customer site. She was quite new to the company, and to SAP, and it was clear that she felt a little out of her depth. She was being asked questions by people from our project team, and all she could do was refer it to one of her colleagues - she just didn't have the experience to handle many of the issues herself.

This is an item that has been raised before, and I know that a couple of the more experienced guys that are SAP Mentors that follow this blog, have also raised questions about the skill levels of many consultants. A given person may have a valid certification, but without at least some practical knowledge, that person might not be much more use than one of our own staff.

For me tho' the main issue seems to be that most of the people that have worked on our project don't seem to know much about basic good practice. I understand that time is money, and they want to get the job done as quickly as possible, but cutting corners is usually going to cause problems. Someone else made the comment that you can do it quick, do it cheap or do it right - but not all three, and we have seen that many times.

This is not made any better by not having any one checking to make sure that these people are following the right procedures. It seems that the SI wants to make that our job - OK, I can see why that is their preference, but at the beginning, we would have no idea what that good practice might be. We know more about it now, but in many ways, it's just too late. They have worked to fairly poor practices, and as a result, we also now have some fairly poor practice in place, and it's not easy to change that.

Well, we have a few new things being developed now - I won't be able to write about them for a little while, but I think that we have some interesting times ahead.

It's broken...

Last week, I made a comment that “the consultants were still working on config changes”, but didn’t go into any details. I can go back over that now…

So I made the point that the consultants were still making some config changes, and doing so directly in the Production system. Normally, all changes should be done in the Development system, checked, then transported to the Test system, before being checked again – and only when it is proven that the changes won’t cause any functional issues, do they then get transported to the Production system.

Now there are certain changes that have to be made directly in a system as they cannot be transported. However, as far as I am aware, SAP best practice still highlights that the correct procedure is to make the config change in the Development system and then once it has been tested making the same change directly in the Test system. It then gets tested again before the change is made directly into the Production system.

The reason for this is simple – if you have a live system, any change can have undesirable effects. If your business is reliant upon a system, you want that system to work all the time – you do not want it going wrong, or working incorrectly. If you follow the correct procedure, you should be able to ensure that any change made will not turn around and bite you in the ass.

But of course there are those people that choose not to follow procedures because either they don’t know any better, or because they think that they do know better. (Dare I suggest that some simply do not care?) In our case, I don’t believe that it is malicious, but I feel that the people concerned have never had to work to the appropriate disciplines.

On Monday morning, people found that they were not able to post anything in the production system. As I arrived at work, it became clear that we had a crisis on our hands. The CEO and FD were already there, discussing the problems and trying to get answers on what had happened and why things were not working.

After investigation, it became apparent that one of the consultants had really messed up – the config changes added to the system were physically preventing any new data from being added and quite a lot of the older data from being processed. I got everyone out of the system in case a restart of the server might fix the issue, but that was no good. After many hours of discussion with the people concerned, it became obvious that the only way to fix the issue was to reverse all of the changes.

That work has been going on all week. In that time, no-one has been able to do very much in SAP – there were a few jobs, some reports could be run, but not very much. Fortunately, the factory was able to continue working for several days as we had some info left over from the previous week, but that dried up after about 3 days. At one stage, the directors were even considering closing down production and sending people home and putting plans in place to do this.

However, by midday yesterday, all of the config changes that caused the problem have now been undone, and people were getting down to running the stuff that they had not been able to do. A couple of the staff from Sales will be working on Saturday and they hope to catch up with what they have missed – Finance have managed to organise 2 billing payment runs, and they are staying on for a couple of hours to get the invoices in the mail.

Meanwhile, the Production manager has started to run one of their scheduling jobs, and the Production Supervisor will try to run another later tonight. I doubt that we will be back to normal by Monday, but we should be more or less OK by Tuesday lunch time – evening at the latest.

So of course, everyone is pretty angry about this – but so far, no word of apology from the SI. The CEO has told them in no uncertain terms that any attempt to bill us for the work will be rejected. I think that we ought to get some sort of compensation, but based upon their previous mess-ups, I think that is highly unlikely to happen.

So we have managed to escape a major disaster  –  yes, things could have gotten a lot worse, and other people have suffered far worse things that we have. But I do feel that we should have not been exposed in that way to begin with, and I do get frustrated that we are suffering because of other people’s behaviour with apparently no recourse to compensation.

Not so happy new year

The new year has come and gone; so our overseas site must have gone live, right? Unfortunately, no.

Although we technically stopped work for Christmas, I was getting emails and phone calls right thru the holidays. I started to feel a little concerned as it looked increasingly like they were struggling with some basic stuff; and on New Years eve, I got a call to tell me that the go-live was postponed until the end of January.

Despite the fact that I had asked many times if the data had all been loaded (and been told that it was), the actual situation is that there are still a number of items of master data that have yet to be loaded. Little things like a load of customer and supplier information, bank details, a lot of product information and the open activity etc.

On top of that, the consultants were still working on config changes (more on that next time) up until Christmas eve - but then stopped work until the first week of January, and so there's a substantial number of items outstanding that haven't even been looked at.

I was asked to go over to the site along with the VP of operations - we had a really bad flight over, then problems at the airport with a lost bag. This delayed us getting to site by several hours, which was made worse by the cab driver taking the wrong route.

By the time we got to the site, he was not happy at all. The consultants were watching some video on a laptop and howling with laughter which really did not sit well with him. When he spoke to the consultants, he made the point that he had been told that everything would be ready, and the delay was down to them entirely. He also said that he did not see why we should continue to pay their company if they were not able to do the work that they themselves had agreed to.

Subsequently, he had a conversation with the director of the SI - who didn't apologise for any of the issues or the delay, but suggested that our VP should apologise for being too demanding! To be honest, I stayed out of it as much as I could - but the conversation we had at the hotel that night showed that our VP is not impressed, and I think that he will not leave this point alone until he gets satisfaction.

So, we spent a week on site, going thru the outstanding items, and trying to get these covered off. Surprisingly, we did manage to get a lot done while I was there, and altho there are many areas still not finished, we could be on track and be ready to go-live in about another 3 weeks.

I will say that I am still a bit concerned at the moment. The consultants have the staff at the site doing some training - by entering data into the live system. Now I accept that it is live data (from real orders etc.) but this should be done within the test system, not the production system. But hey, what do I know about anything.

Some of the results they get are not quite right - well that's to be expected if it's not been tested as it should. They have had to go thru some of the data to make amendments, and they had to make a couple of other config changes that had been put into the test system, but not the development or production system. Before you ask, they still haven't put them in the dev system.

So here we are again - waiting for work to be done that I was told categorically had all been finished. I am planning to go back over again in a couple of weeks, but possibly that may be sooner. I'll try to keep you updated.

We have definitely been here before!

We are rapidly approaching the point for our overseas site to go live – they are due to start using the SAP production system starting on the first working day of the new year. As we only have a few days left before Christmas, there is not a lot of spare time.

The data loading process has been going on for some time now - I asked about how they were getting on a while ago. I was told that it was all done – but when I pressed a bit harder, it was admitted that there were a few items not quite finished. (Such as bank details, purchasing info records, part of the master material, some of the customer’s data and some other equally inconsequential items!) The work to load this is still going thru, but I would not be surprised to find it incomplete on day 1 of their go-live.

It also has to be said that there are still some items that have yet to be configured. Two weeks ago, it was agreed that we would stop all further changes in order to concentrate on the data load. It was suggested that I would do any further role change requests, as this could be arranged at a time when there would be no issues with processing the transports (as it happens, there has been no need for this). However, one of the consultants apparently didn’t get the message – a change was made, pushed thru without reference to anyone and a whole day was spent undoing some of the issues created.

In the mean-time, the SI has sent a letter of complaint that caused some internal discussion. There are a lot of bills that they have submitted that our FD simply will not pay. Of these, more than half are for work they say was done, but that we have no evidence for. For example, they had a person on site for several days over a 3 week period according to one bill, but the person concerned had left their employment and was working in a country on the other side of the world on the dates that they supplied. There have been items for 100s of miles of travel and a night’s accommodation for the same day.  Just to really stick the knife in, they also changed the agreed daily rate billed on some of the items – and when the FD asked who had authorised this, they were unable to give him an answer.

Over the past few months, I have also been trying to look at ways that we can make more use of our SAP system. It seems to me that whatever I might think of its capabilities and shortcomings, having spent as much money on the project as we have, we should try to make sure that we get the most bang for our bucks. There is a specific process that one of the managers has been wanting to do for several years, and altho I have a few concerns about it, it seems that this could be done within SAP. Part of the work is already done, which is a plus – it would just need slightly more data being entered than before.

His concern (and it is a valid one) relates to the output – he wants to do specific things with that, and it was subsequently suggested that it could also be used to improve some of the data provision to customers. These are all very consistent with company strategy and I want to encourage him. However, there are some concerns about how it would be done, and at present we don’t yet have the necessary specific skills internally – this would then require we buy in more consultancy.

Because of those concerns, he has recommended that we look to buy another software product – and to me, this is just utter madness. He wants the company to allocate another $300,000 to fund his project, but what really bothers me is that it would then duplicate the work being done, require more hardware, more software (plus licences, maintenance etc.) and he’s suggesting that it could be at least a year before the project would have any output worth using. Color me crazy, but that does not seem like the best way forward when we have already paid as much for SAP as we have, and could do the job as well by spending a little more on getting some training done.

So – the holidays are nearly upon us. Decorations are up, the tree is dressed, presents are wrapped, parties are being organised. We seem to have enough food in the house to last for the rest of the winter – I’m told that we are inviting lots of friends & family around this year and we must of course provide hospitality (I don’t remember having THAT discussion!).

As this will be my last post for 2011, I would like to wish you all a very merry Christmas, and I hope that whatever the holidays mean for you, that you enjoy them in the company of your friends, family or loved ones - and may the New Year bring you all the success that you would wish.

Back to the training board

I've not posted anything for some weeks - mainly because I've not really had anything to add. Work continues as normal, but there has not a great deal happening with our SAP project.

The go-live date for our overseas site was put back to January of next year. This should have given us time to get some of the data load done, but very little has been added so far. We still have time, but we are still waiting for much of the data, which is still to be cleaned up. However, the staff at the overseas site have actually been doing some training, and they have also been testing what data has been added, as well as manually adding some data of their own. I think that this will be very useful as it means that they will be more comfortable once they start using the system for real.

One thing that has been discussed in some depth is an old favorite of mine - training. I've said a number of times that I feel we should be enhancing our internal skills by getting our project team members and perhaps even some of our end users to attend formal SAP training. To me, it makes sense to invest the money in our staff, and retain knowledge within the business, rather than pay consultants who will be here today, gone tomorrow.

The costs are pretty straight forward - pay a consultant $1500 per day, or get staff on a 2 day training course for about the same money. I'm not sure how many times I've discussed this with people - probably to the point where I get boring. But I've been on some courses, as have some of my staff, and we know the value of the training, and can demonstrate the benefits.

To me, it is so simple - and yet our project team haven't seen it that way. I believe that part of the issue is that I am used to the idea of continuous training - anyone in IT will know the value of this and the need for constantly renewing skills. On top of that, I am also used to the requirement of managing my own training - but clearly, this is not how others see it.

However - it seems that my message is slowly starting to sink in. I've recently had a number of members of the team speak to me about organising training, and it seems likely that they will now carry this thru. The only problem is what courses will be of the most benefit and how to make sure that they get the most from the courses.

I am aware that if they attend a course and it doesn't meet their expectations, they will be less likely to want to go on another. I'm therefore trying to make sure that we have a very clear roadmap of the appropriate modules for each individual. I'm not so sure about the need for some of the certfication tracks - I don't consider it necessary that we need to get people qualified, but we should try to make sure that the plans follow a pattern that will provide appropriate knowledge in each area.

One thing that has been noted - the project team members are a bit more vocal in the way that they describe the lack of knowledge transfer from the consultants. They also seem a bit more prepared to criticize the quality of work carried out, and several of them have highlighted key areas that they want to cover in training that have been those where the consultants did not achieve what was required. I'm not sure that we can do this straight off, but I don't want to put them down too quickly - I'd rather try and keep their new enthusiasm high for as long as possible.

I suppose that once one of them goes on a course and then comes back and can confirm to the others the value of what they have learnt, we will then see a sudden rush of the others - well, that's OK as far as I am concerned. We have a decent sized budget now for the training as long as we can continue to justify it, so I want to see it used to best effect.

Hopefully, I'll have some feedback from at least one of them soon.

Have we been here before?

So - we were supposed to go live at our overseas site today. Up until Wednesday of last week, they were still determined that it would go ahead, come what may. This was despite no more than 10% of the data having been loaded, and we are still waiting for quite a bit of data to be extracted from their legacy system.

I only finally got the confirmation of the delay midday on Thursday - it's been put back to December at the moment, but I'm not sure if that will stand either. I know that the site manager is relieved as he was really concerned that they are just not ready, and I believe that most of the staff there are equally pleased. Our project team are not entirely happy - but they all accept that it would be too much of a risk to try and go ahead.

Now we have a bit more time to try to resolve some of the issues that are still outstanding - but I'm a bit concerned that we may end up with a few more as well. There have been a couple of config changes made and once again, it has caused us some problems. (Did someone mention testing?)

This time it's the purchasing, which is a real pain as that has actually been pretty good for some time now. It appears to be to do with the release strategy - the change means that staff can release purchase orders to any value, instead of the staged release which took us about 3 months to get right. I left the manager in charge of the purchasing department working his way thru a number of items to try and get a handle on exactly where it has gone wrong.

The consultants fixed the printing problem on the purchase order - well almost. There's a bit of an issue with the vendor addressing and a number of the POs have completely the wrong details. I got a message earlier today to tell me that there is a specialist going to take a look - it appears that he has already had access to the system, but he's been using someone else's account. I've now created an account for him, and asked him to use that - I've had no response, so I don't know if he hasn't received the message, or just can't be bothered to do so.

There was also an issue on the finance side (not quite sure what it was) and they had arranged for a consultant to look at the problem - we were told that it would take them about 4 / 5 days. However, the FD was looking at it, and with the help of a Google search, he has fixed the issue. The SI are not pleased at this.

Talking of the SI, I've had one of their people chasing me to outsource the systems to them. This is something that I'm not particularly keen on for a number of reasons. They insisted that it would all be managed in a facility in this country, but when I checked, the data center they use is actually somewhere else in the world - I think it might be either eastern Europe or Malaysia, I've not been able to clarify.

Their sales guy insisted that it would be cheaper to use them - well there would be savings in hardware / software / utilities, but I doubt very much that it would work out any cheaper. Some time ago, I looked into the costs, and based upon what we would need, the price from the only hosting center prepeared to give me an estimate would indicate that we would be looking at $300,0000 a year now, rising to $600,000 in about 3 years time. I can tell you, that would be more than our current IT budget.

However, it's not just the cost that worries me - this system is going to be the key system for the whole business. A decade ago, it wouldn't have mattered if it went off line for a few hours or even a day or two - that's not the case now. Quite simply, a loss of service of more than a few minutes would be problematic, more than a hour could be disastrous.

Based on their performance, can we trust our key system to them? Of course, it's not just the potential loss of service. I don't feel that they have demonstrated that they work in a particularly secure mananer, and I'll be honest, the thought that they might lose data scares the heck out of me. However, it won't be my decision anyway - this is something that the board will decide. Apparently, they are arranging for a junket at a country club for the shareholders and the board, to advise of them how it would work - I have not been invited.

However, I have been asked to put together a presentation of my concerns, highlighting the costs. I've also included some statistics on what we have achieved and how much we have saved by doing it all inhouse. In addition, I've put forward some plans for future development that would allow us to ramp up the resources to deal with required increases as the other sites come on line. We'll see what happens.

Back overseas again next week. I'm really racking up the frequent flyer miles - next time, I'm taking my wife with me!

I'm Back!

I’m back! Before you ask, I had a great time thank you.

Unfortunately, the flight back was delayed – and we got home very late on the Sunday. There were a number of messages waiting on the ansafone (I hadn’t taken my cell phone), and essentially, the messages all said that I was to get over to our overseas site asap. So I packed a fresh suitcase, and got my flights booked – with an early flight, the time differences plus the delays the day before, I was really very tired by the evening.

I spent the week over there, and then it was decided that I should go back again this last week. What with that plus trying to do my normal work and various other chores, this is the first chance I’ve had to sit down and analyze the situation.
I’ve not been told officially, but it appears that a number of staff have been told that the overseas site will not be going live in October after all. I’m not surprised – the data load was supposed to have started the 1st week of August and it hadn’t started by September. Even now, very little data has actually been loaded, and it appears that there are still some unresolved issues with parts of it.

During my first week over there, I spent a bit of time doing some training – about half of the staff there had still not even logged onto the test client, and they are a long way from being confident in using SAP. I also had a very upset General Manager complaining that there was a printing problem. When I checked, it was actually only one document that had the problem – the sales order form, so rather important that they gets this fixed.

I haven’t done that much work on the smart forms, but I took a quick look, and discovered that one their consultants had made the change. Unfortunately, no-one knows why or what for, and the guy isn’t available. I’ve left it to them to chase up, but so far, it’s still not printing at all.

Another issue has also appeared recently that affects the financials. At first, the consultants suggested that it was an authorization problem, but we’ve now identified that certain products are being assigned to the wrong ledger code. Again, no-one is really quite sure why – we have our consultants telling us that the configuration was correct, and their consultants are saying that it’s not. So they’ve made a change and now we have a problem.

Our Accounts Manager is really upset about all of this. A couple of times now, she has spoken to me in private, and it’s clear that she is very distressed about the whole scenario. She’s been with the company a long time, and has been thru some really tough times, but I think that this is the worst she has ever felt about her job. I just don’t know what to say to her anymore.

I’m also a bit concerned about the amount of testing that they have done. If you’ll remember, a bunch of us went over back in April, and we did some full end to end testing of processes and made sure that everything worked – but that was only using a limited amount of test data.

The guys on site have access to the test system and although not all of the live data has been loaded, about 80 – 90% is available. But the testing that they have been doing is not the same end to end testing. Yes they have put a whole ton of quotes, purchase orders, sales orders on the system, but they have not then followed this thru to make sure that orders appear on the MRP or that sales get billed.

It could be argued that we shouldn’t need to test again, as it was proven to work in the development system and we are using the processes in the production system. But the issue is that their consultants have made a few changes and I strongly feel that we can’t be absolutely sure that any of their tests would actually work unless we do try it out.

I did manage to speak to the GM of our overseas site one evening over a beer – he is very much of the opinion that they will not be ready for the start of October. He did suggest that perhaps they could have a staged approach to the go-live, perhaps by working in tandem on both systems. I wouldn’t have a problem with that in principle, but they don’t have that many staff, and I’m not sure they could actually do that.

Oh well, back in to work tomorrow, and maybe I’ll find out a little more about what the plans are.