Still plowing on...

I've been tied up with work again over the last few weeks. Sometimes, it seems that the more we do, the less we have achieved.

We are still having problems with our first overseas site. It appears that there are still items of data that have not yet been loaded. Some of the information for foreign language descriptions of products already in the system, some of the purchasing information records, contact data and even some banking data amongst others. Note that I did ask the question more than once and was told each time that all of the data load had taken place.

I should also highlight that none of this data load has been tested in the Development or Test systems - they gave up doing that a couple of months before New Year, and the data load is being done directly into the Production system. Of course, it will do a couple of items, then fail because something is wrong with the actual data (which I accept is down to us) - but then they just correct it and carry on as if this is the most normal thing in the world. When I suggested that it might be better to at least try in the test system, I was told that was a waste of time.

As for data in the Test and Dev systems - It's been agreed that we are going to sort that out by doing a client copy. The last time that was done, it took more than a week to complete and we had all sorts of issues for a couple of weeks after. I hope that we don't get the same this time as I am trying to do a number of other projects and my staff and I will not have time to fix any issues.

Anyway, a couple of weeks ago, we had the guys from one of the other overseas sites visit with us, so that they could see how the SAP system works. I was told that we would see their whole project team, but only 3 people had travelled over. I've since had a chance to speak to the VP for the site, and made some points to him about the need for a full project team - I also put together some descriptions of work required which have been sent to him.

I've been asked to go over to them next week to talk to their people in a bit more depth. It turns out that they have been promised that our project team will all be available to help out, which is going to be a shock to most of those guys, as no-one has discussed this with them. I've received an invite to a meeting in April and I think they will all be told then - I've been asked not to discuss it with them yet.

There is also another issue that is causing a lot of anger. When we first started, the consultants set up a process within the Sales area that did not work the way we wanted - they told us that was how it had to be and was considered "best practice". How we wanted it to work was just not possible. Once it was working, they found some problems and we had to get a couple of people in to fix those.

When the new set of consultants were working on our overseas site, they set things up so that the process worked the way we wanted - and they said that was actually the way they normally do it. They couldn't change the process for us at that time, but have now agreed a date to do some work to re-do some things so that our Sales people can use the same process in the same way.

To re-cap - we paid for about 4 weeks to set it up, then another 2-3 weeks to "fix" it, and now we will have to pay for another 4 weeks to change it to work the way we wanted in the first place. I wanted to challenge them over this, but I've been told that we will just have to suck it up and pay the money.

The best thing - I've recently had an email from the SI asking if I would be prepared to recommend them! They want to use our site to show a "highly successful" implementation, and possibly even use us as a case study. Astonishing!

The Security Stuff (part 2)

I'm following on from last week's post - because I think that there is a bit more to say about this topic.

If you follow any of the people that promote the concept of "Empowerment" within a business, you'll no doubt be aware that many of these people or companies are talking about allowing staff access to more of the information held within their systems, so that it is easier for decisions to be made at lower levels, rather than having all decisions coming down from the top. This is also sometimes described as "agile" business decision making.

There is something to be said for this idea - if someone has to purchase some goods, and has been doing the job for some years, she may well be quite capable of negotiating the right kind of deal without having to refer it to her line manager. This then means that the manager is left to deal with more important issues - and when I was a younger manager, it was known as delegation. A senior manager once told me that if you cannot delegate, you cannot manage.

The problem is of course that not everyone is entirely capable of making decisions at certain levels, and need to be monitored more closely than some others. It also has to be said that there are more than a few managers and senior executives that would be horrified at the thought of more junior people making even quite minor decisons without referring to them. Even if all they are doing is glancing at an email and then replying "do what you think best", they feel that they are "managing the situation". (A bit like the PHB in the "Dilbert" cartoons!)

So it's necessary to decide how you want to secure your date as well as how you need to secure your data (the two are not the same). How you need to secure the data is often defined by regulatory compliance or the need to prevent commercially sensitive information from going astray. How you want to secure the data is more about how you structure the business and how the various processes work.

Most people would agree that when SAP was first created, the main target was the larger enterprise business. Within this size of business it is generally necessary to provide some separation of duties - you don't want the person that creates a new bank account being able to transfer money, otherwise  he might create an account for himself in the Caribbean and transfer $10 million, before taking a one way flight to his retirement (which he also puts on expenses!).

As a result, much of the way that things are structured in SAP security has been done specifically to provide the required levels of security for those large organisations. The problem is that when you get to the smaller businesses like ours, it's actually much harder to define what you need in terms of security. Worse, this can actually change a lot more frequently as staff will often be doing more types of jobs than their counterparts in the enterprise business.

Now we have a problem - I've been trying to make sure that all of the permissions are properly set-up and correctly tested, but even so, sometimes it's not always easy to do this. I will hold my hands up and admit that there have been many occasions when I was going thru a role and suddenly spotted something that didn't look right, that then needed to be corrected.

Partly this is due to some people taking the view that "give them access to everything, and we can take away what they shouldn't have". This doesn't work - why? Because the only time that you find someone can do something that they shouldn't is when they have done just that, and it has caused a problem. And usually, that is when things are going seriously wrong.

We have seen that, not just the one time, but on a number of occasions. Most recently because we have some transactions that were created for us by one of the consultants based upon standard SAP t-codes. Unfortunately, the new t-codes have absolutely no authorisation checks built into them - and as a result, we have found that some people are able to do some work that was supposed to only be done by the Production Manager once a day to clear up any issues.

Oh, and in case you wondered - when I said that I would take the offending t-code out of the role, the Production Manager asked me to leave it in. He is going to speak to staff to try to make sure that they understand that they shouldn't run this particular process, just use the t-code to look at the data. Yes - I'm sure that will work (until the next time).

We'll do the security stuff later.

A couple of days ago, there was a question posted on SDN about security, that got me thinking about how this had been handled during our implementation. If you haven't seen the question, here's a link and I would suggest that it's worth a quick look.

http://forums.sdn.sap.com/thread.jspa?threadID=2133938&tstart=0

Now I am not a security specialist, altho' I have taken a couple of SANS approved security certs. When I was told that we were going to be implementing SAP, I got talking with one of the guys from the training course that I had done, and he was a bit dismissive of SAP security. He made a comment that SAP security was based upon "security by obscurity", and said that he had never met an SAP consultant that even understood the basics of security.

When I had the chance, I asked the SI senior managers if they worked to ISO27001 - they were happy to say that they did and had been fully accredited. Subsequently, this proved to be less than true - they did not actually know what was involved in the accreditation process, and as far as I can tell have never undertaken any of the work that would be required. (I should also say that they also claimed to have a couple of other accreditations that they are not entitled to.)

When we started work on the project, their Project Leader got me started on creating user accounts, and some basic roles - but only really very simplistic stuff. As we started testing, I found that some of the things that I had been told to do were completely incorrect, and I had to make use of SDN and some other resources to identify just where I was going wrong. Later I also did the ADM940 course which filled in a number of gaps, and I've since also done ADM950 & ADM960.

Now I'm not going to say that this makes me a security expert, far from it. There are a lot of people out there that have a great deal more experience with almost no paper qualifications - and I would be happy to learn from them. But the work that I have done within the training that I have undertaken does give me a level of confidence to be able to say that our consultants really did not know some of the most basic requirements of good security practice. And from what I have sinced learned, they don't seem to be in the minority.

The quote at the head of this post is an actual comment from one of their guys - and for me, this shows a real lack of appreciation of what security is really about. For security to be of any value at all, it has to be built in from the start - trying to bolt it on after the work is done is quite simply never going to work. A bit like having a high powered sports car, and trying to add wide wheel tires to give it more grip - it will just never do what it should do.

We are fortunate - we don't have to conform to the level of compliance that a lot of other businesses have to. We get audited once a year and we have a couple of specific checks done, but that is all. I would say that's a good thing for us - if we had to go thru some of the bad practices to correct them, I'm fairly sure that we would still not have gone live.

Security is not just about PFCG and SU01 - it needs to involve so much more within the software, but it also should cover physical access to servers, workstations and other devices, good working practices, and a structured approach to everything that has to be done within the system administration and management. We've seen first hand what can go wrong when this is not done in the approved manner, and the big problem is that when you're trying to fix a problem after the event, you often only treat the symptom, not the cause.

I will say that in most cases, we never really knew who some of the consultants were that we would get from one month to the next. We never saw any information about them or their skills - I'm not even sure if the SI even bothered to check on those either. Heck, I'm not even sure if they checked to see if the person that came to site was the person that they employed!

I can understand that for some people, the drive to get the systems in and working is the top priority, and everything else is secondary. But security should be something that is fundamental to any good system, and it should be in place from the very beginning of the project. Until it is given that level of attention, it will never work the way that it should. And that is only ever going to reflect badly on SAP, no matter who is really at fault.

We don't know, what we don't know

I'm going to go back to the beginning of our SAP implementation project. That was over 4 years ago now, and at times it seems like it was only a very short while ago - at other times, it seems like we have been doing this forever.

When the original plans were put forward, it was made very clear to everyone that this was not a normal IT project. Although there were some technical issues that only IT were getting involved in, the majority of the work was actually to be done by staff from across the business, and would involve people at all levels, in all departments. We knew that it would take time, and altho' the SI promised otherwise, I was fairly sure that it would take a lot longer than planned.

An ERP implementation (of any flavor) is always going to be a much bigger task than most others - it requires a lot more work because so much of the product integrates data and processes, and it is important to make sure that what happens in one area, doesn't impact on another. There are many things to consider, and even for those businesses that have a clear understanding of what they do and how they do it, the implementation project is going throw a few curve balls into the mix.

Added to that, most ERP products (including SAP) will have specific ways of doing things, and it is important to learn how to do these correctly. With most software programs, it's possible to buy a book, work with a demo verison and learn all you need - OK, somtimes, you also need to undertake specialist training, but with an ERP product, this is just not practical due to the size and complexity of the program.

For that reason, it's necessary to buy in the expertise required. Some companies will do this by hiring extra staff that have had previous experience, sometimes on a permanent basis, but more often as contractors. A more common approach is to use a System Intergrator (SI) which will be a organisation that has prior experience of working in the specific area with the particular product. The really big companies will use one of the big consultancy firms - for the SME type business like us, it's more usual to make use of the medium size or smaller SI.

The idea is pretty simple - the SI will supply people (consultants) at a set rate that have the required expertise. These people will then help us implement the software by providing technical expertise, passing their knowledge onto our staff, and guide us thru the various steps. This should reduce wasted time, enable us to get up to speed quickly, and ensure a succesful project. Well, a good idea in theory!

The problem is of course, that when learning new stuff, you don't know just what you need to know. Certainly if you have been involved in projects, you will have an idea of good project management practices - if you have worked with larger companies, you will understand the need for good communications and dissemination of information. You may even know how to manage project teams and stakeholder requirements. But in the case of SAP, if you have not worked with it previously, it is unlikely that you will know much about the practices required by that product, which is why people buy in those consultants.

That of course presumes that the consultants do know the stuff that we don't know and are willing and able to pass that knowledge on to our staff. Unfortunately, as we have seen that is not always the case. As you might imagine, after the issues that we have had to deal with over the last few weeks, there was a meeting to identify what happened and how to prevent it happening again. At the meeting, it was pretty much agreed that the problem was caused by the consultants not following proper procedure - the procedure which they are in fact supposed to be teaching us.

Of course, the SI dispute this - their people are doing their job and trying to help us by getting the work done as quickly as possible. Their arguement is that it is down to us not following correct procedures and incorrect testing of the work.

I will say that I would not totally disagree - I don't think we always do all of the testing that we should. We also should have the settings locked down much more than they are specifically to protect us from this sort of thing happening. But the problem is that if their people don't set and follow correct procedures, how can our staff know what is or is not the correct way to do something? What's worse is that if people learn bad practice, it is generally much harder then to change the way that they work.

Well enough for now. There are going to be some meetings in the next few weeks, and these will have an impact on what we do next. I am actually looking forward to it, altho' I suspect that I may not feel so pleased afterwards!

Spoke too soon...

I did say last time that we had fixed the issues caused by some bad config changes, and that we should be back to normal. Unfortunately, I jumped the gun a bit - we were still having some issues right up to the beginning of the week.

To be fair, these were mostly smaller issues - someone found that they couldn't post a purchase order because one of the items on the list had some mis-matched data. There was a similar issue when the guys in the warehouse were trying to assemble some stock to a delivery. Late on Friday, there were a couple of invoices that went wrong - and all of this was from data problems caused by the consultant loading config changes directly into the Production client.

We've locked the client against config changes now, but in reality, that won't stop it happening again. The consultants have the ability to unlock the client, and there's not a whole lot that I can do to prevent this. I have detailed what we need to do, which is to limit what access permissions they have, but whenever I make that point, the consultants start complaining about being "unable to do their work".

I will say that quite some time ago, I had the chance to speak to one of the consultants that we had on site for only a few days, and from what she said, it is clear that they generally don't get a lot of support or management when they are working on a customer site. She was quite new to the company, and to SAP, and it was clear that she felt a little out of her depth. She was being asked questions by people from our project team, and all she could do was refer it to one of her colleagues - she just didn't have the experience to handle many of the issues herself.

This is an item that has been raised before, and I know that a couple of the more experienced guys that are SAP Mentors that follow this blog, have also raised questions about the skill levels of many consultants. A given person may have a valid certification, but without at least some practical knowledge, that person might not be much more use than one of our own staff.

For me tho' the main issue seems to be that most of the people that have worked on our project don't seem to know much about basic good practice. I understand that time is money, and they want to get the job done as quickly as possible, but cutting corners is usually going to cause problems. Someone else made the comment that you can do it quick, do it cheap or do it right - but not all three, and we have seen that many times.

This is not made any better by not having any one checking to make sure that these people are following the right procedures. It seems that the SI wants to make that our job - OK, I can see why that is their preference, but at the beginning, we would have no idea what that good practice might be. We know more about it now, but in many ways, it's just too late. They have worked to fairly poor practices, and as a result, we also now have some fairly poor practice in place, and it's not easy to change that.

Well, we have a few new things being developed now - I won't be able to write about them for a little while, but I think that we have some interesting times ahead.

It's broken...

Last week, I made a comment that “the consultants were still working on config changes”, but didn’t go into any details. I can go back over that now…

So I made the point that the consultants were still making some config changes, and doing so directly in the Production system. Normally, all changes should be done in the Development system, checked, then transported to the Test system, before being checked again – and only when it is proven that the changes won’t cause any functional issues, do they then get transported to the Production system.

Now there are certain changes that have to be made directly in a system as they cannot be transported. However, as far as I am aware, SAP best practice still highlights that the correct procedure is to make the config change in the Development system and then once it has been tested making the same change directly in the Test system. It then gets tested again before the change is made directly into the Production system.

The reason for this is simple – if you have a live system, any change can have undesirable effects. If your business is reliant upon a system, you want that system to work all the time – you do not want it going wrong, or working incorrectly. If you follow the correct procedure, you should be able to ensure that any change made will not turn around and bite you in the ass.

But of course there are those people that choose not to follow procedures because either they don’t know any better, or because they think that they do know better. (Dare I suggest that some simply do not care?) In our case, I don’t believe that it is malicious, but I feel that the people concerned have never had to work to the appropriate disciplines.

On Monday morning, people found that they were not able to post anything in the production system. As I arrived at work, it became clear that we had a crisis on our hands. The CEO and FD were already there, discussing the problems and trying to get answers on what had happened and why things were not working.

After investigation, it became apparent that one of the consultants had really messed up – the config changes added to the system were physically preventing any new data from being added and quite a lot of the older data from being processed. I got everyone out of the system in case a restart of the server might fix the issue, but that was no good. After many hours of discussion with the people concerned, it became obvious that the only way to fix the issue was to reverse all of the changes.

That work has been going on all week. In that time, no-one has been able to do very much in SAP – there were a few jobs, some reports could be run, but not very much. Fortunately, the factory was able to continue working for several days as we had some info left over from the previous week, but that dried up after about 3 days. At one stage, the directors were even considering closing down production and sending people home and putting plans in place to do this.

However, by midday yesterday, all of the config changes that caused the problem have now been undone, and people were getting down to running the stuff that they had not been able to do. A couple of the staff from Sales will be working on Saturday and they hope to catch up with what they have missed – Finance have managed to organise 2 billing payment runs, and they are staying on for a couple of hours to get the invoices in the mail.

Meanwhile, the Production manager has started to run one of their scheduling jobs, and the Production Supervisor will try to run another later tonight. I doubt that we will be back to normal by Monday, but we should be more or less OK by Tuesday lunch time – evening at the latest.

So of course, everyone is pretty angry about this – but so far, no word of apology from the SI. The CEO has told them in no uncertain terms that any attempt to bill us for the work will be rejected. I think that we ought to get some sort of compensation, but based upon their previous mess-ups, I think that is highly unlikely to happen.

So we have managed to escape a major disaster  –  yes, things could have gotten a lot worse, and other people have suffered far worse things that we have. But I do feel that we should have not been exposed in that way to begin with, and I do get frustrated that we are suffering because of other people’s behaviour with apparently no recourse to compensation.

Not so happy new year

The new year has come and gone; so our overseas site must have gone live, right? Unfortunately, no.

Although we technically stopped work for Christmas, I was getting emails and phone calls right thru the holidays. I started to feel a little concerned as it looked increasingly like they were struggling with some basic stuff; and on New Years eve, I got a call to tell me that the go-live was postponed until the end of January.

Despite the fact that I had asked many times if the data had all been loaded (and been told that it was), the actual situation is that there are still a number of items of master data that have yet to be loaded. Little things like a load of customer and supplier information, bank details, a lot of product information and the open activity etc.

On top of that, the consultants were still working on config changes (more on that next time) up until Christmas eve - but then stopped work until the first week of January, and so there's a substantial number of items outstanding that haven't even been looked at.

I was asked to go over to the site along with the VP of operations - we had a really bad flight over, then problems at the airport with a lost bag. This delayed us getting to site by several hours, which was made worse by the cab driver taking the wrong route.

By the time we got to the site, he was not happy at all. The consultants were watching some video on a laptop and howling with laughter which really did not sit well with him. When he spoke to the consultants, he made the point that he had been told that everything would be ready, and the delay was down to them entirely. He also said that he did not see why we should continue to pay their company if they were not able to do the work that they themselves had agreed to.

Subsequently, he had a conversation with the director of the SI - who didn't apologise for any of the issues or the delay, but suggested that our VP should apologise for being too demanding! To be honest, I stayed out of it as much as I could - but the conversation we had at the hotel that night showed that our VP is not impressed, and I think that he will not leave this point alone until he gets satisfaction.

So, we spent a week on site, going thru the outstanding items, and trying to get these covered off. Surprisingly, we did manage to get a lot done while I was there, and altho there are many areas still not finished, we could be on track and be ready to go-live in about another 3 weeks.

I will say that I am still a bit concerned at the moment. The consultants have the staff at the site doing some training - by entering data into the live system. Now I accept that it is live data (from real orders etc.) but this should be done within the test system, not the production system. But hey, what do I know about anything.

Some of the results they get are not quite right - well that's to be expected if it's not been tested as it should. They have had to go thru some of the data to make amendments, and they had to make a couple of other config changes that had been put into the test system, but not the development or production system. Before you ask, they still haven't put them in the dev system.

So here we are again - waiting for work to be done that I was told categorically had all been finished. I am planning to go back over again in a couple of weeks, but possibly that may be sooner. I'll try to keep you updated.