I'm still here

As you may imagine, I've been pretty busy over the past few months. There's been a lot to do and not much time to do it all in. I seem to be living out of a suitcase for a substantial amount of the time and the weeks have flown by as I try to get a grip on the things I need to learn and start getting plans for the future in place.

A number of people have suggested that I ought to continue the blog, to highlight the key things that I've learned and offer suggestions for those undertaking a new project. I like the idea of this; as I've said, I would really like this blog to have a more positive purpose and the concept of summaries would fit well with that.

I've started to put together a basic outline; it's still a work in progress, but I hope to be able to post some items on a regular basis. I'm going to say that I imagine that some people might not agree with some of the points I make; I think that's fine and I hope that you will feel free to add your comments where you think that I may not be on the right track.

Keep an eye open; I hope to get a few new postings out in the next few weeks, subject to my work permitting!

That's all, she wrote...

I did say that I would keep you all updated - so here is what's been happening.

Before my last post, I'd had a number of interviews and was waiting for some response. About a week after the post, I got a call from one of those companies, asking me to attend a second interview. This was to take the form of a presentation to their entire board of directors - so not too much pressure!

On the actual day of the presentation, I found out that in fact there were only two candidates - myself in the morning, and another guy in the afternoon. I'd put together a short presentation using PowerPoint, illustrating some of the key achievements and how I had managed the various projects at my current role. It could have been a poor start as their presentation equipment wasn't functioning - but I quickly worked out what was wrong and after that, it went very smoothly.

The actual presentation was about 20 minutes and was followed by a Q & A session that I'd been told would probably be just over an hour - in fact it went well over 2 hours. A lot of it was about the specific areas that each director had concerns over, but the biggest single session was about SAP. They have plans to implement this, and have had several meetings with an SI who gave them the SAP line, but it seems that they've all heard of the various failed projects, and wanted to hear from someone that had actually worked on a project to get an independent view.

I really got a buzz out of the meeting - I seemed to get on well with the people concerned and I got the feeling that we connected really well. Certainly, I felt strongly that I could easily have worked with them and would have been more than happy to do so. When I left, I was a bit drained, but really happy that I had put myself across well, and felt that I'd done a great job.

Anyway, a couple of days later I got a call - and the response was that they were going to offer the job to the other guy. However, it was made clear that they had found it a really tough choice - it appears that both of us had done well, and in the end, the other guy got it because he had been on two SAP projects to my one.

What was interesting was a comment made by the CEO. He highlighed that we had both been asked similar questions about SAP, and our answers were almost identical. We'd both highlighted the kinds of problems that could be faced and how they should go about addressing these. Specifically about controlling the SI and managing expectation of users, as well as things such as testing of data and following correct procedures.

So, I felt pretty good about the result altho' I didn't quite get the job. But then 2 hours after the call, the phone rang again. This time, it was from another company that I had spoken to almost 2 months before - and they wanted to offer me a job. The role is pretty much what I am currently doing in terms of managing an IT department in a manufacturing environment. It's a big company so there is a lot to do, but I'm confident that I can do a good job for them

What was interesting is that they have actually considered using SAP in the past, but did nothing about it - they still have an old ERP system that seems to have been around since Adam was a lad. I don't know if they will re-consider, but I get the feeling that part of the reason they want to get me on board is that they will have to look again at updating their systems, and want to get someone with experience of an SAP implementation within their staff.

There was a slight hold up on getting the details confirmed and some queries about travel - it appears that they want me to work away from what will be my home site for a while. It appears that they have some specific issues and want these to be addressed - I don't think that it will be possible to get it done in the time allowed, but I think that getting a good understanding is important.  Once that was all sorted, I told my current employer that I would be leaving.

Their response was not that positive - well I'm not that surprised. But instead of asking me to leave immediately which I fully expected, they wanted me to stay on for a couple of weeks. Altho' I've tried to ensure that I spread the knowledge around, there are a couple of areas where I have done more of the work, and they wanted to make sure that there would a smooth hand over.

Anyway, that is now in the past. I start my new role tomorrow and I must confess to be a tad nervous as well as quite excited. I have a weeks orientation in the manufacturing section of their second biggest site, plus they want me to visit a couple of overseas sites as well. I think I made need to get a second suitcase - drop one off, and pick the second up one before heading off again!

The family are also quite excited - and a bit pleased that we won't have to move straight away. It's possible that we may not move at all - I have the option to work from a place not too far away, and still look after all of the sites. Technology, it's what we do - and remote working means that we can be as effective from any given site as any other (or even from home).

So, for the moment that will probably be it. I had thought about adding a couple of extra posts to this blog, just to summarise some of the things that I had learned, but I'm not too sure yet if I will have the time. As you'll be aware, I've kept this blog anonymous as I wanted to make sure that I didn't put myself in the way of losing my job. Altho' I'm moving on. I'm going to keep it that way as I may be working on an SAP project in the future, and don't want to leave myself exposed.

I'm going to say a big thank you to all of you for following this blog over the last 3 years. It's been a roller coaster ride with lots of ups and downs at various times. The comments that many of you have made have been a big help - and I'm so greatful that you made the time to read it. I want to offer a big thank you to the SAP Mentor guys who showed that there are people out there that know and care what is happening. Another big thanks to Bill Wood - I appreciated the offer, even tho' I didn't take it up.

I want you all to know that it's been a great experience writing this blog, one that I would not have missed for the world. I wish you all the best in whatever career you follow and hope that you will all achieve the success that you work so hard for.

Very best wishes to you all.

"Sapmesideways"

The fat lady is warming up

It's been a while since my last post - not because I've had nothing to write about, but due to the amount of work I've had to do.

A couple of months ago, (just after my last post) we had another issue with a consultant doing work that then caused a problem - I didn't actually see what it was as I was at an overseas site and it was fixed before I got back to the office. But it caused a lot of issues with the system performance, to the point where almost no-one could work again.

Having gone thru the relevant monitoring information, I could highlight that most of the time our system worked really well - I also had the relevant metrics to show just how bad the performance was in the period when the problem was occuring. Once the problem was resolved, the performance improved considerably - and I was able to highlight that the issue was not a fault of the system. But the senior managers were not happy with what I was telling them and wanted some confirmation.

They had asked the SI to look at the system the last time there was an issue, but the guy that was sent to site really wasn't a Basis specialist. He checked thru some of the obvious transactions, but didn't really provide any hard facts - he didn't even bother asking for the Earlywatch report which could have given him some hard data. Instead, he just told them that the system was not powerful enough - we needed more processors, more memory, more network cards - basically more of everything.

Now over the last few years, the SI has been quite agressive on the mergers & acquisitions trail - I know that they have acquired at least 3 other companies, and I think that one of these has a stake in a data hosting facility. Last year, the directors from the SI took our shareholders to a junket at a country club, and I'm pretty sure that they were really just promoting the concept of us moving to the hosted system that is now part of their business.

I understand that the shareholders were told lots of things about how great the service is and how affordable it could be. I've also discovered that it was suggested that with a hosted system, I would not be needed, and if I was no longer around, my wages could pay for this - I'm not sure how much the SI thinks I get paid, but I know that if they got rid of the entire IT department, it still wouldn't pay for the hosted system. Added to that, I still have my doubts about how good their service is likely to be.

As it happens, some weeks ago I met up with a guy that is a Basis consultant - he is self employed and has worked for some of the biggest organisations around. I managed to get authorisation to get him in for a couple of days and he has gone over our systems very carefully and provided a detailed report on what he found.

What was clear is that we are not doing too bad - there were a couple of minor tweaks that he suggested might be of some use, but he primarily said the system is well maintained and has sufficient capacity for what we need for the next couple of years. He also highlighted where some of the consultants from the SI have not been following best practice, something that has been annoying me for ages.

But even with all of that, it seems that the decision has been made to go to a hosted system, even if they will not actually come right out and say so. I don't know if there is a timetable yet, but based on a couple of comments, it would see that they are planning to move the systems sometime around the early part of next year - based on how the SI have done in the past, I might suggest that it won't actually happen until the later part of next year.

As you may guess, I was not too pleased to hear this. I have in fact approached the CEO and other directors to see if anyone will explain what is happening - but they seem reluctant to discuss in too much detail. However, they won't deny it either which does make me feel very unsettled. I even put my concerns in writing, highlighting the costs and potential concerns - but still they won't talk to me about any plans they might have.

I must admit that after the problems we faced at the beginning of the year, I sat down with the family to talk about our options for the future. My oldest daughter is at that age where it might cause some issues if we had to move, but they are all aware of what is happening, and they do actually support me. They would obviously prefer to stay here, but if we have to move, so be it. I started brushing off my resume and sent it off to the various agencies. I've also been networking like crazy over the last few months, trying to see if anyone knows of any decent vacancies.

Since March, I've actually had 4 interviews - and they all came back really positively. But so far, nothing yet..... But I am hopeful that I will hear something this week - fingers crossed. Whatever happens, I will let you know.

Monday morning blues

On Monday morning, I was on my way to work when I was called on the cell phone asking if I knew that the Production system was running slow. I didn't, but said I would look at it as soon as I got in. While I was getting set-up I had a couple more calls from people to tell me that things were not going well.

When I eventually logged on, I saw that there was a job that had been running since the Friday before - and it was logged as being one of the consultants that hadn't been on site for some time. I asked around and eventually was told that this particular consultant was due to be on site - he had been organised to look at a particular issue.

When they guy actually arrived, he looked thru the system and checked out what was going on. It turns out that he had started this job remotely because he needed to check on some data - he thought it was going to run for maybe 6 - 8 hours. Instead it was still running after 60 hours and only finished later that day. It also didn't do what he wanted.

After he did some further checks, it turns out that a change had been made to the system that would actually cause an issue with the program that he had tried to run. No-one knew about the details as the guy that made the change is long gone, out of the country and there were no references to the work anywhere. In fact the consultant onsite wasn't entirely sure what change had been made. He did say that it had only been done in the production system - there was no transport for the change, and when he tried his program in the Dev system, it ran OK.

It now appears that we are going to have to pay to get one of their guys in to look at this now - they need to try to figure out what was done and then first correct it, and second, try to do whatever job it was supposed to do properly. I asked if the work would be  FoC (well it was their guy's mistake) but no such luck. In fact, they don't actually know how long it will take to investigate the issue - possibly 4 -5 days which we will have to pay for before they tell us how much it will cost to fix.

This highlights a couple of points. First of all, most of the work that the SI has done has never really been documented, even tho' that was a point raised right back at the beginning. We were told categorically, that they would document all changes that were made - in fact, we have very little evidence for any of their changes.

Another major issue, is that a lot of the work has never really passed any form of Quality Control gateway test. Again this is an issue that was raised on many occasions, and we were assured that we would have the power of sign off on everything. Yes there have been items that we have signed off on, but there are a ton more that we haven't. In some cases, we don't know that work has been done, and are only finding out when things go seriously wrong, and investigation uncovers the root cause.

I've spoken to numerous people and the documentation that we have been given by the SI right thru the business is so poor as to be virtually useless. We have very little information on a lot of the changes, and it's difficult to see what we can do now other than get a specialist in that can uncover some of these, and hope that we can get them documented in future.

I've also highlighted that they often ignore good practice - and here was another example. A change was made directly in the production system without having been tested in the other systems. I'm also concerned that some of our people have learnt these bad habits - I regularly get asked to do something directly in PRD, and if I refuse and point out the correct procedure, I get an argument that the consultants do it that way.

We should have had a proper change management procedure enforced from the beginning, and I will say that when I queried this, I was told that SAP had a robust procedure that the SI staff would follow. There is a procedure, but the consultants don't bother to use it, and it sometimes seems that they see it as a PITA to be avoided rather than good practice to ensure that we don't get problems.

For me tho' the big issue is that all of these problems are completely avoidable - there is simply no excuse. I wish that I had a way to identify just how much of the money that we have paid has actually be wasted on work that does not do what it should, or has never been tested correctly.

I just hope that we don't get off to another poor week this Monday.

Moral musing

Earlier in the week, Greg Smith of Goldman Sachs quit his job. This was a high profile position, and he had a lot of experience, a great deal of authority and was earning a considerable sum of money - yet he felt that he could not continue working there. He arranged for a letter outlining his reasons for quitting to be published - if you want to read it, there is a copy at this link. http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-goldman-sachs.html

I know exactly what Greg was writing about; some years ago, I worked for a company that was doing some things that were morally wrong, and once I realised the situation, I couldn't get out quick enough. But it's always going to be a difficult decision, and I think that he has shown some considerable moral fiber and courage in doing it so publicly.

I have to say that I understand his views and that he is probably doing the honorable thing. I doubt that he will suffer too much - no doubt, he has saved enough to be able to stay off welfare and he may even find that there are other organizations that will be prepared to offer him something as his knowledge will be highly valuable. But it got me thinking about my reasons for doing this blog, so I thought that I would go through them again.

First of all, I will say that when I started writing, it was quite simply to vent my frustration at what I saw was a project that was in serious trouble. Having been a project manager in my past, I was not happy at the lack of communication between the SI and the project team. I could see that work was not getting done on time, project milestones were being missed or ignored, and I had serious doubts about the abilities of the various consultants that were working on the project. Even at that stage with limited knowledge of SAP, I could see that the quality of work being completed was poor at best, and the knowledge transfer was almost non-existant.

However, I am more of a positive person by nature (honest!) and after the first few posts, I wanted to try and make the blog more of an objective analysis of what was happening. I felt that if I could highlight some of the key issues, then others might see these and perhaps this would help them to avoid similar issues in their own projects.

I've kept the blog anonymous - quite simply, if I identified myself, the company that I work for or the SI, I'm pretty sure that before midday of the next working day, I would no longer be working in my current position. I might also find it difficult to get a new job - with the current climate and an ex-employer that won't provide a reference, it would be a bad situation. Although I do have some money put aside, it's no where near as much as Greg Smith - I simply cannot afford the grand gesture.

I will also say that I accept some of my writing is about my opinions - other may look at the same situation and say that things are not as bad as I make out. While I may think that some of the consultants working on the project are not doing their jobs well, that is just my view, and as I can't give them the right of reply, it would be unreasonable to identify them too closely.

I do try to provide an objective view, but in order to keep my anonymity, I have done things such as change dates and sequences of events so that it would be harder to positively identify myself or my company. I don't believe that any of the changes I have made would make a huge difference, but I feel that it is right to make that point clear.

We have an SAP system that is working and thanks to the hard work of the various members of our project team, it is doing a lot of what we wanted. There are still areas that are not entirely satisfactory, and we are working on those - as far as we can, we are trying to build up our internal skills so that we can rely on our own people. If we mess it up, then it will be no-one's fault but ours, and if the consultants want to come back and laugh at our attempts, they are welcome to do so.

There is a possibility that I may come across as arrogant - I take pride in my work, I like to think that I am damn good at my job, and I can point to the fact that the IT in the company is light years ahead of what it was when I first joined them. I've had a number of people visit from outside, and they are generally very impressed with what we have achieved - we are way ahead of the curve for a business of our size. But I don't believe that I have the answer to every single problem, and I am willing to listen to ideas or advice from anyone.

Hopefully, this blog will be a useful resource for some people - if just one person finds something in my writing that helps their project in some way, I feel that it will have been worth it all.

Still plowing on...

I've been tied up with work again over the last few weeks. Sometimes, it seems that the more we do, the less we have achieved.

We are still having problems with our first overseas site. It appears that there are still items of data that have not yet been loaded. Some of the information for foreign language descriptions of products already in the system, some of the purchasing information records, contact data and even some banking data amongst others. Note that I did ask the question more than once and was told each time that all of the data load had taken place.

I should also highlight that none of this data load has been tested in the Development or Test systems - they gave up doing that a couple of months before New Year, and the data load is being done directly into the Production system. Of course, it will do a couple of items, then fail because something is wrong with the actual data (which I accept is down to us) - but then they just correct it and carry on as if this is the most normal thing in the world. When I suggested that it might be better to at least try in the test system, I was told that was a waste of time.

As for data in the Test and Dev systems - It's been agreed that we are going to sort that out by doing a client copy. The last time that was done, it took more than a week to complete and we had all sorts of issues for a couple of weeks after. I hope that we don't get the same this time as I am trying to do a number of other projects and my staff and I will not have time to fix any issues.

Anyway, a couple of weeks ago, we had the guys from one of the other overseas sites visit with us, so that they could see how the SAP system works. I was told that we would see their whole project team, but only 3 people had travelled over. I've since had a chance to speak to the VP for the site, and made some points to him about the need for a full project team - I also put together some descriptions of work required which have been sent to him.

I've been asked to go over to them next week to talk to their people in a bit more depth. It turns out that they have been promised that our project team will all be available to help out, which is going to be a shock to most of those guys, as no-one has discussed this with them. I've received an invite to a meeting in April and I think they will all be told then - I've been asked not to discuss it with them yet.

There is also another issue that is causing a lot of anger. When we first started, the consultants set up a process within the Sales area that did not work the way we wanted - they told us that was how it had to be and was considered "best practice". How we wanted it to work was just not possible. Once it was working, they found some problems and we had to get a couple of people in to fix those.

When the new set of consultants were working on our overseas site, they set things up so that the process worked the way we wanted - and they said that was actually the way they normally do it. They couldn't change the process for us at that time, but have now agreed a date to do some work to re-do some things so that our Sales people can use the same process in the same way.

To re-cap - we paid for about 4 weeks to set it up, then another 2-3 weeks to "fix" it, and now we will have to pay for another 4 weeks to change it to work the way we wanted in the first place. I wanted to challenge them over this, but I've been told that we will just have to suck it up and pay the money.

The best thing - I've recently had an email from the SI asking if I would be prepared to recommend them! They want to use our site to show a "highly successful" implementation, and possibly even use us as a case study. Astonishing!

The Security Stuff (part 2)

I'm following on from last week's post - because I think that there is a bit more to say about this topic.

If you follow any of the people that promote the concept of "Empowerment" within a business, you'll no doubt be aware that many of these people or companies are talking about allowing staff access to more of the information held within their systems, so that it is easier for decisions to be made at lower levels, rather than having all decisions coming down from the top. This is also sometimes described as "agile" business decision making.

There is something to be said for this idea - if someone has to purchase some goods, and has been doing the job for some years, she may well be quite capable of negotiating the right kind of deal without having to refer it to her line manager. This then means that the manager is left to deal with more important issues - and when I was a younger manager, it was known as delegation. A senior manager once told me that if you cannot delegate, you cannot manage.

The problem is of course that not everyone is entirely capable of making decisions at certain levels, and need to be monitored more closely than some others. It also has to be said that there are more than a few managers and senior executives that would be horrified at the thought of more junior people making even quite minor decisons without referring to them. Even if all they are doing is glancing at an email and then replying "do what you think best", they feel that they are "managing the situation". (A bit like the PHB in the "Dilbert" cartoons!)

So it's necessary to decide how you want to secure your date as well as how you need to secure your data (the two are not the same). How you need to secure the data is often defined by regulatory compliance or the need to prevent commercially sensitive information from going astray. How you want to secure the data is more about how you structure the business and how the various processes work.

Most people would agree that when SAP was first created, the main target was the larger enterprise business. Within this size of business it is generally necessary to provide some separation of duties - you don't want the person that creates a new bank account being able to transfer money, otherwise  he might create an account for himself in the Caribbean and transfer $10 million, before taking a one way flight to his retirement (which he also puts on expenses!).

As a result, much of the way that things are structured in SAP security has been done specifically to provide the required levels of security for those large organisations. The problem is that when you get to the smaller businesses like ours, it's actually much harder to define what you need in terms of security. Worse, this can actually change a lot more frequently as staff will often be doing more types of jobs than their counterparts in the enterprise business.

Now we have a problem - I've been trying to make sure that all of the permissions are properly set-up and correctly tested, but even so, sometimes it's not always easy to do this. I will hold my hands up and admit that there have been many occasions when I was going thru a role and suddenly spotted something that didn't look right, that then needed to be corrected.

Partly this is due to some people taking the view that "give them access to everything, and we can take away what they shouldn't have". This doesn't work - why? Because the only time that you find someone can do something that they shouldn't is when they have done just that, and it has caused a problem. And usually, that is when things are going seriously wrong.

We have seen that, not just the one time, but on a number of occasions. Most recently because we have some transactions that were created for us by one of the consultants based upon standard SAP t-codes. Unfortunately, the new t-codes have absolutely no authorisation checks built into them - and as a result, we have found that some people are able to do some work that was supposed to only be done by the Production Manager once a day to clear up any issues.

Oh, and in case you wondered - when I said that I would take the offending t-code out of the role, the Production Manager asked me to leave it in. He is going to speak to staff to try to make sure that they understand that they shouldn't run this particular process, just use the t-code to look at the data. Yes - I'm sure that will work (until the next time).

We'll do the security stuff later.

A couple of days ago, there was a question posted on SDN about security, that got me thinking about how this had been handled during our implementation. If you haven't seen the question, here's a link and I would suggest that it's worth a quick look.

http://forums.sdn.sap.com/thread.jspa?threadID=2133938&tstart=0

Now I am not a security specialist, altho' I have taken a couple of SANS approved security certs. When I was told that we were going to be implementing SAP, I got talking with one of the guys from the training course that I had done, and he was a bit dismissive of SAP security. He made a comment that SAP security was based upon "security by obscurity", and said that he had never met an SAP consultant that even understood the basics of security.

When I had the chance, I asked the SI senior managers if they worked to ISO27001 - they were happy to say that they did and had been fully accredited. Subsequently, this proved to be less than true - they did not actually know what was involved in the accreditation process, and as far as I can tell have never undertaken any of the work that would be required. (I should also say that they also claimed to have a couple of other accreditations that they are not entitled to.)

When we started work on the project, their Project Leader got me started on creating user accounts, and some basic roles - but only really very simplistic stuff. As we started testing, I found that some of the things that I had been told to do were completely incorrect, and I had to make use of SDN and some other resources to identify just where I was going wrong. Later I also did the ADM940 course which filled in a number of gaps, and I've since also done ADM950 & ADM960.

Now I'm not going to say that this makes me a security expert, far from it. There are a lot of people out there that have a great deal more experience with almost no paper qualifications - and I would be happy to learn from them. But the work that I have done within the training that I have undertaken does give me a level of confidence to be able to say that our consultants really did not know some of the most basic requirements of good security practice. And from what I have sinced learned, they don't seem to be in the minority.

The quote at the head of this post is an actual comment from one of their guys - and for me, this shows a real lack of appreciation of what security is really about. For security to be of any value at all, it has to be built in from the start - trying to bolt it on after the work is done is quite simply never going to work. A bit like having a high powered sports car, and trying to add wide wheel tires to give it more grip - it will just never do what it should do.

We are fortunate - we don't have to conform to the level of compliance that a lot of other businesses have to. We get audited once a year and we have a couple of specific checks done, but that is all. I would say that's a good thing for us - if we had to go thru some of the bad practices to correct them, I'm fairly sure that we would still not have gone live.

Security is not just about PFCG and SU01 - it needs to involve so much more within the software, but it also should cover physical access to servers, workstations and other devices, good working practices, and a structured approach to everything that has to be done within the system administration and management. We've seen first hand what can go wrong when this is not done in the approved manner, and the big problem is that when you're trying to fix a problem after the event, you often only treat the symptom, not the cause.

I will say that in most cases, we never really knew who some of the consultants were that we would get from one month to the next. We never saw any information about them or their skills - I'm not even sure if the SI even bothered to check on those either. Heck, I'm not even sure if they checked to see if the person that came to site was the person that they employed!

I can understand that for some people, the drive to get the systems in and working is the top priority, and everything else is secondary. But security should be something that is fundamental to any good system, and it should be in place from the very beginning of the project. Until it is given that level of attention, it will never work the way that it should. And that is only ever going to reflect badly on SAP, no matter who is really at fault.

We don't know, what we don't know

I'm going to go back to the beginning of our SAP implementation project. That was over 4 years ago now, and at times it seems like it was only a very short while ago - at other times, it seems like we have been doing this forever.

When the original plans were put forward, it was made very clear to everyone that this was not a normal IT project. Although there were some technical issues that only IT were getting involved in, the majority of the work was actually to be done by staff from across the business, and would involve people at all levels, in all departments. We knew that it would take time, and altho' the SI promised otherwise, I was fairly sure that it would take a lot longer than planned.

An ERP implementation (of any flavor) is always going to be a much bigger task than most others - it requires a lot more work because so much of the product integrates data and processes, and it is important to make sure that what happens in one area, doesn't impact on another. There are many things to consider, and even for those businesses that have a clear understanding of what they do and how they do it, the implementation project is going throw a few curve balls into the mix.

Added to that, most ERP products (including SAP) will have specific ways of doing things, and it is important to learn how to do these correctly. With most software programs, it's possible to buy a book, work with a demo verison and learn all you need - OK, somtimes, you also need to undertake specialist training, but with an ERP product, this is just not practical due to the size and complexity of the program.

For that reason, it's necessary to buy in the expertise required. Some companies will do this by hiring extra staff that have had previous experience, sometimes on a permanent basis, but more often as contractors. A more common approach is to use a System Intergrator (SI) which will be a organisation that has prior experience of working in the specific area with the particular product. The really big companies will use one of the big consultancy firms - for the SME type business like us, it's more usual to make use of the medium size or smaller SI.

The idea is pretty simple - the SI will supply people (consultants) at a set rate that have the required expertise. These people will then help us implement the software by providing technical expertise, passing their knowledge onto our staff, and guide us thru the various steps. This should reduce wasted time, enable us to get up to speed quickly, and ensure a succesful project. Well, a good idea in theory!

The problem is of course, that when learning new stuff, you don't know just what you need to know. Certainly if you have been involved in projects, you will have an idea of good project management practices - if you have worked with larger companies, you will understand the need for good communications and dissemination of information. You may even know how to manage project teams and stakeholder requirements. But in the case of SAP, if you have not worked with it previously, it is unlikely that you will know much about the practices required by that product, which is why people buy in those consultants.

That of course presumes that the consultants do know the stuff that we don't know and are willing and able to pass that knowledge on to our staff. Unfortunately, as we have seen that is not always the case. As you might imagine, after the issues that we have had to deal with over the last few weeks, there was a meeting to identify what happened and how to prevent it happening again. At the meeting, it was pretty much agreed that the problem was caused by the consultants not following proper procedure - the procedure which they are in fact supposed to be teaching us.

Of course, the SI dispute this - their people are doing their job and trying to help us by getting the work done as quickly as possible. Their arguement is that it is down to us not following correct procedures and incorrect testing of the work.

I will say that I would not totally disagree - I don't think we always do all of the testing that we should. We also should have the settings locked down much more than they are specifically to protect us from this sort of thing happening. But the problem is that if their people don't set and follow correct procedures, how can our staff know what is or is not the correct way to do something? What's worse is that if people learn bad practice, it is generally much harder then to change the way that they work.

Well enough for now. There are going to be some meetings in the next few weeks, and these will have an impact on what we do next. I am actually looking forward to it, altho' I suspect that I may not feel so pleased afterwards!

Spoke too soon...

I did say last time that we had fixed the issues caused by some bad config changes, and that we should be back to normal. Unfortunately, I jumped the gun a bit - we were still having some issues right up to the beginning of the week.

To be fair, these were mostly smaller issues - someone found that they couldn't post a purchase order because one of the items on the list had some mis-matched data. There was a similar issue when the guys in the warehouse were trying to assemble some stock to a delivery. Late on Friday, there were a couple of invoices that went wrong - and all of this was from data problems caused by the consultant loading config changes directly into the Production client.

We've locked the client against config changes now, but in reality, that won't stop it happening again. The consultants have the ability to unlock the client, and there's not a whole lot that I can do to prevent this. I have detailed what we need to do, which is to limit what access permissions they have, but whenever I make that point, the consultants start complaining about being "unable to do their work".

I will say that quite some time ago, I had the chance to speak to one of the consultants that we had on site for only a few days, and from what she said, it is clear that they generally don't get a lot of support or management when they are working on a customer site. She was quite new to the company, and to SAP, and it was clear that she felt a little out of her depth. She was being asked questions by people from our project team, and all she could do was refer it to one of her colleagues - she just didn't have the experience to handle many of the issues herself.

This is an item that has been raised before, and I know that a couple of the more experienced guys that are SAP Mentors that follow this blog, have also raised questions about the skill levels of many consultants. A given person may have a valid certification, but without at least some practical knowledge, that person might not be much more use than one of our own staff.

For me tho' the main issue seems to be that most of the people that have worked on our project don't seem to know much about basic good practice. I understand that time is money, and they want to get the job done as quickly as possible, but cutting corners is usually going to cause problems. Someone else made the comment that you can do it quick, do it cheap or do it right - but not all three, and we have seen that many times.

This is not made any better by not having any one checking to make sure that these people are following the right procedures. It seems that the SI wants to make that our job - OK, I can see why that is their preference, but at the beginning, we would have no idea what that good practice might be. We know more about it now, but in many ways, it's just too late. They have worked to fairly poor practices, and as a result, we also now have some fairly poor practice in place, and it's not easy to change that.

Well, we have a few new things being developed now - I won't be able to write about them for a little while, but I think that we have some interesting times ahead.

It's broken...

Last week, I made a comment that “the consultants were still working on config changes”, but didn’t go into any details. I can go back over that now…

So I made the point that the consultants were still making some config changes, and doing so directly in the Production system. Normally, all changes should be done in the Development system, checked, then transported to the Test system, before being checked again – and only when it is proven that the changes won’t cause any functional issues, do they then get transported to the Production system.

Now there are certain changes that have to be made directly in a system as they cannot be transported. However, as far as I am aware, SAP best practice still highlights that the correct procedure is to make the config change in the Development system and then once it has been tested making the same change directly in the Test system. It then gets tested again before the change is made directly into the Production system.

The reason for this is simple – if you have a live system, any change can have undesirable effects. If your business is reliant upon a system, you want that system to work all the time – you do not want it going wrong, or working incorrectly. If you follow the correct procedure, you should be able to ensure that any change made will not turn around and bite you in the ass.

But of course there are those people that choose not to follow procedures because either they don’t know any better, or because they think that they do know better. (Dare I suggest that some simply do not care?) In our case, I don’t believe that it is malicious, but I feel that the people concerned have never had to work to the appropriate disciplines.

On Monday morning, people found that they were not able to post anything in the production system. As I arrived at work, it became clear that we had a crisis on our hands. The CEO and FD were already there, discussing the problems and trying to get answers on what had happened and why things were not working.

After investigation, it became apparent that one of the consultants had really messed up – the config changes added to the system were physically preventing any new data from being added and quite a lot of the older data from being processed. I got everyone out of the system in case a restart of the server might fix the issue, but that was no good. After many hours of discussion with the people concerned, it became obvious that the only way to fix the issue was to reverse all of the changes.

That work has been going on all week. In that time, no-one has been able to do very much in SAP – there were a few jobs, some reports could be run, but not very much. Fortunately, the factory was able to continue working for several days as we had some info left over from the previous week, but that dried up after about 3 days. At one stage, the directors were even considering closing down production and sending people home and putting plans in place to do this.

However, by midday yesterday, all of the config changes that caused the problem have now been undone, and people were getting down to running the stuff that they had not been able to do. A couple of the staff from Sales will be working on Saturday and they hope to catch up with what they have missed – Finance have managed to organise 2 billing payment runs, and they are staying on for a couple of hours to get the invoices in the mail.

Meanwhile, the Production manager has started to run one of their scheduling jobs, and the Production Supervisor will try to run another later tonight. I doubt that we will be back to normal by Monday, but we should be more or less OK by Tuesday lunch time – evening at the latest.

So of course, everyone is pretty angry about this – but so far, no word of apology from the SI. The CEO has told them in no uncertain terms that any attempt to bill us for the work will be rejected. I think that we ought to get some sort of compensation, but based upon their previous mess-ups, I think that is highly unlikely to happen.

So we have managed to escape a major disaster  –  yes, things could have gotten a lot worse, and other people have suffered far worse things that we have. But I do feel that we should have not been exposed in that way to begin with, and I do get frustrated that we are suffering because of other people’s behaviour with apparently no recourse to compensation.

Not so happy new year

The new year has come and gone; so our overseas site must have gone live, right? Unfortunately, no.

Although we technically stopped work for Christmas, I was getting emails and phone calls right thru the holidays. I started to feel a little concerned as it looked increasingly like they were struggling with some basic stuff; and on New Years eve, I got a call to tell me that the go-live was postponed until the end of January.

Despite the fact that I had asked many times if the data had all been loaded (and been told that it was), the actual situation is that there are still a number of items of master data that have yet to be loaded. Little things like a load of customer and supplier information, bank details, a lot of product information and the open activity etc.

On top of that, the consultants were still working on config changes (more on that next time) up until Christmas eve - but then stopped work until the first week of January, and so there's a substantial number of items outstanding that haven't even been looked at.

I was asked to go over to the site along with the VP of operations - we had a really bad flight over, then problems at the airport with a lost bag. This delayed us getting to site by several hours, which was made worse by the cab driver taking the wrong route.

By the time we got to the site, he was not happy at all. The consultants were watching some video on a laptop and howling with laughter which really did not sit well with him. When he spoke to the consultants, he made the point that he had been told that everything would be ready, and the delay was down to them entirely. He also said that he did not see why we should continue to pay their company if they were not able to do the work that they themselves had agreed to.

Subsequently, he had a conversation with the director of the SI - who didn't apologise for any of the issues or the delay, but suggested that our VP should apologise for being too demanding! To be honest, I stayed out of it as much as I could - but the conversation we had at the hotel that night showed that our VP is not impressed, and I think that he will not leave this point alone until he gets satisfaction.

So, we spent a week on site, going thru the outstanding items, and trying to get these covered off. Surprisingly, we did manage to get a lot done while I was there, and altho there are many areas still not finished, we could be on track and be ready to go-live in about another 3 weeks.

I will say that I am still a bit concerned at the moment. The consultants have the staff at the site doing some training - by entering data into the live system. Now I accept that it is live data (from real orders etc.) but this should be done within the test system, not the production system. But hey, what do I know about anything.

Some of the results they get are not quite right - well that's to be expected if it's not been tested as it should. They have had to go thru some of the data to make amendments, and they had to make a couple of other config changes that had been put into the test system, but not the development or production system. Before you ask, they still haven't put them in the dev system.

So here we are again - waiting for work to be done that I was told categorically had all been finished. I am planning to go back over again in a couple of weeks, but possibly that may be sooner. I'll try to keep you updated.