Software engineering is a notoriously difficult discipline. You get the customers requirements, then write the code, only to find that they forgot to tell you something. You modify the code, then they ask for another change. You add that modification, but it doesn't quite work correctly - they gave you the incorrect sequence of events, or you made a spelling mistake. Each iteration of change brings you closer to the desired result, but then there is another problem caused by an external factor - a change in hardware, an updated driver that's not compatible, a new legal or regulatory condition.
So for as long as there has been software, there have been modifications to software. In the early days of mainframes, these were added directly to the code on the machine by the programmers. As PCs became more common, and more software was being produced, it became impossible for the programmer to make the changes and most users wouldn't know how to. So the concept of the patch came about - a means for the programmer to make a modification to code and then issue it to the end user, without having to completely re-install the software.
The biggest example of this of course is Microsoft. Back in the 90s, they developed a process of patching and slowly began to automate this to ensure that as far as possible, the end user PC would be protected against software faults of whatever type. This works well for the average user, but within a business, it can be a bit of a PITA. Imagine having 500 PCs all suddenly accessing the Internet to download the latest patches - it would cause a lot of problems with bandwidth. Now imagine the problem with 5,000 or 50,000 PCs.
Now we don't have quite that number, but it's still an area that we need to control. We looked at the Microsoft options for automated patching, but weren't completely happy. About 3 years ago, we found a software product that actually manages the patching process and does it very well - and not just for the OS, but for lots of other software products as well. It downloads a single copy of each of the patches and places them in a repository at each one of our sites, then automatically scans and updates the machines in a controlled way. We've found that it works very well indeed - and it saves us a great deal of time.
Unfortunately, it's not possible to add SAP into this process. It appears that they are developing a process for managing patches thru Solution Manager, but as our system integrator people didn't install it it, we've got a rather laborious manual process instead. We are at the early stage of setting up a Solution Manager system and I hope that we will be able to use it eventually, but it has to be fitted in amongst other tasks.
Now when the consultants first arrived nearly 3 years ago, they told us that they would have the first system (DEV) set-up and working within a week of the original project launch date ready for the BPOs (Business Process Owners) to start working on the sandbox client. They also said that the other 2 systems (QAS, PRD) would then be completed within another 2 weeks. Unfortunately, the initial project launch meeting was delayed by 2 weeks and they didn't start work on installing the DEV system for a further 3 weeks. This wasn't completed straight away and it was a further 6 weeks before it was ready for use. The other 2 systems weren't even started at that stage and in fact, neither was ready until about a month before our original go live date, some 9 months after the project launch.
At the time of the first delay, we didn't even have data in the QAS system, let alone the PRD system. As it had taken so long, I raised the question of establishing if there was a need for patching the software, but they just brushed the question aside. I did some research on the topic, and found out about the transaction SPAM (Support Package Application Manager I believe) - a rather unfortunate name for adding patches!
When I first ran the transaction, I wasn't really sure of what I was looking at - there seemed to be a number of green items in the list of applied patches, but rather a lot of yellow ones as well. After further research, I discovered a description of the process to add these items. It wasn't entirely accurate. but it was sufficient for me to turn most of the yellow items to green and I was rather pleased with myself. Unfortunately, the same process had to be carried out in each system - a lengthy process. But after several weeks, I had finally sorted all of the items out, including a update to the SPAM transaction itself and a kernel update as well.
It later turned out that the items in the list were just a series of patches that had been loaded when the systems were installed, but the guy hadn't completed the process. I then found that there were a lot more that hadn't been applied that weren't in the list. Unfortunately, I haven't found a way to work out what patches are missing apart from looking at the patch number and searching on the SWDC (Soft Ware Download Center) of the SAP portal for the ones with the next numbers.
In the early part of last year, I started to download these patches - I then hit another snag. After a certain date, all new patches have to be confirmed before they can be downloaded. This has to be done within a Solution Manager system - as we don't have one, a bit of a problem. We have a way around this, thanks to SAP support, but this was the point at which I realised we would have to have a SolMan system.
As if that was enough of a stumbling block, a few months later, SAP introduced a new system of "Maintenance Certificates". These have to be downloaded in the same way as a normal license, but they only last for 3 months. If you don't have a valid maintenance certificate, you cannot apply a patch. It turns out that none of the consultants were aware of this, and I've since found that there are some SAP personnel that didn't know about it.
Despite all this, I started to get the patches applied to the systems. The DEV system was completed without issue right up to the last patch - but that then caused a problem. It turns out that the consultants had made a configuration change but neglected to advise anyone. Adding the patch caused a serious problem with a couple of processes. They said that they would fix it and tell me what they had done, but I heard nothing more. In the mean time, they asked that I not patch the other two systems.
Now that was about 10 months ago. I sat down recently and worked out how many more new patches have been issued and it is a heck of a lot. I know that some of them will only take about 10 - 20 minutes to apply, but others will take 2 -3 hours or more. I did test with one on the QAS system - it caused an issue. However, thanks to a response from the forum on the SAP Developers Network, I now know what is required to fix most issues that occur during the patching.
The problem is that I think we are now going to have to follow a very tedious process for many months to come. Each patch will have to be applied to the QAS system, wait to see if breaks anything and then work out what is needed to fix it before we try on the production system, This has to be fitted in amongst other jobs and the PRD system can only be done at weekends. As you can imagine, this goes down well with the family!
The estimate is that there is about 80 - 100 hours of work just to get the QAS and PRD systems to the same level as DEV. Probably 6-8 weekends if we are lucky and work evey weekend - however that will not happen, so it's more likely to take over 4 months. After that, then with a bit of good fortune, we may have our SolMan system running and we can see if there is a less labor intensive process.
Sunday, 28 February 2010
Thursday, 11 February 2010
Looking over the parapet
When I first started as a junior manager, one of the more senior people made a comment to me - when you are working, every so often you need to lift your head up over the parapet to see what's going on. It's very easy to get tied up in the daily work and the various issues involved in that and forget to take a look at the bigger picture. So I thought that I would take a break from writing about the daily problems that we face on our SAP project and a take a more helicopter view. (Of course there is a chance that I could get my head shot off!)
Prior to 2006, each of the separate business units was operating its own systems - and there were a lot of those. Of course, this lead to problems - systems didn't integrate well (if at all), data was not transferable, they were awkward to manage and maintain and required considerable administration. Although the staff didn't see this, the were also costly to operate - in most cases, we had people employed just to manipulate data, without adding any value to the work that they did or benefit to the company.
The senior managers knew that this was an issue - some of the departmental heads also realized this (although most didn't and probably couldn't have cared less anyway). It was fairly clear that the situation had to change and that there was a prime case for an ERP system. The main arguments were straight forward - one time data entry instead of multiples, consolidated data processing, cost savings from reduced labor costs, and more agile response to market trends and customer demands. It was also thought that better and more informed decisions would be available throughout the business, streamling the decison making process. Eventually, the choice was made to go with SAP.
Over the past few years, I've read a number of books, white papers and various other materials - many say the same thing, that implementing SAP is a Business Change project not an IT project. I would totally agree, but would add that this is true for a lot of what we would normally refer to as an "IT Project". For example, we installed a new telephone system, an IT project, right? Wrong! - a business project as it changed the way that a lot of people worked. We've also installed a new WAN link and that must be an IT Project. No - again, it changes the way that things are done throughout the business so it is most definitely a business change project. In so many cases, there is an "IT project" but that is a smaller part of a larger business change project.
When the consultants organized the very first meeting, they gave a PowerPoint presentation which I still have. They made this very point about the SAP project being a Business Change project and there it is on slide 17, bullet point 3. Unfortunately, they didn't emphasize this and a lot of people still saw it as an "IT Project".
The problem is that so many people automatically assume that they are not involved in an "IT project" as it will be too technical, and immediately switch off. I suspect that happened to a number of the members of our project team. That did change eventually, and I think that most of the project team now accepts that it is not just IT. However, that is not the case for all managers and certainly not all staff. Far too many still see the project as purely an "IT Project" and treat it as such, and I think that this is an area where we have not done very well.
There is no question that we have tried to get everyone involved in the project - we've allocated resources, supplied relevant information, and provided the means for everyone to play with the Sandbox system so that they can learn about it. People are encourage to use the training materials to learn and make constructive criticism. Considerable efforts were made to ensure that everyone was involved - but depsite this, there are still senior managers, key users and other staff memebers that really don't know what they are supposed to be doing. No matter what we say, they still see it as an "IT project" and therefore nothing to do with them.
So what to do? I wish that I could answer that. Clearly it has to come from thetop, and the CEO needs to make sure that all senior managers are on board. It could be argued that we could have done a better job of communicating with staff. I won't accept that we haven't tried, but I woulkd have to acknowledge that we haven't gotten thru to everyone. I think that this will have an impact - without everyone at least trying, it will take longer to get everything working as it should do.
Unfortunately, I suspect that this will have an impact on our future use of SAP and will mean that we won't get some of the payback that we should - or at least, not as soon as we should do. But then, we are dealing with real people and this is a common issue for many projects and not just endemic to our company or SAP. People are creatures of habit and hate change - and there are those that will be atagonistic to any proposed transformation, no matter what the benefits. But then - that's life!
Prior to 2006, each of the separate business units was operating its own systems - and there were a lot of those. Of course, this lead to problems - systems didn't integrate well (if at all), data was not transferable, they were awkward to manage and maintain and required considerable administration. Although the staff didn't see this, the were also costly to operate - in most cases, we had people employed just to manipulate data, without adding any value to the work that they did or benefit to the company.
The senior managers knew that this was an issue - some of the departmental heads also realized this (although most didn't and probably couldn't have cared less anyway). It was fairly clear that the situation had to change and that there was a prime case for an ERP system. The main arguments were straight forward - one time data entry instead of multiples, consolidated data processing, cost savings from reduced labor costs, and more agile response to market trends and customer demands. It was also thought that better and more informed decisions would be available throughout the business, streamling the decison making process. Eventually, the choice was made to go with SAP.
Over the past few years, I've read a number of books, white papers and various other materials - many say the same thing, that implementing SAP is a Business Change project not an IT project. I would totally agree, but would add that this is true for a lot of what we would normally refer to as an "IT Project". For example, we installed a new telephone system, an IT project, right? Wrong! - a business project as it changed the way that a lot of people worked. We've also installed a new WAN link and that must be an IT Project. No - again, it changes the way that things are done throughout the business so it is most definitely a business change project. In so many cases, there is an "IT project" but that is a smaller part of a larger business change project.
When the consultants organized the very first meeting, they gave a PowerPoint presentation which I still have. They made this very point about the SAP project being a Business Change project and there it is on slide 17, bullet point 3. Unfortunately, they didn't emphasize this and a lot of people still saw it as an "IT Project".
The problem is that so many people automatically assume that they are not involved in an "IT project" as it will be too technical, and immediately switch off. I suspect that happened to a number of the members of our project team. That did change eventually, and I think that most of the project team now accepts that it is not just IT. However, that is not the case for all managers and certainly not all staff. Far too many still see the project as purely an "IT Project" and treat it as such, and I think that this is an area where we have not done very well.
There is no question that we have tried to get everyone involved in the project - we've allocated resources, supplied relevant information, and provided the means for everyone to play with the Sandbox system so that they can learn about it. People are encourage to use the training materials to learn and make constructive criticism. Considerable efforts were made to ensure that everyone was involved - but depsite this, there are still senior managers, key users and other staff memebers that really don't know what they are supposed to be doing. No matter what we say, they still see it as an "IT project" and therefore nothing to do with them.
So what to do? I wish that I could answer that. Clearly it has to come from thetop, and the CEO needs to make sure that all senior managers are on board. It could be argued that we could have done a better job of communicating with staff. I won't accept that we haven't tried, but I woulkd have to acknowledge that we haven't gotten thru to everyone. I think that this will have an impact - without everyone at least trying, it will take longer to get everything working as it should do.
Unfortunately, I suspect that this will have an impact on our future use of SAP and will mean that we won't get some of the payback that we should - or at least, not as soon as we should do. But then, we are dealing with real people and this is a common issue for many projects and not just endemic to our company or SAP. People are creatures of habit and hate change - and there are those that will be atagonistic to any proposed transformation, no matter what the benefits. But then - that's life!
Subscribe to:
Posts (Atom)