A Patch in time

Software engineering is a notoriously difficult discipline. You get the customers requirements, then write the code, only to find that they forgot to tell you something. You modify the code, then they ask for another change. You add that modification, but it doesn't quite work correctly - they gave you the incorrect sequence of events, or you made a spelling mistake. Each iteration of change brings you closer to the desired result, but then there is another problem caused by an external factor - a change in hardware, an updated driver that's not compatible, a new legal or regulatory condition.

So for as long as there has been software, there have been modifications to software. In the early days of mainframes, these were added directly to the code on the machine by the programmers. As PCs became more common, and more software was being produced, it became impossible for the programmer to make the changes and most users wouldn't know how to. So the concept of the patch came about - a means for the programmer to make a modification to code and then issue it to the end user, without having to completely re-install the software.

The biggest example of this of course is Microsoft. Back in the 90s, they developed a process of patching and slowly began to automate this to ensure that as far as possible, the end user PC would be protected against software faults of whatever type. This works well for the average user, but within a business, it can be a bit of a PITA. Imagine having 500 PCs all suddenly accessing the Internet to download the latest patches - it would cause a lot of problems with bandwidth. Now imagine the problem with 5,000 or 50,000 PCs.

Now we don't have quite that number, but it's still an area that we need to control. We looked at the Microsoft options for automated patching, but weren't completely happy. About 3 years ago, we found a software product that actually manages the patching process and does it very well - and not just for the OS, but for lots of other software products as well. It downloads a single copy of each of the patches and places them in a repository at each one of our sites, then automatically scans and updates the machines in a controlled way. We've found that it works very well indeed - and it saves us a great deal of time.

Unfortunately, it's not possible to add SAP into this process. It appears that they are developing a process for managing patches thru Solution Manager, but as our system integrator people didn't install it it, we've got a rather laborious manual process instead. We are at the early stage of setting up a Solution Manager system and I hope that we will be able to use it eventually, but it has to be fitted in amongst other tasks.

Now when the consultants first arrived nearly 3 years ago, they told us that they would have the first system (DEV) set-up and working within a week of the original project launch date ready for the BPOs (Business Process Owners) to start working on the sandbox client. They also said that the other 2 systems (QAS, PRD) would then be completed within another 2 weeks. Unfortunately, the initial project launch meeting was delayed by 2 weeks and they didn't start work on installing the DEV system for a further 3 weeks. This wasn't completed straight away and it was a further 6 weeks before it was ready for use. The other 2 systems weren't even started at that stage and in fact, neither was ready until about a month before our original go live date, some 9 months after the project launch.

At the time of the first delay, we didn't even have data in the QAS system, let alone the PRD system. As it had taken so long, I raised the question of establishing if there was a need for patching the software, but they just brushed the question aside. I did some research on the topic, and found out about the transaction SPAM (Support Package Application Manager I believe) - a rather unfortunate name for adding patches!

When I first ran the transaction, I wasn't really sure of what I was looking at - there seemed to be a number of green items in the list of applied patches, but rather a lot of yellow ones as well. After further research, I discovered a description of the process to add these items. It wasn't entirely accurate. but it was sufficient for me to turn most of the yellow items to green and I was rather pleased with myself. Unfortunately, the same process had to be carried out in each system - a lengthy process. But after several weeks, I had finally sorted all of the items out, including a update to the SPAM transaction itself and a kernel update as well.

It later turned out that the items in the list were just a series of patches that had been loaded when the systems were installed, but the guy hadn't completed the process. I then found that there were a lot more that hadn't been applied that weren't in the list. Unfortunately, I haven't found a way to work out what patches are missing apart from looking at the patch number and searching on the SWDC (Soft Ware Download Center) of the SAP portal for the ones with the next numbers.

In the early part of last year, I started to download these patches - I then hit another snag. After a certain date, all new patches have to be confirmed before they can be downloaded. This has to be done within a Solution Manager system - as we don't have one, a bit of a problem. We have a way around this, thanks to SAP support, but this was the point at which I realised we would have to have a SolMan system.

As if that was enough of a stumbling block, a few months later, SAP introduced a new system of "Maintenance Certificates". These have to be downloaded in the same way as a normal license, but they only last for 3 months. If you don't have a valid maintenance certificate, you cannot apply a patch. It turns out that none of the consultants were aware of this, and I've since found that there are some SAP personnel that didn't know about it.

Despite all this, I started to get the patches applied to the systems. The DEV system was completed without issue right up to the last patch - but that then caused a problem. It turns out that the consultants had made a configuration change but neglected to advise anyone. Adding the patch caused a serious problem with a couple of processes. They said that they would fix it and tell me what they had done, but I heard nothing more. In the mean time, they asked that I not patch the other two systems.

Now that was about 10 months ago. I sat down recently and worked out how many more new patches have been issued and it is a heck of a lot. I know that some of them will only take about 10 - 20 minutes to apply, but others will take 2 -3 hours or more. I did test with one on the QAS system - it caused an issue. However, thanks to a response from the forum on the SAP Developers Network, I now know what is required to fix most issues that occur during the patching.

The problem is that I think we are now going to have to follow a very tedious process for many months to come. Each patch will have to be applied to the QAS system, wait to see if breaks anything and then work out what is needed to fix it before we try on the production system, This has to be fitted in amongst other jobs and the PRD system can only be done at weekends. As you can imagine, this goes down well with the family!

The estimate is that there is about 80 - 100 hours of work just to get the QAS and PRD systems to the same level as DEV. Probably 6-8 weekends if we are lucky and work evey weekend - however that will not happen, so it's more likely to take over 4 months. After that, then with a bit of good fortune, we may have our SolMan system running and we can see if there is a less labor intensive process.