Enterprise Level Attributes and Data Governance

Lately I’ve been seeing a lot of discussion throughout the MDM community regarding the topic of building data governance around master data and whether or not MDM can be used as a tool to aid this process because of its ability to rapidly develop a governance process using the native workflow engine. The big question seems to be whether or not MDM should be recommended as a workflow tool for companies facing governance problems with master data. Unfortunately it’s not the easiest question to answer because the two options depend heavily on the scenario.

It's been my experience that a common misunderstanding of the "central master data management" scenario is that there is one system of truth for all of the master data in your landscape. As we know, that doesn't provide any real business benefit. The only people that benefit from that are the end-users (data stewards) because they only have one tool they need to worry about. The real business benefit of a central master data management scenario is the ability to identify what pieces of data in your landscape are truly important to your business. What are the attributes that you really care about providing data integrity, governance, and standards around? Separating those and putting them in a tool like MDM can provide a huge benefit to a business because they can easily adapt standards and provide oversight around your "enterprise level attributes" which can help keep your data clean and free of duplicates which ultimately affects the business processes that use the master data.

Now, a lot of customers want to do this, but a lot of them aren't ready for the organizational change that is associated with an implementation like this. If you are going to do a central master data management scenario then the business should form an organization (EDM group?) that focuses on enterprise data, and all they care about is the attributes that have a direct impact to the quality of the business processes affected downstream. I believe the right thing to do here is to try to get the business to adapt a new procedure for governance if they want to go down the "central master data management" path. However, I don’t think it's a good idea to pitch a central master data management scenario if they are simply trying to build a workflow for master data. So one important question is what is the driver here? Are they interested in building a workflow for master data, and you are curious as to how MDM could play a role? Or are they interested in doing a central create scenario and want to figure out how to build workflow around that? If it's the latter of the two I would suggest you try to get them to adopt a new procedure. If it's the first, then I would say leave MDM out of the picture because plenty of clients build workflow around master data without MDM and it works just fine. It's hard to force an organizational change if the business if not expecting it.

The best scenarios I have seen are where there is a team of data stewards that are involved with the governance process of creating the enterprise level record, then that gets sent out to the partner systems (ECC perhaps) where a separate workflow process occurs for the local attributes, and is performed by a different group of users. Putting all of the attributes in MDM can lead to some complex issues, especially if there are multiple partner systems involved, because people can get confused as to which attributes should be maintained for each partner system and ultimately end up decreasing the overall data quality.

Attacking Partner Functions

As many of you know SAP is pitching MDM as the tool for three particular scenarios: consolidation, harmonization, and central data management. The term central data management has led a lot of people to false ideas about the scenario. Based on that phrasing it would seem as though the idea is to incorporate your entire attribute set in one environment, this being MDM of course. The goal would be to have one view of master data across your entire system landscape, whether it is a SAP system or not. However, many of us that have worked with MDM on a central data management project know that this is not exactly the case. The real idea is to identify a subset of attributes that you consider to be enterprise level and manage those centrally. This allows you to easily provide oversight, governance, and data standards to enforce data integrity on the attributes that are the most important to your particular business.

The concept of partner functions being enterprise level can be debated endlessly, and to make a long story short, it’s not really clear whether or not partner functions should (from a functional perspective) be managed in MDM. The reasons to include it are clear. Many businesses that develop a workflow process around the creation of vendors or customers want to have a separate flow depending on the function associated. However, there are some severe technical restraints to building partner functions into your MDM data model which should definitely be considered before you embark down that road.

Let’s use customer data as an example. In R/3 the partner functions are nested inside the sales data of a customer record. If you relate this to an IDoc you can visualize the structure of tables where KNVM represents the sales data segment, and KNVP is the partner function segment which is nested under KNVM. Consequently, when you extend a customer to a new sales area, new partner functions are generated. This is done because you may want to maintain different partner functions based on the sales area of the customer. For all intents and purposes, to model this in MDM you would use a qualified lookup table. Unfortunately, MDM doesn’t support qualified lookup tables being nested within other qualified lookup tables. In other words, if you were to model sales area in MDM you would need to use a qualified lookup table. Then to correctly model partner functions, you would need to build another qualified lookup table which serves as a qualifier to your sales area qualified lookup table. This may sound tricky, but it’s pretty straightforward, unfortunately it’s not supported by the current version of MDM. I’ve heard rumors that this will be supported in future versions, but for the meantime we have to assume that it won’t be and look at what we have to work with in the current version.

Because we are faced with this limitation it forces us to look at alternative options to modeling partner functions in MDM. Let’s say, for example, you decided to use a qualified lookup table to represent your partner functions, and linked it to the main table. You would need to have a non-qualifier which links to the sales organization. Similarly, your sales area would also need a non-qualifier which links to the sales organizations. This is the only way that you can maintain a link between the sales area and the associated partner functions. It may look something like this:

The first problem (and most obvious) that you will encounter is during syndication. In most cases you will have multiple partner functions for each sales area, which may look like this:

When you attempt to syndicate the records out, if you are using the standard IDoc formats provided by SAP (in this case DEBMDM), you will be mapping the sales area to E1KNVVM, and partner functions to E1KNVPM.

What will ultimately happen in this case is the partner functions will all get syndicated regardless of what sales org it’s associated with. So your XML file will end up incorrectly having multiple instances of the same function for each sales area:

To solve this problem you first need to map the sales org to a field within the partner function segment. Like this:

Next you need to employ some logic in XI which can strip out the extra partner functions that aren’t associated with a particular sales area. In other words, lets say you have a customer which is extended to sales org 5, and for that sales org you have 4 partner functions: a, b, c, and d. You also extend to sales org 10, and for that sales org you have 4 partner functions: e, f, g, and h. You syndication file will have 5(a,b,c,d,e,f,g,h), 10(a,b,c,d,e,f,g,h). What you want is 5(a,b,c,d), 10(e,f,g,h). XI can perform this logic for you such as the following:

Basically, you are saying “don’t create the partner function unless it’s associated with the same sales org as the sales area”. It’s not the prettiest solution, but at least it works.

Unfortunately, there are more complex issues at hand. It’s standard configuration in ECC to allow partner functions to be automatically generated whenever you extend a customer to a new sales area. Replicating this functionality is incredibly difficult in MDM. Assignments can’t be used to create new records; similarly they can’t be used to create qualified links. For this reason, it’s impossible to create the default partner functions with standard functionality of the data manager. I’ve explored as all options using workflows, assignments, and any other thing I could think of. This means you basically have to employ a custom solution of some sort. One way to do this would be via the Java API, but I’ve never attempted to do this. The reason you must make sure to generate the partner functions automatically is because if you don’t the IDoc will fail when it is sent to ECC. The inbound function module will attempt to process the sales area extension and when it gets to the partner functions it will blow up if the default partner functions aren’t there. This is a problem because most users won’t remember to enter the partner functions when they are creating a new record in MDM. You can enforce this with a validation, but it will be a nightmare for the user because there is no real intelligent way to relay this information to him.

A further complication exists in the fact that ECC will allow you to change the partner number, but it will validate that the partner number exists in the system before allowing you to commit the changes. In MDM there isn’t a standard way to do this. So for example, if I have a customer record extended to sales org ‘5’, and out of the four default partner functions, I want to change partner function ‘a’ such that the number is customer 000500, we can’t validate that 000500 exists in the system. This will also cause a failure when the IDoc is processed by ECC, ultimately weakening the data integrity.

In short, I think for the time being it’s safe to say that partner functions are pretty much impossible to implement perfectly in MDM. It’ll be interesting to see what developments come in future releases, but for now I don’t think the support is there without developing extensively using an API.

Using the Import Manager

There are a lot of tutorials on the SDN that describe the process of syndicating records out of MDM. One of which I wrote and can be found at this location. Additionally, there are plenty of web logs and guides that can be found online which discuss the general principals of building an interface from MDM to ECC. However, I haven’t seen a lot of good resources for instructing someone how to build an import map. I’ve also noticed that there are a lot of questions on the SDN forums regarding import maps. For this reason I decided to do a small introduction to building import maps with the MDM Import Manager tool. This is an ideal process for several occasions; initial data loads from a source system, and ongoing interfaces with remote systems such as ECC.

The first order of business is to determine what format your source data is in. In my case, I’m going build a map as if it were an inbound interface of data coming from ECC, so I’ll use the standard delivered IDoc structure in the format of an XML schema. I have downloaded the CREMDM04 IDoc in XML format from ECC using transaction WE60. I’ve also created a sample repository to represent the vendor domain. If you have your XML schema handy, we can go ahead and upload it into your repository by going to the XML Schema part of the Admin section in your repository and then adding your schema.

Now we’re ready to build the map, so load up the repository and open the MDM Import Manager. As the Import Manager starts up you will be prompted to enter the file type (XML), remote system (I created one called ECC_INBOUND_VENDOR), and a file containing the data you wish to load. In my case, I generated a sample XML file by extracting data from ECC using a XML file port. If you are using XI, you could have XI generate the file as well. Once you load the file into the Import Manager we can begin by selecting the source table and the target table. To do this go to View -> Source Table, and select the root segment, in my case it is CREMDM04. To select the destination table, go to View -> Destination Table, and select the main table for your repository, in my case it’s called Vendors.

You will most likely have the E1LFA1M segment of the IDoc in your source data, but depending on your source data you may also have some additional segments. In my case, I have E1PADTEL (telephone address data) and E1LFB1M (company code data). In order to add all three of these segments to the map, you need to find them in the Source Hierarchy and add them by right clicking and selecting Lookup, then choose the fields you want to include in the map. To make things simple, I am going to select all fields, however in a real scenario you would want to include only the fields that are going to be interfaced over from XI to optimize performance of the import server. In my case, as I mentioned earlier, I have two more segments in my source data which are sub-segments of the E1LFA1M node. To include those, drill down into the E1LFA1M segment until you find the other segments, and follow the same procedure to include them in the map.

Next, you can change tabs to the Map Fields / Values tab and begin the mapping process. This is where you select the field from the source and determine where the data will go when it’s imported into your repository. For example, if you are using the CREMDM04 format, then you can simply find the element named LIFNR in your source, and map it to the Vendor Number field in your repository by selecting the element on the source side, selecting the Vendor Number field on the target side, and pressing the map button between them.

Once you have done this you should see a green arrow icon appear next to the fields to indicate that they are mapped.

Performing the mapping can take some time, but to speed the process up SAP has delivered some mapping spreadsheets which tell you what fields in the CREMDM IDoc need to be mapped to each corresponding field in the standard delivered vendor repository. Most customers tend to create their own repository which is optimized for only the enterprise level attributes, so you will probably either need to generate a mapping spreadsheet yourself, or have a functional analyst provide one to you. In any case, once you have performed your mappings, your ready to go to the next step.

Switch tabs to the Match Records tab. This is where you will select what your key is. What this means more specifically is that when records are imported into MDM, you need a way to identify whether the record already exists, or if it’s a new record altogether. In most cases, the vendor number is used as the key, since this will always be unique. I will use the vendor number as the key in my example. Select the vendor number field in the Value Matching section, and press the add button.

Once you have done this, MDM will then a list of the records to import, and give you the number of records that are new creates, and the number of records which already matched the key in the repository. Now you have to select what you want your import action to be. In our case, we want our import action to be “Create”, which will generate a new record for all that didn’t match on the key.

We’re almost done! Switch to the Import Status tab and view the action items. If there is nothing left for you to do, the action item will say “Ready to import”. Many times, you will have to go back to the mapping tab to complete a mapping. The status may say “Map Account Group Values” or something like that, which will indicate that you have source data being mapped to a check table in your repository, so in order to maintain data integrity, you need to select the value in the lookup table for each of the values in the source field. Often times the Automap button will work for this.

When you are ready, press the red exclamation point at the top of the screen that says “Execute Import” and your import will get kicked off! That’s all there is to it! Piece of cake, right?

I hope this was helpful to anyone that’s getting started with the import manager!

Generating the Number

One of the most popular issues that I’ve seen in a central create scenario is the question of how to generate the number. If the creation of a new record occurs in MDM, then how does the ECC number get generated? Everyone seems to want MDM to operate the same way that ECC does, which allows you to configure different number ranges based on material types or account groups (depending on what business object you are working with). Unfortunately this is a difficult task to do for several reasons. I’m going to first address why it looks simple to do, but isn’t quite as simple as it looks. Then I’ll talk about some possible technical solutions that I have experimented with. Finally, I’ll address the issue the correct way, which is by designing the correct system architecture to support global numbering.

At first it seems like creating the number is a simple task in MDM. After all, MDM has a field type called Auto ID which generates a new unique number that increments by one each time a new record is established in the repository. Additionally, you can configure a lookup table to store account groups. For those of you that are familiar with MM configuration in R/3, you know that it’s a pretty simple process to configure a number range. You can simply define your account groups in IMG, and then use transaction XKN1 to define the ranges. At first I thought a decent solution would be to use an assignment that would take the Auto ID number and add that to a range.

Unfortunately, there’s one major issue with this approach. There will be gaps in the number range. It doesn’t matter if you create a PAY, CARR, or ICOM record, the Auto ID is going to increment its value regardless. Therefore, your number may be 90000030 even though you’ve only created one PAY record, because there were 29 other records created in another range. This becomes a big issue if you want to use small ranges because you can quickly run out of numbers for a particular account group.

So I decided to try taking it a step further by maintaining a table with the ranges. It seems simple enough, until you try to increment the range value. If you try to use an assignment on the main table then you’ll quickly discover that you can’t modify a record in a lookup table from an assignment on the main table. Furthermore, you can’t use an assignment inside the lookup table because it is unable to read data from the main table so it doesn’t know what the account group is.

So the next thing I tried was using a workflow. I figured I would branch the workflow based on a validation that checked the account group. Once the workflow had branched I would call an assignment in the lookup table that would increment the correct range. So I would have three assignments, each would increment only one range, and only one would get called depending on the path the workflow took from the validation. This seemed like it would work well except that if a workflow is built on the main table it can’t reference assignments in a lookup table. Also, a workflow on the main table can’t launch a workflow from inside a lookup table. This pretty much means I was stuck with no good way to increment the value of a range if and only if it was a particular account group.

So where to go from here? The next best thing would be to use the qualified range feature of the remote systems in the MDM Console. Basically, you can configure a number range for your remote key based on the system that you are syndicating to.

Then, in the syndication map, you map the remote key to the number field. Then, when you syndicate a record out of the repository it will take the appropriate number range for that remote system which meets the criteria of the record and populates the number field in the file generated by the syndicator. This actually works really well; the only downfall is that the number is not generated in the repository, only in the file which is syndicated. This is a problem for most businesses because they want to be able to see the number in the MDM repository so they know what the number is in ECC.

This brings us to my alternative solution, which is to build a custom program in the Java API that generates the number. I hate building custom programs because I try to stay as close to standard functionality as possible, however sometimes if the business requirements are strict you simply can’t avoid it. On one of my projects they absolutely wouldn’t bend this requirement, and by the time they realized how difficult it was to generate the number in MDM, the number ranges were already solidified because other teams had started building their configuration around them. This meant we had to solve the problem in MDM. In order to write a custom program to generate the number, you have to think about how that program is going to get called. Additionally, you need to consider the longevity of the solution. We integrated the custom code into a Portal iView so the user didn’t even know it was getting called in the backend. Besides, the decision had already been made that all end-users were going to use an iView.

First, I built a table in the data model to store the account groups. Next, I built a table called Number Ranges which stores the range, as well as a key which is a unique identifier for each range.

Finally, I built a table called Key Mapping which stored a list of the account groups, and then had a field which was a lookup to the Number Range table in which we could map the key of a range to an account group.

So the Java program would start by first grabbing the account group that was selected for the record, then jumping to the Key Mapping table and finding out what key was associated with that particular account group, and finally it would go into the Number Range table and find out what range was associated with that key. Then it would increment the number and assign it to the number field on the main table. This solution allowed the user to add new account groups, and change the ranges in the future and the Java program would work just the same because the ranges weren’t hard coded into the program.

Now, before I give anyone the wrong idea, I want to mention that in my opinion none of the methods mentioned above are any good. I personally believe that the best way to generate the number is by designing your landscape and integration architecture correctly. The best implementation that I've ever seen used a global number in MDM at the time of creation (new record), and then each corresponding remote system (ECC included) generated it's own number when the record was interfaced in. That number was then sent back to MDM and updated in the key mappings table. The idea is that MDM stores a global key, and then the key mappings table stores reference to all the numbers in your partner systems. I really do think this methodology is using MDM the way it was designed to work.

Why Workflow in MDM?

The ability to develop workflows around master data in MDM is a highly valued feature of the product. Traditional business workflow in R/3 has long been a crucial part of any implementation, and SAP provides many workflows for standard processes. If there isn’t a delivered workflow for a certain process you can develop your own. It’s long been one of the most powerful features of SAP. However, SAP never addressed the issue of workflows for master data until MDM.

Before I get too deep into this topic I should at least explain what a workflow is to anyone who may be reading this and is unfamiliar with the term. A workflow is simply the way a task is distributed and passed among different people in an organization. For example, if you would like to request vacation there is a workflow process. You submit your request, therefore creating a task for your manager who must either approve the vacation request, or reject it. If he rejects it, he can either reject it altogether, or he could pass it back to you to modify, therefore creating a task for you to complete. Each task would represent a work item in a workflow. The way that tasks are transferred from you to your manager would represent the flow.

So why do companies want to develop workflow around master data? Many businesses see a major value in developing a process for approving a master data record before it gets established in the R/3 system. A typical example of this would incorporate an accounts receivable department. For example, let’s say you want to create a record in your system for a customer that you want to do business with, however someone in another department (maybe AR) discovers that the credit rating for that customer is bad, so they don’t want to do business with them; therefore you don’t want to establish a record in your system. This is a bad example, but it illustrates the point. Another reason for developing workflow around master data is to maintain data integrity by having different processors and stewards review and edit the records after the initial entry. As we know, one of the great strengths of using MDM as part of an EDM project is to cleanse existing data. Of course, this doesn’t do too much good if you let your master data get dirty again, so developing a workflow can help prevent this from happening. Automated data integrity through system validation checks is one way to maintain data integrity, but nothing beats good old fashioned human review, especially if the two are combined.

Unfortunately, SAP never developed a workflow procedure for the creation of new master data in the traditional applications. Many customers have chosen to adopt a paper workflow process. In other words, they use a form that is filled out and passed from person to person with a final sign off, and then someone goes into SAP and manually creates the approved record. A major drawback to this is the fact that you can’t get the native system validations in most paper workflows, such as check tables and data validations. If you want to add a new vendor for a particular purchasing organization, how do you know what reconciliation accounts are available for that purchasing org without the help of the MM configurations? Another drawback is the fact that paper workflows typically take a long time complete, therefore affecting efficiency.

The customers that have chosen to develop workflows in R/3 (or ECC) have run into many common problems. The first issue is that once a record is established in your business system, it’s very difficult to get rid of. In fact, the only way to get rid of it is to “flag” the record for deletion; you can’t simply delete the record. Additionally, if a record is established in your system you need to prevent it from being used until the record has completed its workflow procedure. One way to do this would be to block the record at all levels, then of course you need to remember to unblock the record when it has finished the workflow. Some companies have opted for more complex methods, such as developing a z-table (SAP terminology for a custom table in the database) to store the record data while it is part of a workflow. An ABAP program can then enter the data into the correct system tables once the record is approved. Again, you run into system validation issues in the sense that a z-table won’t contain the traditional logic configured in MM, so you would need a workaround for this.

SAP’s answer to this common issue is MDM. With MDM you can easily (I mean, really easily) develop workflows around your master data maintenance process and change them on the fly. The MDM can be integrated into Microsoft Visio with a plug-in that is downloadable on the service marketplace. This allows you to build workflows in Microsoft Visio, and they represent an actual process in MDM. It’s very cool considering most functional analysts use Microsoft Visio to draw the flow for the spec anyway, so it just makes sense that the Visio flow is the actual technical workflow.

Building the workflow in MDM can satisfy the majority of issues that companies face with a paper process for master data. For one thing, the data integrity can be upheld by the use of validations. Validations are expressions which are built into MDM that can check if a certain criteria are met for a certain attribute, or set of attributes. An example of this would be to make sure a number that is entered falls within a certain range. The expression builder is impressive, and lets us build rather sophisticated logic very easily. This solves the issue of upholding data integrity validated at the time of entry, therefore less work has to be done in the long run. Another major benefit to using MDM as the workflow engine is the ability to delete the record entirely if it is rejected. In MDM you can actually delete records and there is no trace of them in your business system. That of course leads me to another benefit; you don’t have to worry about blocking the record because it’s not in your business system at all. Additionally, once the record is approved it can be sent out to all the partner systems at the same time.

There are still issues to be sorted out with the MDM workflow engine, but in general it’s a major improvement to what was currently delivered as a recommended process for master data, that of course being nothing. The most notable drawback to using MDM as the workflow engine is that you can’t get some of the very complex data validations that are inherent in MM and SD configuration. For example, you can configure the sales offices that are available for each particular sales area. Because a sales area is a combination of three values from three different tables, you have a complex relationship between the sales area and the sales office in the database. Performing this sort of data validation in MDM is extremely difficult, and generally not recommended. It can be counter argued that your sales office is most likely not an enterprise level attribute, and consequently doesn’t belong in your MDM data model to begin with, especially since it will probably not have any bearing on weather or not the record is approved.

I’ve done some rather extensive work with MDM workflows, so I’ll be posting some technical pointers in the near future, but I first wanted to explain the benefits of MDM workflow and why it’s useful.

Introduction

The concept of enterprise data management (EDM) has obviously been around for a while, but recently the idea of centralizing master data and developing standardized processes around it is a newly branded concept for SAP. With the new tool, MDM, businesses are changing the way in which they work with master data, from both a process perspective and a technical perspective. I have been working with the SAP MDM tools for quite some time now, and I’ve been working with master data for even longer. I’ve noticed that as the demand for MDM has increased, so has the curiosity among the technical community. The SDN and SAP help pages have plenty of good information, but there aren’t a whole lot of other places to find documentation and lessons learned.

I have decided to start this page to serve partially as a technical journal of the issues that I encounter within the EDM domain at work, as well as a “tips and tricks” page for others. I’m also hopeful that I can meet other people who are interested in working with MDM and other master data technology. I have previously written blogs on the SDN, and a blog series that I wrote was even featured on the MDM home page on the SDN. However, when I write on the SDN I feel as though I must conform to a certain standard, and I can’t add a certain personality to the posts. For this reason, it’s a good idea for me to start a separate page that I can use to write whatever I feel. Aside from discussing issues that I’ve encountered, and posting MDM tips, I also would like to discuss general EDM philosophy here, which is something I feel rather awkward about on the SDN. I have been thinking about starting a blog such as this for a while, but stopped myself because I didn’t want to admit that I’m interested in what I work with. However, I’ve given in and come to the realization that there is no escaping the fact that I am indeed a geek. Unfortunately, there’s no hiding it, so I may as well write about the stuff that I like doing.

MDM is one of the trickiest components of a SAP landscape to work with, because finding good resources can be a difficult process for companies. From my experience, there are two types of people that get into MDM. First, there are the old school MM (materials management) and SD (sales and distribution) functional consultants who have worked with master data for many years. Now that many clients of SAP are finding a major benefit in centralizing their master data and going through a harmonization process, many of the MM and SD consultants are learning MDM to help their clients. The other type of people that I’ve seen working with MDM is technical folk who just want to learn the new tool. The functional consultants often have a hard time adjusting to MDM because while they are used to doing hands on configuration work, few of them have experience with real techie topics such as interface development, or programming logic. Alternatively, the technical folk that get into MDM have a difficult time because the whole purpose of MDM is to serve as a tool for the EDM process. For this reason, you really need to know a lot about the types of data you are working with, and how it trickles down into the touch points within the business. Techies rarely have this understanding, so it’s difficult for them to be well rounded as a MDM consultant. So MDM really is a lacking skill set among the SAP community right now. In order to bridge that gap I think it’s very important to meet new people who are working with the technology and gain experience from others. That is another main reason for starting this blog. I want to meet new people, learn from them, and hopefully give something back as well.

I suppose that’s enough of an introduction, I look forward to speaking with anyone who reads my blog and is as interested in EDM as I am.