Now you want to (or have to) compare data from different sources, correlate, mix or combine it in every which way. Reasons for this abound, a couple that we keep seeing at our customers we would like to describe in a little more detail.
Sales Data from Wholesalers
Let us assume that you do not directly deliver to your customers, but indirectly through wholesalers. I.e. your end customers are not your direct customers, but customers of your customers (namely the wholesalers). Now you would like to have the sales of said wholesalers in your system, itemized by end customer and product, in order for your sales team to be able to help out the wholesalers a little.
Thus you convince your wholesalers to provide the sales data, and from now on you receive a file with the data on a regular basis – daily, weekly, monthly, … Unfortunately there are a couple of problems:
Problem no. 1: Different Data Transfer Methods
The data communication methods that your wholesalers use, are quite divers – from email to carrier pigeon almost everything is in place. Nevertheless it has to be assured that every file is imported into the system once, and indeed exactly once. Not less and not more. Accordingly, acknowledgments have to be sent back and forth. Since luckily you are acquainted with Systrion and know that they occupy themselves with → Data Transfer, we consider this problem solved.
Problem no. 2: Different Formats
Every wholesaler sends the data in a different format. Once it is Excel, once a text file, once you receive the proprietary format of a big software company with three letters. The sequence and the specific formats of the individual records will of course also differ.
With this problem, too, the experience from the area of → Data Transfer can help. So called ETL tools (extraction, transformation, loading) help extract the data from individual formats, transform it into the target format and then load it into the target system or database, respectively. So we can solve that one as well.
Problem no. 3: Deviating Codes or Metrics
Your wholesalers send you (as stated above, after quite some convincing) their sales data – but don’t you believe that their codes and metrics match yours. Every wholesaler has his own system, so that your product number 4711 may be called 0815 by wholesaler A and 4716 by wholesaler B. Also it is very likely that the wholesalers don’t use “piece” like you may do, but carton, pallet or full truck load. Your only possibility to match the data seems to be the manual comparison of further information taken from product and customer master data, like product text or address data, and the subsequent conversion of the quantities.
Before the employees assigned to this rewarding task hurl themselves from the roof top, they should know this: Systrion has developed fuzzit®, a self-learning system based on so called Fuzzy Logic, which solves such problems almost automatically.
The system identifies similar sets of data almost in the same way a person would do it, and suggests pairs with probabilities: “Thomas Smith, living in ABC-Street 137” is with a probability of 98% the same as “Tohmas Smth, living in ACB Street 173”.
Using standard logic, as a computer would normally do, these records would be seen as different. Using fuzzy logic, however, small typing mistakes and transposed digits do not become absolute differentiating criteria. The person in charge can then confirm or reject the suggested match. Records that stay the same for every data transmission can be marked for automatic matching.
Proven in all of Europe
These building blocks for data transfer, standardization and matching from Systrion are used by a branded goods company (Fortune 500) in all of Europe, in order to supplement their own, direct sales data as efficiently as possible with the indirect sales data coming from their wholesalers.
Correlation of Market Research Data to Customer Master Data
A similar challenge like the one under Sales Data from Wholesalers is the correlation of purchased market research data to internal customer master data.
A short explanation: Market research companies like Nielsen, Trade Dimensions and the like provide analyses about market participants. Let us assume that, as a branded goods company, you supply the supermarkets of a certain city and would like to receive detailed information about these supermarkets, e.g. with regard to their surface distribution related to food and non-food articles. You can then purchase this information from the above mentioned companies.
Let us further assume that you would like to import this information into your own ERP or CRM system, in order to combine it with your customer master data. Your aim is that on opening a customer master record, your sales force not only sees the address of that customer, but also the additional information that is relevant for sales.
The challenges that you run into upon purchase of this external information, are similar to those in our first case.
To start with, you have to get the data in a readable format into your own system. You could have received the data via email, FTP or on a CD-ROM. You have to extract it, transform it into the target format, and load it into the new system. After that, in this case also, you will have to match external and internal records based on address information, since the external records normally will not be delivered with your internal customer numbers.
Comparison Customer Master and Sanction Lists
This is, in a manner of speaking, a negative correlation. Sanction lists are provided by the governments of different nations, for example the United States of America or Germany. They are lists with companies or individuals that are known or suspected to be close to terrorist organizations or states. Every company is obligated to avoid doing business with the companies, institutions or individuals named on these lists. In case they disregard this obligation, either on purpose or inadvertently, they run the risk of appearing on those lists themselves. They would thus quickly develop a sales problem, since at least government agencies and public administrations will definitely not accept any company on those lists as supplier.
The challenges in matching the own customer master with the sanctions lists are similar to those already described. To begin with, there is the quality of the data in these lists – not always is the content in the fields what the header pretends. Sometimes for example the whole name is found in a field that should only contain the surname.
Then there are several lists, depending on the country you come from. It is thus advisable first to consolidate the lists, before you compare them to your customer master. The already mentioned challenges with regard to different data sources, formats etc. remain of course valid.
Last not least there will be again a matching of names and addresses from the sanctions lists with those from the internal customer master, as well as a reporting of the results. Here as well, a manual, Excel-based process turns out to be rather tedious, especially since it should be repeated on a regular basis – the lists are continuously updated, after all.
Semi-automatic Process with the Aid of Tools from Systrion
With Systrion’s tools for data transfer, cleansing and matching you make the regular examination of your customer master an easy exercise. The current sanctions lists will be provided to you in a cleaned and consolidated form, the correlation with your customer master will be carried out with the help of fuzzit® in shortest time, and you will get a report with the potentially matching records.