SAP Using a Data Warehouse BI

The reporting, analysis, and interpretation of business data is of central importance to a company in guaranteeing its competitive edge, optimizing processes, and enabling it to react quickly and in line with the market.

Company data is usually spread across several applications that are used for entering data. Analyzing this data is not only difficult because it is spread across several systems but because the data is saved in a form that is optimized for processing, not analysis. Data analysis represents additional system load which affects operative data processing. Furthermore, the data has come from heterogeneous applications and is therefore only available in heterogeneous formats which must first be standardized. The applications also only save historic data to a limited extent. This historic data can be important in analysis.

Therefore separate systems are required for storing data and supporting data analysis requirements. This type of system is called a data warehouse.

A data warehouse serves to integrate data from heterogeneous sources, transform, consolidate, clean up and store this data, and stage it efficiently for analysis and interpretation purposes.

SAP Architecture of a Data Warehouse BI

There are many different definitions of a data warehouse. However, they all favor a layer-based architecture.

Data warehousing has developed into an advanced and complex technology. For some time it was assumed that it was sufficient to store data in a star schema optimized for reporting. However this does not adequately meet the needs for consistency and flexibility in the long run. Therefore data warehouses are now structured using a layer architecture. The different layers contain data in differing levels of granularity. We differentiate between the following layers:

● Persistent staging area

● Data warehouse

● Architected data marts

● Operational data store

Persistent Staging Area

After it is extracted from source systems, data is transferred to the entry layer of the data warehouse, the persistent staging area (PSA). In this layer, data is stored in the same form as in the source system. The way in which data is transferred from here to the next layer incorporates quality-assuring measures and the transformations and clean up required for a uniform, integrated view of the data.

Data warehouse

The result of the first transformations and clean up is saved in the next layer, the data warehouse. This data warehouse layer offers integrated, granular, historic, stable data that has not yet been modified for a concrete usage and can therefore be seen as neutral. It acts as the basis for building consistent reporting structures and allows you to react to new requirements with flexibility.

Architected Data Marts

The data warehouse layer provides the most multidimensional analysis structures. These are also called architected data marts. This layer satisfies data analysis requirements. Data marts are not necessarily to be equated with the terms summarized or aggregated; here too you find highly granular structures but they are focused on data analysis requirements alone, unlike the granular data in the data warehouse layer which is application neutral so as to ensure reusability.

The term “architected“ refers to data marts that are not isolated applications but are based on a universally consistent data model. This means that master data can be reused in the form of Shared or Conformed Dimensions.

Operational Data Store

As well as strategic data analysis, a data warehouse also supports operative data analysis by means of the operational data store. Data can be updated to an operational data store on a continual basis or at short intervals and be read for operative analysis. You can also forward the data from the operational data store layer to the data warehouse layer at set times. This means that the data is stored in different levels of granularity: while the operational data store layer contains all the changes to the data, only the days-end status, for example, is stored in the data warehouse layer.

The layer architecture of the data warehouse is largely conceptual. In reality the boundaries between these layers are often fluid; individual data memory can play a role in two different layers. The technical implementation is always specific to the organization.

SAP Enterprise Data Warehouse EDW BI

The type of information that a data warehouse should deliver is largely determined by individual business needs. In practice this often results in a number of isolated applications which are referred to as

SAP Enterprise Data Warehouse EDW BI

The type of information that a data warehouse should deliver is largely determined by individual business needs. In practice this often results in a number of isolated applications which are referred to as

SAP Building and Running a Data Warehouse BI

The structure and running of a data warehouse in general, and an enterprise data warehouse in particular, is highly complex and cannot be tackled without the support of adequate tools. Business Intelligence in SAP NetWeaver offers an integrated solution that represents the entire data warehouse process from extraction, to the data warehouse architecture, to analysis and reporting.

Data Warehousing as an Element of Business Intelligence in SAP NetWeaver Provides:

● Data staging:

○ Extraction, transformation, loading (ETL) of data: All data sources can be accessed by means of extraction in the background (via JDBC, file, XMLA, ODBO, ..). Extractors are delivered for SAP applications or can be generated. The standard applications of other providers can be accessed by integrating the ETL tools of non-SAP providers.

○ Real-time data warehousing: Event-near availability of data in the operational data store can be realized using real-time data acquisition technology.

○ Remote data access: Data can be accessed without being saved in the BI system using VirtualProviders (see below).

● Modeling a layer architecture: InfoCubes support the modeling of star schemas (with one large fact table in the center and several surrounding dimension tables) in the architected data mart layer. VirtualProviders allow you to access source data directly. InfoCubes can be combined in virtual star schemas (MultiProvider) using Shared or Conformed Dimensions (master data tables).

The persistent staging area, data warehouse layer and operational data store are built from flat stores, the DataStore objects.

InfoObjects (characteristics and key figures) form the basis of the InfoCube or DataStore object description. You ensure vertical consistency by using the same InfoObjects in the various layers and thus avoid the interface problems that can arise if you use heterogeneous tools to build the layers.

● Transformation: Transformation rules serve to clean up and consolidate data.

● Modeling the data flow: Data transfer processes serve to transfer the data to the different stores. You use process chains to schedule the data processing and observe it using a monitor.

● Staging data for analysis: You can define queries based on any InfoProvider using the Business Explorer. BEx queries form the basis of the applications that are available to users in the portal or based on Microsoft Excel.

SAP Data Warehousing: Step by Step BI

Purpose

To build a data warehouse, you have to execute certain process steps.

Process Flow

1. Data modeling

○ Creating InfoObjects: Characteristics

○ Creating InfoObjects: Key Figures

○ Creating DataStore objects

○ And/or creating InfoCubes

○ And/or creating InfoSets

○ And/or creating MultiProviders

○ Or creating VirtualProviders

2. Metadata and Document Management

○ Installing BI Content

○ Creating documents

3. Setting up the source system:

○ Creating SAP source systems

○ And/or creating external systems

○ And/or creating file systems

4. Defining extraction:

○ For SAP source systems: Maintaining DataSources

○ Or for a SOAP-based transfer of data: Creating XML DataSources

○ Or for transferring data with UD Connect: Creating a DataSource for UD Connect

○ Or for transferring data with DB Connect: Creating a DataSource for DB Connect

○ Or for files: Creating DataSources for File Source Systems

○ Or for transferring data from non-SAP systems

○ Creating InfoPackages

5. Defining transformations:

○ Creating transformations

6. Defining data distribution:

○ Using the data mart interface

○ Creating open hub destinations

7. Defining the data flow:

○ Creating data transfer processes

○ Creating process chains

8. Scheduling and monitoring:

○ Checking process chain runs

○ Monitor for extraction processes and data transfer processes

SAP Data Warehousing Workbench BI

Purpose

The Data Warehousing Workbench (DWB) is the central tool for performing tasks in the data warehousing process. It provides data modeling functions as well as functions for controlling, monitoring, and maintaining all the processes in SAP NetWeaver BI that are related to the procurement, retention, and processing of data.

Structure of the Data Warehousing Workbench

The following figure shows the structure of the Data Warehousing Workbench:

Navigation Pane Showing Functional Areas of Data Warehousing Workbench

When you call the Data Warehousing Workbench, a navigation pane appears on the left-hand side of the screen. You open the individual functional areas of the Data Warehousing Workbench by choosing the pushbuttons in the navigation pane. The applications that are available in these areas are displayed in the navigation pane; in the modeling functional area, you see the possible views of the object trees.

Object Trees or Applications in the Individual Functional Areas

If object trees or applications are assigned to a functional area in the navigation pane, you call them by clicking them once in the right-hand side of the screen.

Application Toolbar

In all functional areas, the Data Warehousing Workbench toolbar contains a pushbutton for showing or hiding the navigation pane. It also contains pushbuttons that are relevant in the context of the individual functional areas and applications.

Menu Bar

The functions that you can call from the menu bar of the Data Warehousing Workbench depend on the functional areas.

Status Bar

The system displays information, warnings, and error messages in the status bar.

Features

Functional Areas of the Data Warehousing Workbench

Functional Area	Documentation
Modeling	Modeling
Administration	Administration guide: Enterprise Data Warehousing
Transport Connection	Transporting BI Objects and Copying BI Content
Documents	Documents
BI Content	Transporting BI Objects and Copying BI Content
Translation	Translating Text for BI Objects
BI Metadata Repository	Metadata Repository

SAP Data Warehousing Workbench - Modeling BI

Purpose

In the Modeling functional area of the Data Warehousing Workbench, you can display BI objects and the corresponding dataflow in a structured way in object trees. You can create new objects, call applications and functions for objects and define the dataflow for the objects.

Structure of Data Warehousing Workbench: Modeling

The following graphic illustrates the structure of the Data Warehousing Workbench: Modeling:

The Modeling functional area consists of various screen areas. As well as the menu, title and status bars, the modeling screen contains the following four screen areas:

· Modeling pushbutton bar

· Navigation pane in the left-hand area of the screen

· View of selected object tree in right-hand area of screen and, with open applications, the middle area of the screen

· Open application in the right-hand area of the screen

Basic Navigation Options

From the navigation pane, you can click on an entry in the object tree list to open the view of this object tree.

In the object tree, you can expand the nodes to navigate in the objects. You can jump to the corresponding application, usually the object maintenance display, by double clicking on the object name in the tree. The application is called in the right-hand area of the screen.

The modeling pushbutton bar contains the following pushbuttons and provides the following navigation options:

· Previous object: Jumps to the application that was called before the present application.

The navigation pane and tree display do not change, since these are displayed independently of the forwards/backwards navigation in the applications.

Similarly, the open application is still displayed if you call another tree.

· Next object: Jumps to the application that was called after the present application.

The navigation pane and tree display do not change, since these are displayed independently of the forwards/backwards navigation in the applications.

Similarly, the open application is still displayed if you call another tree.

· Show/hide navigator: Hides the navigation pane and tree display if both are currently displayed.

Shows the navigation pane if the tree display is shown.

Shows the tree display if the navigation pane is shown.

This function is only possible if an application has been called.

· Tree display full screen/half screen: Hides or shows the tree display.

This is only possible if an application has been called.

You can remove the navigation pane and object tree from the display by choosing (hide navigation pane or hide tree).

You can display information on additional navigation functions in the navigation pane and information on the structure and functions of the object tree in the legends () for the object trees.

SAP Data Flow in the Data Warehouse BI

The data flow in the Data Warehouse describes which objects are needed at design time and which objects are needed at runtime to transfer data from a source to BI and cleanse, consolidate and integrate the data so that it can be used for analysis, reporting and possibly for planning. The individual requirements of your company processes are supported by numerous ways to design the data flow. You can use any data sources that transfer the data to BI or access the source data directly, apply simple or complex cleansing and consolidating methods, and define data repositories that correspond to the requirements of your layer architecture.

With SAP NetWeaver 7.0, the concepts and technologies for certain elements in the data flow were changed. The most important components of the new data flow are explained below, whereby mention is also made of the changes in comparison to the past data flow. To distinguish them from the new objects, the objects previously used are appended with 3.x.

Data Flow in SAP NetWeaver 7.0

The following graphic shows the data flow in the Data Warehouse:

In BI, the metadata description of the source data is modeled with DataSources. A DataSource is a set of fields that are used to extract data of a business unit from a source system and transfer it to the entry layer of the BI system or provide it for direct access.

There is a new object concept available for DataSources in BI. In BI, the DataSource is edited or created independently of 3.x objects on a unified user interface. When the DataSource is activated, the system creates a PSA table in the Persistent Staging Area (PSA), the entry layer of BI. In this way the DataSource represents a persistent object within the data flow.

Before data can be processed in BI, it has to be loaded into the PSA using an InfoPackage. In the InfoPackage, you specify the selection parameters for transferring data into the PSA. In the new data flow, InfoPackages are only used to load data into the PSA.

Using the transformation, data is copied from a source format to a target format in BI. The transformation process thus allows you to consolidate, cleanse, and integrate data. In the data flow, the transformation replaces the update and transfer rules, including transfer structure maintenance. In the transformation, the fields of a DataSource are also assigned to the InfoObjects of the BI system.

InfoObjects are the smallest units of BI. You map the information in a structured form that is required for constructing InfoProviders.

InfoProviders are persistent data repositories that are used in the layer architecture of the Data Warehouse or in views on data. They can provide the data for analysis, reporting and planning.

Using an InfoSource, which is optional in the new data flow, you can connect multiple sequential transformations. You therefore only require an InfoSource for complex transformations (multistep procedures).

You use the data transfer process (DTP) to transfer the data within BI from one persistent object to another object, in accordance with certain transformations and filters. Possible sources for the data transfer include DataSources and InfoProviders; possible targets include InfoProviders and open hub destinations. To distribute data within BI and in downstream systems, the DTP replaces the InfoPackage, the Data Mart Interface (export DataSources) and the InfoSpoke.

You can also distribute data to other systems using an open hub destination.

In BI, process chains are used to schedule the processes associated with the data flow, including InfoPackages and data transfer processes.

Uses and Advantages of the Data Flow with SAP NetWeaver 7.0

Use of the new DataSource permits real-time data acquisition as well as direct access to source systems of type File and DB Connect.

The data transfer process (DTP) makes the transfer processes in the data warehousing layers more transparent. The performance of the transfer processes increases when you optimize parallelization. With the DTP, delta processes can be separated for different targets and filtering options can be used for the persistent objects on different levels. Error handling can also be defined for DataStore objects with the DTP. The ability to sort out incorrect records in an error stack and to write the data to a buffer after the processing steps of the DTP simplifies error handling. When you use a DTP, you can also directly access each DataSource in the SAP source system that supports the corresponding mode in the metadata (also master data and text DataSources).

The use of transformations simplifies the maintenance of rules for cleansing and consolidating data. Instead of two rules (transfer rules and update rules), as in the past, only the transformation rules are still needed. You edit the transformation rule on an intuitive graphic user interface. InfoSources are no longer mandatory; they are optional and are only required for certain functions. Transformations also provide additional functions such as quantity conversion and the option to create an end routine or expert routine.

Constraints

Hierarchy DataSources, DataSources with the transfer method IDoc as well as DataSources for BAPI source systems cannot be created in the new data flow. They also cannot be migrated. However, DataSources 3.x can be displayed with the interfaces of the new DataSource concept and be used in the new data flow to a limited extent. More information: Using Emulated 3.x DataSources.

Migration

More information about how to migrate an existing data flow with 3.x objects can be found under Migrating Existing Data Flows.

Data load in SAP BW Data Uploading

What is the strategy to load for example 500,000 entries in BW (material master, transactional data)?

How to separate this entries in small packages and transfer it to BW in automatic?

Is there some strategy for that?

Is there some configuration for that?

See OSS note 411464 (example concerning Info Structures from purchasing documents) to create smaller jobs in order to integrate a large amount of data.

For example, if you wish to split your 500,000 entries in five intervals:

- Create 5 variants in RMCENEAU for each interval
- Create 5 jobs (SM36) that execute RMCENEAU for each variant
- Schedule your jobs
- You can then see the result in RSA3

Loading Data From a Data Target

Can you please guide me for carrying out his activity with some important steps?

I am having few request with the without data mart status. How can I use only them & create a export datasource?

Can you please tell me how my data mechanism will work after the loading?

Follow these steps:

1. Select Source data target( in u r case X) , in the context menu click on Create Export Datasources.
DataSource ( InfoSource) with name 8(name of datatarget) will be generated.

2. In Modelling menu click on Source Systems, Select the logical Source System of your BW server, in the context menu click on Replicate DataSource.

3. In the DataModelling click on Infosources and search for infosource 8(name of datatarget). If not found in the search refresh it. Still not find then from DataModelling click on Infosources, in right side window again select Infosources, in the context menu click on insert Lost Nodes.
Now search you will definately found.

4. No goto Receiving DataTargets ( in your case Y1,Y2,Y3) create update rules.
In the next screen select Infocube radio button and enter name of Source Datatarget (in u r case X). click Next screen Button ( Shift F7), here select Addition radio button, then select Source keyfield radio button and map the keyfields form Source cube to target cube.

5. In the DataModelling click on Infosources select infoSource which u replicated earlier and create infopackage to load data..

Difference in number of data records

I have uploaded data from R/3 to BW (Controlling Datasources).

The problem is that when i use the extractor checker (rsa3) in R/3 for a
specific datasource (0CO_OM_OPA_1) it shows me that there are 1600 records.

When i load this datasource in BW it shows me that there are 400.000
records. I'm uploading data to "PSA only".

Any ideas why this is happening ?

Thanks

-----Reply Message-----
Subject: RE: Difference in number of data records

Check the 'data recs/call' and 'number of extract calls' parameters in
RSA3. Most likely the actual extract is only making one call with a larger
data rec/call number. The extraction process will collect data records
with the same key so less data has to be transferred to the BW. When you
run RSA3 you are probably getting similar records (that would normally
collect) in different data packets thereby creating more records. Try
running RSA3 with a much higher (2000) recs/call for several calls.

BI Data Transfer Using the Data Mart Interface in BW

The data transfer is the same as the data transfer in an SAP system. The system reads the data, taking into account the specified dimension-specific selections, from the fact tables of the delivering BI system.

Delta Process

Using the data mart interface, you can transfer data by full upload as well as by delta requests.

A distinction is made between InfoCubes and DataStore objects.

● The InfoCube that is used as an export DataSource is first initialized, meaning that the current status is transferred into the target BI system. When the next upload is performed, only those requests are transferred that have come in since initialization. Different target systems can also be filled like this.

Only those requests are transferred that have been rolled up successfully in the aggregates. If no aggregates are used, only those requests are transferred that are set to Qualitative OK in the InfoCube administration.

● For DataStore objects, the requests in the change log of the DataStore object are used as the basis for determining the delta. Only the change log requests that have arisen from reactivating the DataStore object data are transferred.

Restriction:

You can only make one selection for each target system for the delta.

You first make a selection using cost center 1 and load deltas for this selection. Later on, you also decide to load a delta for cost center 2 in parallel to thecost center 1 delta. The delta can only be fully requested for both cost centers, meaning that it is then impossible to separately execute deltas for the different selections.

BI Data Extraction from SAP Source Systems in BW

Purpose

Extractors are part of the data retrieval mechanisms in the SAP source system. An extractor can fill the extraction structure of a DataSource with the data from SAP source system datasets.

Replication makes the DataSource and its relevant properties known in BI.

For the data transfer to the input layer of BI, the Persistent Staging Area (PSA), define the load process with an InfoPackage in the scheduler. The data load process is triggered by a request IDoc to the source system when the InfoPackage is executed. We recommend that you use process chains for execution.

Process Flow

There are application-specific extractors, each of which is hard-coded for the DataSource that was delivered with BI Content, and which fill the extraction structure of this DataSource.

In addition, there are generic extractors with which you can extract more data from the SAP source system and transfer it into BI. Only when you call up the generic extractor by naming the DataSource does it know which data is to be extracted, and from which tables it should read it from and in which structure. This is how it fills different extraction structures and DataSources.

You can run generic data extraction in the SAP source system application areas such as LIS, CO-PA, FI-SL and HR. This is how LIS, for example, uses a generic extractor to read info structures. DataSources are generated on the basis of these (individually) defined info structures. We speak of customer-defined DataSources with generic data extraction from applications.

Regardless of application, you can generically extract master data attributes or texts, or transaction data from all transparent tables, database views or SAP query functional areas or using the function module. You can generate user-specific DataSources here. In this case, we speak of generic DataSources.

The DataSource data for these types are read generically and transferred into BI. This is how generic extractors allow the extraction of data that cannot be made available within the framework of BI Content.

PlugIn for SAP Systems

BI-specific source system functions, extractors and DataSources are delivered for specific SAP systems by plug-ins.

Communication between the SAP source system and BI is only possible if the appropriate plug-in is installed in the source system.

Data Distribution in BW BI

Use

As well as data staging and data processing, the data warehousing capabilities in BI also offer processes for distributing data.

You can distribute data within the BI system or load it for other applications in other systems. In the latter case, the BI system is the source, or hub, of the data transfer.

Features

The data transfer process is available for distributing data both within the BI system and to other systems. For more information, see Data Transfer Process.

The data mart interface is available for distributing data to other BI systems. For more information, see Data Mart Interface.

The open hub service is still available for distributing data to other SAP systems or non-SAP systems, however, we recommend that you use the new open hub destination concept. The open hub destination can now be used as an independent object in the data transfer process. For more information, see Open Hub Destination.

Data Transfer Process in BW BI

Definition

Object that determines how data is transferred between two persistent objects.

Use

You use the data transfer process (DTP) to transfer data within BI from one persistent object to another object, in accordance with certain transformations and filters. In this respect, it replaces the data mart interface and the InfoPackage. As of SAP NetWeaver 7.0, the InfoPackage only loads data to the entry layer of BI (PSA).

The data transfer process makes the transfer processes in the data warehousing layer more transparent. Optimized parallel processing improves the performance of the transfer process (the data transfer process determines the processing mode). You can use the data transfer process to separate delta processes for different targets and you can use filter options between the persistent objects on various levels. For example, you can use filters between a DataStore object and an InfoCube.

Data transfer processes are used for standard data transfer, for real-time data acquisition, and for accessing data directly.

Integration

The following graphic illustrates how the data transfer process is positioned in relation to the objects in the dataflow of BI and the other objects in BI process control:

The InfoPackage controls the transfer of data from the source to the entry layer of BI. The data transfer process controls the distribution of data within BI. The graphic illustrates an example of a data update from the DataSource to an InfoProvider. The data can be updated from an InfoProvider to another InfoProvider using a data transfer process. The data transfer process can also be used to control data distribution from a BI system into any target outside of the BI system. For this purpose, a data transfer process with an open hub destination is used as the target.

Features

You use a process chain to define a data transfer process. Alternatively, you can define a data transfer process for an InfoProvider in an object tree in the Data Warehousing Workbench. We recommend that you use process chains. In this case, the data transfer process is executed when it is triggered by an event in the predecessor process in the process chain. Alternatively, in process chain maintenance, you can execute a data transfer process in the background. A debug mode is also available.

The request is an instance that is generated at the runtime of the data transfer process. The request is processed in the steps that have been defined for the data transfer process (extraction, transformation, filter and so on). The monitor for the data transfer process request shows the header information, request status, and the status and messages for the individual processing steps.

With a data transfer process, you can transfer data either in full extraction mode or in delta mode. In full mode, the entire dataset of the source is transferred to the target; in delta mode, only the data that was posted to the source since the last data transfer is transferred. The data transfer process controls delta handling and therefore allows you to fill several targets with different deltas from one source. With a data transfer process, you do not need to explicitly initialize the delta method as you do when copying data with an InfoPackage.

The data transfer process supports you in handling data records with errors. When you define the data transfer process, you can determine how the system responds to errors. At runtime, the incorrect data records are sorted and written to an error stack (request-based database table). A special error DTP further updates the data records from the error stack into the target. It is easier to restart failed load processes if the data is written to a temporary storage after each processing step. It also allows you to find records that have errors. In the monitor for the data transfer process request or in the temporary storage for the processing step (if filled), you can display the data records in the error stack. In data transfer process maintenance, you determine the processing steps after which you want to store data temporarily.

Creating Data Transfer Processes in BW BI

Use

You use the data transfer process (DTP) to transfer data from source objects to target objects within BI. You can also use the data transfer process to access InfoProvider data directly.

Prerequisites

You have used transformations to define the data flow between the source and target object.

Procedure

Creating Data Transfer Processes Using Process Chains

You are in the plan view of the process chain that you want to use for the data transfer process.

The process type Data Transfer Process is available in the Loading Process and Postprocessing process category.

...

1. Use drag and drop or double-click to include the process in the process chain.

2. To create a data transfer process as a new process variant, enter a technical name and choose Create.

The dialog box for creating a data transfer process appears.

3. Select

Creating Data Transfer Processes for Real-Time Data Acquisition in BW BI

Use

You use the data transfer process (DTP) for real-time data acquisition to transfer data to the DataStore object from the PSA. In the DataStore object, the data is available for use in reporting.

Prerequisites

You have used transformations to define the data flow between the DataSource and the DataStore object.

The selections for the data transfer process do not overlap with selections in other data transfer processes.

Procedure

The starting point when creating a data transfer process is the DataStore object into which you want to transfer data. In the Data Warehousing Workbench, an object tree is displayed and you select the DataStore object.

...

1. In the context menu, choose Create Data Transfer Process.

The dialog box for creating a data transfer process appears.

2. Select DTP for Real-Time Data Acquisition as the DTP Type.

3. As the source object, select the DataSource from which you want to transfer data to the DataStore object.

The input help for the source object shows the selection of DataSources that already exist in the data flow for the DataStore object. An additional List pushbutton is available. This allows you to select a DataSource from the complete list of BI DataSources.

4. Choose Continue.

The data transfer process maintenance screen appears.

The header data for the data transfer process shows the description, ID, version, and status of the data transfer process, along with the delta status.

5. On the Extraction tab page, specify the parameters:

a. Delta is chosen as the extraction mode for real-time data acquisition.

b. If necessary, determine filter criteria for the delta transfer. To do this, choose Filter.

This means that you can use multiple data transfer processes with disjunctive selection conditions to efficiently transfer small sets of data from a source into one or more targets, instead of transferring large volumes of data. You can specify individual selections, multiple selections, intervals, selections based on variables, or routines. To change the list of InfoObjects that can be selected, choose Change Selection.

The icon next to pushbutton Filter indicates that predefined selections exist for the data transfer process. The quick info text for this icon displays the selections as a character string.

c. Semantic grouping is not used for real-time data acquisition. The data is read from the source in packages.

6. On the Update tab page, specify the parameters:

Make the settings for error handling. You define:

○ How you want to update valid records when errors occur.

○ How many errors can occur before the load process terminates.

Note that this setting only has an impact while repairing a DTP request (repair) and during the conversion of the DTP to standard DTP (for example, to correct an error during extraction).

For real-time data acquisition, specify in the InfoPackage the maximum number of failed attempts that is permitted when the daemon is accessing data before the data transfer terminates with an error.

For more information about error handling settings, see Handling Data Records with Errors.

7. On the Execute tab page, determine the parameters:

On this tab page, the process flow of the program for the data transfer process is displayed in a tree structure.

a. Specify the status that you want the system to adopt for the request if warnings are to be displayed in the log.

b. Specify how you want the system to define the overall status of the request.

8. Check, save, and activate the data transfer process.

9. With Assign Daemon you go to the monitor for real-time data acquisition if there is already an InfoPackage for RDA for the DataSource that is the source for this DTP.

If there is not yet an InfoPackage for RDA for the DataSource that is the source for this DTP, the system informs you that you must first create an InfoPackage for RDA before you can assign the DTP.

Alternatively you can go to the monitor for real-time data acquisition from the context menu entry Assign RDA Daemon of the data transfer process if you are in the Data Warehousing Workbench.

Result

The data transfer process is assigned to the DataSource.

If the DataSource is already assigned to a daemon, the data transfer process appears in the monitor for real-time data acquisition under this daemon and the DataSource. It is now available for data processing by the daemon.

If the DataSource has not yet been assigned to a daemon, the data transfer process appears in the monitor for real-time data acquisition under the DataSource in the area Unassigned Objects. The data transfer process, the corresponding DataSource, the InfoPackage and possibly further associated data transfer processes are assigned to the specified daemon in the context menu of the data transfer process with Assign Daemon..

Creating Data Transfer Processes for Direct Access in BW BI

Use

You use a data transfer process for direct access to access the data in an InfoProvider directly.

Prerequisites

You have used transformations to define the data flow between the source and target object.

Procedure

The starting point when creating a data transfer process is the target into which you want to transfer data. In the Data Warehousing Workbench, an object tree is displayed and you have highlighted the target object, a VirtualProvider.

...

1. In the context menu, choose Create Data Transfer Process.

The dialog box for creating a data transfer process appears.

DTP for Direct Access is displayed as the type of the data transfer process.

2. Select the type of source object.

Supported object types are DataSources, InfoCubes, DataStore objects and InfoObjects (texts and attributes, if they are released as InfoProviders).

3. Select the object from which you want to transfer data into the target.

When you select the source object, input help is available. Input help shows you the selection of objects that already exist in the data flow for target object. If only one object exists in the data flow, this is selected by default.

An additional List pushbutton is available. This allows you to select a source object from the complete list of objects that exist for this object type.

4. Choose Continue.

The data transfer process maintenance screen appears.

The header data for the data transfer process shows the description, ID, version, and status of the data transfer process, along with the delta status.

On the Extraction tab page, the system displays information about the adapter, the format of the data and additional source-specific settings.

On the Update tab page, the system displays information about the target.

On the Execute tab page, the system displays the processing mode for direct access and the process flow of the program for the data transfer process.

You do not need to make any settings in the data transfer process.

5. Check the data transfer process, save and activate it.

Choose Goto ® Overview of DTP to display information about the source and target objects, the transformations, and the last changes to the data transfer process.

Result

You can use the data transfer process to access data directly.

Handling of Data Records with Errors in BW BI

Use

On the Update tab page in the data transfer process (DTP), the error handling settings allow you to control how the system responds if errors occur in the data records when data is transferred from a DTP source to a DTP target.

These settings were previously made in the InfoPackage. When using data transfer processes, InfoPackages only write to the PSA. Therefore, error handling settings are no longer made in the InfoPackage but in the data transfer process.

Features

Settings for Error Handling

For a data transfer process (DTP), you can specify how you want the system to respond when data records contain errors. If you activate error handling, the records with errors are written to a request-based database table (PSA table). This is the error stack. You can use a special data transfer process, the error DTP, to update the records to the target.

Temporary storage is available after each processing step of the DTP request. This allows you to determine the processing step in which the error occurred.

Checks for Incorrect Data Records

The following table provides an overview of where checks for incorrect data records can be run:

Where Is Check Run?	Examples of Incorrect Data Records
In the transformation	Field contains invalid characters or lowercase characters Error during conversion Error during currency translation A routine returns a return code <> 0 Characteristic value is not found for master data Error reading master data Customer-specific formula results in error
When data is updated to the master data table or text table	Invalid characters in keys or navigation attributes If no SID exists for the value of the navigation attribute If no language field is defined for texts Invalid "from" and "to" dates Duplicate data records with relation to keys Overlapping and invalid time intervals
When data is updated to the InfoCube	If no SID exists for the characteristic value
When checking referential integrity of an InfoObject against master data tables or DataStore objects	If no SID exists for the characteristic value

"Repairing" with Error DTP

You create an error DTP for an active data transfer process on the Update tab page. You run it directly in the background or include it in a process chain so that you can schedule it regularly in the context of your process chain. The error DTP uses the full update mode to extract data from the error stack (in this case, the source of the DTP) and transfer it to the target that you have already defined in the data transfer process.

Activities

...

1. On the Extraction tab page under Semantic Groups, define the key fields for the error stack.

This setting is only relevant if you are transferring data to DataStore objects with data fields that are overwritten. If errors occur, all subsequent data records with the same key are written to the error stack along with the incorrect data record; they are not updated to the target. This guarantees the serialization of the data records, and consistent data processing. The serialization of the data records and thus the explicit definition of key fields for the error stack is not relevant for targets that are not updated by overwriting.

The default value and possible entries for the key fields of the error stack for DataStore objects that overwrite are shown below:

Default Value/Possible Entries	Which Fields of the Source?
Default value for the key fields of the error stack (fields selected by the system)	All fields of the source that are uniquely assigned to a key field of the target. That is, all the fields of the source that are directly assigned to a key field of the DataStore object in the transformation.
Fields that can also be selected as key fields for the error stack

Error Stack in BW BI

Definition

A request-based table (PSA table) into which erroneous data records from a data transfer process are written. The error stack is based on the data source, that is, records from the source are written to the error stack.

Use

At runtime, erroneous data records are written to an error stack if the error handling for the data transfer process is activated. You use the error stack to update the data to the target destination once the error is resolved.

Integration

In the monitor for the data transfer process, you can navigate to the PSA maintenance by choosing Error Stack in the toolbar, and display and edit erroneous records in the error stack.

With an error DTP, you can update the data records to the target manually or by means of a process chain. Once the data records have been successfully updated, they are deleted from the error stack. If there are any erroneous data records, they are written to the error stack again in a new error DTP request.

When a DTP request is deleted, the corresponding data records are also deleted from the error stack.

Examples for Using the Error Stack in BW BI

Consistent Error Handling for Aggregation

Number of Records in Source is Greater than Number of Records in Target

During the transformation, the data records for request 109882 are aggregated to one data record. If, for example, there is no SID for the characteristic value order number 1000, the record is interpreted as erroneous. It is not updated to the target. Those data records that form the aggregated data record are written to the error stack.

Number of Records in Source is Less than Number of Records in Target

During the transformation, the data record for request 109882 is duplicated to multiple data records. If, for example, there is no SID for the characteristic value calendar day 07-03-2005, the record is interpreted as erroneous. The duplicated records are not updated to the target. The data record that formed the duplicate records is written to the error stack. In the error stack, the record is listed as containing an error every time it duplicates data records with errors.

Consistent Error Handling with Respect to Order in Which Data Records are Written to Error Stack

Update to DataStore Object: 1 Request

The Order Number field is the key for the error stack. During the transformation, data record 02 of request 109882 is marked as containing errors. In addition to the erroneous data record, all subsequent data records for the request that contain the same key are written to the error stack. In this example, this is data record 03. This ensures that when error records are updated with the error DTP, the records are serialized correctly and newer data is not inadvertently overwritten by older data. Data record 01 has the same key as the incorrect data record 02 (order number 1000), but is correct and it occurred before the incorrect data record. Data record 01 is therefore copied into the target of the DTP. The order of the data records is not changed.

Updating to DataStore Object: Multiple Requests – Error in First Request

The Order Number field is the key for the error stack. During the transformation, data record 02 of request 109882 is marked as containing errors. In addition to the erroneous data record, all subsequent data records, including the following requests that have the same key, are written to the error stack. In this example, data record 01 for request 109883 is written to the error stack in addition to data record 02 for request 109882.

Updating to DataStore Object: Multiple Requests – Error in Subsequent Request

The Order Number field is the key for the error stack. During the transformation, data record 01 of request 109883 is identified as containing errors. It is written to the error stack. Any data records from the previous request that have the same key were updated successfully to the target.