iWay DataMigrator User's Guide > Designing a Data Flow > Specifying Data Target Options
In this section: |
After the data targets have been added to the flow, there are two types of options that govern the data flow's behavior when copying data into the target(s).
Data flow properties include:
Target-specific properties, include:
Load options specify the way DataMigrator loads data into the targets. If you're moving data to more than one target, they will all use the same load type. The load type is displayed on all target objects and, if changed in one target, the change will be reflected in all the target objects.
Load options vary depending on the type of target. They can include:
If you select this option, you can set a behavior when duplicate records are found from the If the record exists drop-down menu. Key Matching Logic options can be set on a per target basis:
Note: When loading relational data targets, the Reject, Update, and Delete options can adversely affect performance because they determine the existence of a key value on an incoming record before performing the specified action. This is done by issuing a SELECT command against the target table, then waiting for the response. If one of these actions is required, try to limit the volume of records in the incremental change. These actions will perform best if there are unique keys on the table.
If you select Update the existing record or Delete the existing record from the If the record exists drop-down menu, you can also set behavior when the record does not exist using the If the record does not exist drop-down menu:
Note: This option is only available for DB2, MS SQL Server, ORACLE, Informix, and Sybase ASE. For other databases, the number of records in a block will default to 1 and can't be changed. For details about loading relational targets, see How to Set Options for Relational Targets, and Target Properties Window for Relational Targets.
Insert records from memory:
The options available will depend on the RDBMS of your target type.
DB2 targets have the following options:
INSERT - Insert the rows only. This is the default.
REPLACE - Drops the existing destination table, and creates a new table.
Note: this option is not available for z/OS.
Informix targets have the following options:
Table locking prevents other users from changing data in a table that is being loaded. Locking is necessary for maintaining data integrity while concurrent users access database information.
Recognizes hexadecimal data in character fields.
Checks the syntax, but not the data being loaded.
Numbers of rows to ignore before starting to load data.
The row interval to commit or write transactions to the database.
The maximum number of errors allowed before loading stops. The default is the target RDBMS's default.
Nucleus targets have the following options:
Starts the server in exclusive mode. No other database sessions may be established.
The server must already be started in multi-user mode and listening on the default port 1285.
Creates a new a file
Appends to existing log.ndl file.
Issues a commit,
Issues a rollback.
Number of bytes to skip before loading data.
Specifies the character to represent null.
Number of rows to skip before loading data.
Number of rows to process before stopping.
Disconnects connections.
Skips rows with unprintable characters
NDL server password. Required if Single User (ndls) mode and a server password actually exists.
Oracle targets have the following options:
A direct path load builds blocks of data in memory and saves these blocks directly into the extents allocated for the table being loaded. A direct path load uses the field specifications to build whole Oracle blocks of data, and write the blocks directly to Oracle data files. Direct path load is much faster than conventional load, but entails some restrictions.
Size (in bytes) of the bind array. When the array is full, the array is loaded into the target.
Size (in bytes) of the read buffer.
If yes, the transaction will not generate redo entries.
APPEND - If data already exists in the table, new rows are appended to it. If data does not already exist, the new rows are simply loaded. This is the default.
REPLACE - Drops the existing destination table, and creates a new table.
INSERT - Insert the rows only.
Specifies the number of logical records from the beginning of the file that should not be loaded. By default, no records are skipped.
Specifies the maximum number of logical records to load. By default all records are loaded. No error occurs if fewer than the maximum number of records are found.
Specifies the maximum number of insert errors to allow. If the number of errors exceeds this value, the load is terminated. The default is 50.
The row interval to commit or write transactions to the database.
Version of the SQL*Loader utility.
Red brick targets have the following options:
Row number to begin copying data to the target.
Row number to end copying data to the target.
The combination of language and location.
The maximum number of discarded records allowed before loading stops. The default is the target RDBMS's default.
File where duplicate records are discarded. The records will be stored for possible reloading.
File where discarded records based on referential integrity will be stored for possible reloading.
INSERT - Insert the rows only. If the table is not empty, the load operation ends. This is the default.
APPEND - Used to insert additional rows of data into an existing table. Each new row must have a primary-key value that does not already exist in the table. Otherwise, the record is discarded.
REPLACE - Replaces the entire contents of a table.
MODIFY - Used to insert additional rows or to update existing rows in a table. If the input row has the same primary-key value as an existing row, the new row replaces the existing row. Otherwise, it is added as a new row.
UPDATE - Updates existing rows in an existing table. Each new row must have a primary-key value that is already present in the table. Otherwise, the record is discarded.
MODIFY AGGREGATE - If the primary key of the input row matches an existing row in the table, the existing row is updated as defined for the specified aggregate operator. If the primary key of the input row does not match an existing row in the table, the row is inserted.
UPDATE AGGREGATE - If the primary key of the input row does not match the primary key of a row already in the table, the input row is discarded. If it does match an existing row, the existing row is updated as defined for the specified aggregate operator.
OFF - Indexes are updated when each input row is inserted into the data file, which provides better performance when the data being loaded contains many duplicate rows. This is the default.
ON - Overrides the global optimize mode setting in the rbw.config file.
SQL Server targets have the following options:
Version of the file used to store the format information for each field.
The maximum number of errors allowed before loading stops. The default is the target RDBMS's default.
Row number to begin copying data to the target.
Row number to end copying data to the target.
The packet size in bytes.
The row interval to commit or write transactions to the database.
Sybase targets have the following options:
Version of the file used to store the format information for each field.
The maximum number of errors allowed before loading stops. The default is the target RDBMS's default.
Row number to begin copying data to the target.
Row number to end copying data to the target.
The row interval to commit or write transactions to the database.
Sybase IQ targets have the following options:
Is the delimiter character(s) used between fields. It is used by the intermediate flat file used to load the database.
The delimiter can be up to four characters and can be specified as:
TAB - a tab character. This is the default.
a - A character string, for example ~.
0x nn - a hex code, for example, 0x44 (a comma), or 0x0D0A (a return and a linefeed). The hex code uses ASCII for Windows or UNIX systems and EBCDIC for IBM Mainframes.
Note: For a Sybase Adaptive Server IQ target, the load will fail if the database returns any errors, such as a unique index constraint violation.
Teradata targets have the following options:
The maximum number of rejected records allowed before loading stops. The default is the target RDBMS's default.
The Teradata Director Program Id.
Account Id used to access the database.
Row number to begin copying data to the target.
Row number to end copying data to the target.
Name of the work table.
This table provides information about all errors that occur during the acquisition phase of your Update operator job, as well as some errors that occur during the application phase if the Teradata RDBMS cannot build a valid primary index.
This table provides information about uniqueness violations, field overflow on columns other than primary index fields, and constraint errors.
INSERT - Inserts the rows only. This is the default.
UPSERT – Does inserts for missing update rows.
Note: This option is available for adapters configured with Call-Level Interface (CLI).
The Target Properties window opens to the Main Options tab.
No changes does not delete the records already in a data target.
Delete all rows from table deletes all rows and creates a database log.
Truncate table deletes all rows from the table but does not generate a database log. This is faster option that Delete all rows from table.
Note: Truncate table is not supported by DB2.
Key Matching Logic allows you to set a behavior when loading records.
Insert records from memory speeds the loading of the data target by inserting a block of rows at once. You can set the row interval to commit or write transactions and the number of records to load in a block. This option:
Bulk load utility via a disk file uses database bulk loaders instead of iWay to insert data into a target. DataMigrator automates bulk loading for Ingres, Informix, Microsoft SQL Server, IBM DB2, NCR Teradata, Nucleus, Oracle, Sybase Adaptive Server Enterprise, and Sybase Adaptive Server IQ. You can set format version, maximum number of errors, first and last row to copy, packet size and row interval to commit or write transactions.
Slowly Changing Dimensions enables you to load a data target with column values that change over time. There are two column types. Type I simply overwrites the value. Type II allows you track activity before and after the change. Type II changes are handled either via an activation flag or end date/begin date. When you update a Type II column, you don't actually change its value. Instead, you update the activation flag to inactive, or the end date to today's date by default. You then add a new record with the new value and the activation flag on or the begin date set.
This option is only available for existing relational targets.
Enable Change Data Capture enables you to load a data target with only the records that have changed. This feature is an optional, add on component to DataMigrator.
The Target Properties window is available by right-clicking a target object and selecting Target Properties.
This shows the Key Matching Logic Load option.
The Insert records from memory Load option looks like this:
The Bulk load utility via a disk file Load option looks like this for SQL Server. Other relational targets will have different options.
The Slowly Changing Dimensions Load option looks like this:
The Enable Change Data Capture Load option looks like this:
The Main Options tab of Target Properties window for relational targets contains the following target options:
Is the target type.
Is the connection for the data target. For a relational data target, this is a database server. For ODBC, this is a data source.
Is the directory where the synonym is located.
Is the synonym's name.
Is the name of data target table.
Is the number of key columns.
These options are only available for existing targets.
Does not delete the records already in a data target.
Deletes all rows and creates a database log.
Deletes all rows from the table but does not generate a database log. This is faster option that Delete all rows from table.
The Main Options tab also includes the following load options:
Note: Although Load Type appears in every data target, it is set on a per flow basis. Changing the type in one target will reset it for all targets.
Specifies DataMigrator's behavior while loading data.
Key Matching Logic options can be set on a per target basis.
If you select this option, you can set a behavior when duplicate records are found from the If the record exists drop-down menu. Key Matching Logic options are:
Include the record allows the relational database target to handle duplicates.
Note: If you select Include the record, the record is passed directly to the relational database, which determines whether to accept it or not. If inserting a record would result in a duplicate key, the RDBMS will reject it due to a Unique Index constraint violation and return an error. Processing continues even if such errors occur, up to the number of errors specified under Stop processing after __ DBMS errors in the flow's General properties.
Reject the record issues a SELECT command against the target table to see if a record exists with the key values. If there are no key columns, DataMigrator screens all columns. If the record is found, the record is rejected.
Update the existing record updates records if the key value on the input record, or entire record if there are no keys, are found in the table. All non-key values are updated in the target.
Delete the existing record deletes the record if the key on the input record (or entire record if no keys) is found in the table.
If you select Update the existing record or Delete the existing record from the If the record exists drop-down menu, you can also set behavior when the record does not exist using the If the record does not exist drop-down menu:
Include the record includes the key in the data target.
Reject the record does not include the key in the data target.
Note: The Reject, Update, and Delete options can adversely affect performance because they determine the existence of a key value on an incoming record before performing the specified action. This is done by issuing a SELECT command against the target table, then waiting for the response. If one of these actions is required, try to limit the volume of records in the incremental change. These actions will perform best if there are unique keys on the table.
Commit every row(s) specifies the row interval to commit or write transactions to the database.
When you specify this load type you need to specify:
Commit every row(s) specifies the row interval to commit or write transactions to the database.
Number of records to load in a block specifies how many records you want to process at a time.
These options will depend on your target type.
This option is only available for existing relational targets.
Enables you to load a data target with column values that change over time. There are two column types. Type I simply overwrites the value. Type II allows you track activity before and after the change. Type II changes are handled either via an activation flag or end date/begin date. When you update a Type II column, you don't actually change its value. Instead, you update the activation flag to inactive, or the end date to today's date. You then add a new record with the new value and the activation flag on by default or the begin date set.
Commit every row(s) specifies the row interval to commit or write transactions to the database.
Enables you to load a data target with only the records that have changed. This feature is an optional, add on component to DataMigrator. When this option is selected, the Prior to Load option is eliminated.
Commit every row(s) specifies the row interval to commit or write transactions to the database.
The Main Options tab of the Target Properties window opens.
No changes does not delete the records already in a data target.
Drop Table drops and recreates the data target.
Include the record includes the duplicate record in the data target.
Reject the record issues a SELECT command against the target table to see if a record exists with the key values. If there are no key columns, DataMigrator screens all columns. If the record is found, the record is rejected.
Update the existing record updates records if the key value on the input record, or entire record if there are no keys, are found in the table. All non-key values are updated in the target.
Delete the existing record deletes the record if the key on the input record (or entire record if no keys) is found in the table.
Include the record includes the key in the data target.
Reject the record does not include the key in the data target.
Note:
The Target Properties window is available by right-clicking a target object and selecting Target Properties.
The Main Options tab of the Target Properties window for FOCUS/FDS or XFOCUS targets contains the following fields/options:
Is the target type.
Is the directory where the synonym is located.
Is the synonym's name.
Note: Synonym names cannot be longer than eight characters.
Is the application directory for the data file pointed to by the synonym or <physical location> if the file is not written to an application directory.
Note: For USS servers there is an additional option, <MVS Dataset>, which allows you to specify the fully qualified location of an MVS dataset, using the format:
"//'qualif.target01.FOCUS'"
Is the name of the data file pointed to by the synonym.
Is the number of key columns.
Does not drop (delete) the data target.
Drops and recreates the data target.
Since DataMigrator uses Key Matching Logic to load FOCUS/FDS or XFOCUS targets, you can set a behavior when duplicate records are found from the If the record exists drop-down menu. These options can be set on a per target basis. They are:
Include the record includes the duplicate record in the data target.
Reject the record issues a SELECT command against the target table to see if a record exists with the key values. If there are no key columns, DataMigrator screens all columns. If the record is found, the record is rejected.
Update the existing record updates records if the key value on the input record, or entire record if there are no keys, are found in the table. All non-key values are updated in the target.
Delete the existing record deletes the record if the key on the input record (or entire record if no keys) is found in the table.
If you select Update the existing record or Delete the existing record from the If the record exists drop-down menu, you can also set behavior when the record does not exist using the If the record does not exist drop-down menu:
Include the record includes the key in the data target.
Reject the record does not include the key in the data target.
Commit every row(s) specifies the row interval to commit or write transactions to the database.
The Main Options tab of the Target Properties window opens.
No changes does not delete the data target. New records are appended.
Delete File drops and recreates the data target.
Note:
The Target Properties window is available by right-clicking a target object and selecting Target Properties.
The Main Options tab of the Target Properties window for Delimited Flat File Targets contains the following fields/options:
Is the target type.
Note: Delimited Flat File will only appear as a target type if it is configured on the server.
Is the directory where the synonym is located.
Is the synonym's name.
Is the application directory for the data file pointed to by the synonym or <physical location> if the file is not written to an application directory.
Is the name of the data file pointed to by the synonym.
No changes
Delete File
Is the delimiter character(s) used between fields. The delimiter can be up to 30 characters. It can be specified as:
TAB - a tab character. This is the default.
a - A character string, for example ~.
0x nn - a hex code, for example, 0x44 (a comma), or 0x0D0A (a return and a linefeed). The hex code uses ASCII for WINDOWS or UNIX systems and EBCDIC for IBM Mainframes.
Inserts column headings as the first row of data and surrounds the column names with the character specified in the Enclosure field.
This character is used to surround the column headings when Header is set to Yes.
The Target Properties window is available by right-clicking a target object and selecting Target Properties.
The Main Options tab of the Target Properties window for Flat File Targets contains the following fields/options:
Is the target type.
Is the directory where the synonym is located.
Is the synonym's name.
Is the application directory for the data file pointed to by the synonym or <physical location> if the file is not written to an application directory.
Is the name of the data file pointed to by the synonym.
Specifies a code page for the target data.
Does not delete the data target.
Deletes and recreates the data target.
The Target Properties window is available by right-clicking a target object and selecting Target Properties.
The Main Options tab of the Target Properties window for Formatted File Targets contains the following fields/options:
Is the target type.
Is the format of the target data.
Is the directory where the synonym is located.
Is the synonym's name.
Is the application directory for the data file pointed to by the synonym or <physical location> if the file is not written to an application directory.
Note: For USS servers there is an additional option, <MVS Dataset>, which allows you to specify the fully qualified location of an MVS dataset, using the format:
"//'qualif.target01.data'"
Is the name of the data file pointed to by the synonym.
Does not delete the data target.
Deletes and recreates the data target.
Note: For USS servers there is an additional option, <MVS Dataset>, which allows you to specify the fully qualified location of an MVS dataset, using the format:
"//'qualif.target01.data'"
Setting a data flow's record logging options allows you to write particular types of transactions to log files.
Although Record Logging options appear in every data target, they are set on a per flow basis. Changing the options in one target will reset them for all targets.
The Target Properties window opens.
The Record Logging tab of the Target Properties window opens.
Record logging options are optional and are not available when the data target is a formatted file.
The Target Properties window is available by right-clicking a target object and selecting Target Properties.
The following types of transactions can be logged: