Informatica Guide Guru99
Informatica Guide Guru99
Informatica in 1 Day
By Krishna Rungta
Chapter 2: Architecture
1. Informatica Domain
2. PowerCenter Repository
3. Domain Configuration
4. Properties of the domain
5. Powercenter client & Server Connectivity
6. Repository Service
7. Integration Service
8. Sources & Targets
Chapter 5: Mappings
1. Why do you need Mapping?
2. Components of Mapping
3. What is Stage Mapping?
4. How to Create a Mapping
5. Mapping Parameters and Variables
6. How to Create Mapping Parameter
7. How to Create Mapping Variable
Chapter 6: Workflows
1. Open Workflow Manager
2. Workflow Monitor
3. How to open Workflow Monitor
4. Task View
5. Gantt Chart View
Chapter 8: Transformations
1. Transformations & Filter Transformation
2. Source Qualifier Transformation
3. Aggregator Transformation
4. Router Transformation
5. Joiner Transformation
6. Rank Transformation
7. Sequence Generator Transformation
8. Transaction Control Transformation
9. Lookup and Re-usable Transformation
10. Normalizer Transformation
11. Performance Tuning for Transformation
Chapter 1: Introduction
Informatica Powercenter ETL/Data Integration tool is a most widely used tool and in
the common term when we say Informatica, it refers to the Informatica PowerCenter
tool for ETL.
For example, you can connect to an SQL Server Database and Oracle Database both
and can integrate the data into a third system.
The latest version of Informatica PowerCenter available is 9.6.0. The different editions
for the PowerCenter are
Standard edition
Advanced edition
Premium edition
The popular clients using Informatica Powercenter as a data integration tool are U.S
Air Force, Allianz, Fannie Mae, ING, Samsung, etc. The popular tools available in the
market in competition to Informatica are IBM Datastage, Oracle OWB, Microsoft SSIS
and Ab Initio.
Informatica is better than its competitors as it offers a wide range of product editions.
So the user can opt for a specific edition based upon the requirement. Informatica is
constantly featured as Data Integration product leader in the Gartner Magic Quadrant
listing.
Informatica is available for all the popular platforms. It offers cloud-based services so
that with minimal setup an industry can use this tool. Informatica offers real-time
data integration, web services integration, Business to business data integration
(B2B), Big data edition, Master Data Management and connectors for social media and
Salesforce. Forbes has quoted Informatica as the next Microsoft, this itself reflects the
market share Informatica is having over its competitors.
Informatica offers a rich set of features like operations at row level on data,
integration of data from multiple structured, semi-structured or unstructured
systems, scheduling of data operation. It also has the feature of metadata, so the
information about the process and data operations are also preserved.
Chapter 2: Architecture
Before we learn how to use Informatica, we need to understand what are the
important components of Informatica and how it works.
For example, in the below screenshot, you can see under domain window the folder
"Domain_Rajesh" is created under which we have created a node name
"node01_rajesh" and services as "guru99 integration services".
Node is a logical representation of a machine inside the domain. Node is required
to run services and processes for Informatica.
You can have multiple nodes in a domain. In a domain, you will also find a gateway
node.
The gateway node is responsible for receiving requests from different client tools and
routing those requests to different nodes and services.
There are three Informatica Client tools available in Informatica Powercenter. They
are Informatica
Designer
Workflow Monitor
Workflow Manager
These clients can access to the repository using repository service only.
The repository services use locks on the objects, so multiple users cannot modify the
same object same time.
You can enable version control in the repository. With the version control feature, you
can maintain different versions of the same object.
Valid: Valid objects are those objects whose syntax is correct according to
Informatica. These objects can be used in the execution of workflows.
Invalid: Invalid objects are those who does not adhere to the standard or rules
specified. When any object is saved in Informatica, it is checked whether its syntax
and properties are valid or not, and the object is marked with the status
accordingly.
Impacted: Impacted objects are those whose child objects are invalid. For
example in a mapping if you are using a reusable transformation, and this
transformation object becomes invalid then the mapping will be marked as
impacted.
Domain Configuration
As mentioned earlier, domain is the basic administrative control in Informatica. It is
the parent entity which consists of other services like integration service, repository
service, and various nodes.
The domain configuration can be done using the Informatica admin console. The
console can be launched using web browsers.
Once open in a web browser it prompts for administrator login. The password is set
during the Informatica installation.
After login into the Informatica domain, the home page looks something like this.
In the left pane it shows the existing nodes, repository services, integration services
under the domain.
On the main window, it shows the status of those services, whether those are up or
down.
Resilience timeout – If any of the integration service or repository services goes down
then resilience timeout is the no of seconds the application service tries to connect to
those services.
Restart Period – It is the maximum number of seconds the domain spends to restart a
service.
Dispatch Mode – It is the policy used by the load balancer to dispatch tasks to various
nodes.
The mappings and objects that we create in these client tools are saved in the
Informatica repository which resides on the Informatica server. So the client tools
must have network connectivity to the server.
On the other hand, PowerCenter client connects to the sources and targets to import
the metadata and source/target structure definitions. So it also must have connectivity
to the source/target systems.
Repository Service
The repository service maintains the connections from Powercenter clients to the
PowerCenter repository. It is a separate multi-threaded process, and it fetches, inserts
and updates the metadata inside the repository. It is also responsible for maintaining
consistency inside the repository metadata.
Integration Service
Integration service is the executing engine for the Informatica, in other words, this is
the entity which executes the tasks that we create in Informatica. This is how it works
For example, it can combine data from an oracle table and a flat file source.
Relational– these types of sources are database system tables. These database
systems are generally owned by other applications which create and maintain this
data. It can be a Customer Relationship Management Database, Human Resource
Database, etc. for using such sources in Informatica we either get a replica of these
datasets, or we get select privileges on these systems.
Flat Files - Flat files are most common data sources after relational databases in
Informatica. A flat file can be a comma separated file, a tab delimited file or fixed
width file. Informatica supports any of the code pages like ascii or Unicode. To use the
flat file in Informatica, its definitions must be imported similar to as we do for
relational tables.
Chapter 3: Configure Clients and
Repositories
The following components needs to be set up before you can start working in
Informatica PowerCenter.
1. Informatica domain
2. creating repository service
3. creating integration service
4. Creating users
During installation of Informatica server, system prompts for setting up the domain
credentials and database connection.
You need to use the same credentials when you access the domain console.
Troubleshooting- if the home page doesn't open up, check whether Informatica
server process and database service are up and running. If not, start the same.
Step 2) Enter the domain credentials (these credentials are generated during server
installation).
Step 3) On successful login the Informatica administrator home page will appear.
You can select oracle, DB2, Sybase or SQL Server databases to create a repository
Step 1)
Step 4) The Repository service will be created, and will be listed under the domain
tree.
Troubleshooting – if there is an error during service creation, check whether the
correct database credentials are provided, and connection string is correct. After any
changes made to repository properties, recycle the repository service to make the
changes effective.
Step 4) Once repository content is created, change the repository service from
exclusive to normal mode
Note – If the repository service is in exclusive mode, users would not be able to
connect to the repository
Step 5)
1. Click on the Integration service in domain tree and Select Actions option.
2. Select the Enable service option
When both integration service and repository services are created and enabled, they
should be in running status.
How to Configuring client and domain
Before you start using the Informatica integration and repository service of the
Informatica server, you have to configure the Informatica domain at the client. To add
the domain at client following steps are required.
1. Click on username
2. Select privileges tab
3. Select the domain.
Now you will be able to connect to the repository using client tools.
This completes the exercise of repository service creation, integration service creation,
domain configuration, user creation and adding a repository in the client.
Chapter 4: Source Analyzer and Target
Designer
A Source in ETL is an entity from where you pull the records, and then you store these
records in temporary tables (staging tables) or Informatica transformation caches. On
the basis of mapping design document/business requirement you make a change in
these data records (transform the data) and then you load the transformed data in
another tables structures, called target tables.
In every Informatica mapping, there will always be a source and a target. To manage
different sources and targets in Informatica, you have to use source analyzer and
target designer. These tools are integrated into Powercenter designer tool and can be
launched from there itself.
With the help of Source analyzer, you can create or import different types of sources
in Informatica. Similarly by using target designer you can create or import different
types of targets. Informatica provides you the feature to either create source/target
from scratch by designing its structure or you can create source/target by importing
the definitions. When you Import Source from the database, the following metadata
gets imported.
After Successful login, the folders of the user will be listed under the repository name
Note –
Step 2 – We will now create ODBC connection (If you already have ODBC
connection, move to step 3)
2. On the next page, Select user DSN tab and click Add button.
A DSN (Data Source Name) is basically a data structure, and it contains information
about a specific details (database type, location, user details, etc.). This information is
required by ODBC driver so that it can connect to that database.
3. When you click on the add button, you will see a list of drivers for various
databases (Oracle, SQL Server, Sybase, Microsoft Access, Excel, etc.) The driver
which you select depends on what database you want to connect. Select the driver
for the data source. Here in guru99 you will be using oracle database tables.
Select Oracle wire protocol driver.
4. On the next page, select the general tab and enter database details. Then Click on
test connect.
Note – Database details are specific to your installation depending on hostname, port
and SID you selected during database installation. You can get this details by
opening tnsnames.ora file located in your oracle installation folder.
5. Once you made a Test Connect, enter database username, password and then
select "OK" button
Step 3 - In guru99, you will be using oracle's Scott/Tiger schema tables. If you do not
have these tables in the database, create those using this script.
Note – here in this tutorial, both source and target table are present in the same
database schema (guru99). However, you can also create a target in a different
schema.
Step 1- In the Informatica Designer, click target designer icon to launch target,
designer.
Step 2- In next step
To create folder
Mapping is the basic Informatica object with the help of which we can define the data
transformation details and source/target object characteristics. Mappings help us to
define the data transformation at the individual column levels for each row. Even in a
single mapping you can handle multiple sources and targets.
Components of Mapping
Basic components of a mapping are
Source tables
Mapping parameters and variables
Target objects
Mapping transformations
There are various objects that constitute a mapping. A mapping can consist of
sources, targets, mapping parameter and variables, mapplets, various
transformations, and user-defined functions.
Mapping Source: Mapping sources are the objects from where you fetch the
source data. It can be a database table, flat file, XML source or COBOL file source
Mapping target: Mapping target is our destination objects where final processed
data gets loaded. Mapping target can be a relational table of a database, a flat file or
XML file. Sources and targets are mandatory in any mapping, their type can differ
Mapping Parameters and Variables: Mapping parameters and variables helps
you to create temporary variable objects which will help you to define and store
temporary values while mapping data processing. Mapping parameters and
variables are optional users defined data types, which can be created for a mapping
and can be referenced and updated for a specific requirement. We will learn more
about mapping parameters and variables in this section
Mapplets: They are objects which consist of a set of transformation, source or
targets. Mapplets are generally created to reuse the existing functionality of a set of
transformations. It can be used in any no of mappings.
Having a local stage table offers various advantages, like production downtime, won't
affect your ETL system because you have your own "employee_stage" table, instead of
referring to production "employee" table. In a Production system, there can be other
operations and processes which affect the performance. However, when you have
replica staging table, only ETL processes will access it. This offers performance
benefits.
In Stage Mappings,
For example, if your source table contains employee details of deptno 10, 20, 30, and
40. The staging table can be a table having employee records of deptno 10 & 30 only.
The purpose of creating stage tables in Data warehouse is to make the process of data
transformation efficient by fetching only those data which is relevant to us and also to
minimize the dependency of ETL/Data Warehouse from the real-time operational
system.
Naming Convention – mapping names are prefixed with 'm_' followed by source
and target table names separated by underscore sign.
Example – if you are loading emp_target table from the emp table, then mapping
name can be 'm_emp_emp_target'.
Step 1 – Launching Mapping Designer
Note – When you import any relational (database) table in a mapping, an additional
object of source qualifier type will also be created. This source qualifier
transformation is necessary and helps Informatica integration service to identify the
source database table and its properties. Whenever you import a source table, source
qualifier transformation will also be created. You should never delete a source
qualifier object in a mapping.
Step 5 – In this step we will,
Step 6 – Right click anywhere in the mapping designer empty workspace and select
option – Arrange all iconic.
After selecting the option "Arrange all Iconic", the workspace will look like this.
Step 7 - In Informatica, We design with the flow from left to right. So, source tables
should be at the left side, and target tables should be at right. To arrange tables in our
workspace, Select the "emp_target" table then drag and drop it to the right side of the
emp table.
After this rearrangement, the workspace will look like this.
Step 8 - Now you have source and target tables in your mapping, but the mapping is
not yet complete. The source and target tables should be linked to complete a
mapping.
Double click on SQ_EMP table and EMP_TARGET table to change the view of them
from iconic to the graphic.
Step 9 – Right Click on mapping designer workspace and select "Autolink" by name
option.
The Source and the Target tables will be linked, and connecting arrows will appear
from source to the target table.
Note – Here you have linked all source columns to the respective target table column.
It means, for every source record fetched, all the columns of the target will get loaded.
If you want to exclude any specific column from getting loaded, click on that column
link and press delete key from the keyboard. The link will get removed, and the target
column will not get loaded.
Step 11 - Use shortcut "ctrl+s" to save changes to your mapping. On the output
window, you can see the message of mapping validation/parsing. It confirms that your
mapping is valid. Also, there will be a temporary green tick mark next to the mapping
name in mapping folder tree indicating mapping is done successfully.
In mappings there can be a requirement, where we need to pass variable to the
mapping or there can be a scenario where we need to calculate temporary variables
and further required to store them for the next session run of the mapping. For these
purposes, we create mapping parameters and variables.
Mapping parameters are those data types whose value once assigned Mapping variables are objects which can be
remains constant throughout the mapping run. Like if you have referenced throughout the mapping run (to
created a mapping parameter deptno=20, then the value 20 will be access their values) and their values can be
constant for the whole mapping run. The parameter wherever it will reassigned. For example, a mapping
be referenced will always return value 20 for that instance of variable of total_salary can be used in a
mapping run. For a new mapping instance, the parameter value can mapping, and its value can be updated
be redefined. based on salaries.
The mapping parameters and variables are specific to a mapping only. They cannot be
referenced in another mapping.
Note – mapping parameter and variable names always begin with $$.
Summary
Mappings are important in Informatica to match source data with target as per project
requirements. We have discussed stage mappings and concept of Mapping Variables
and Parameters.
Chapter 6: Workflows
Workflow - It defines how to run tasks like session task, command task, email
task, etc.
To create a workflow
A Workflow is like an empty container, which has the capacity to store an object you
want to execute. You add tasks to the workflow that you want to execute. In this
tutorial, we are going to do following things in workflow.
For Example, in your mapping if you have source table in oracle database, then you
will need oracle connection so that integration service can connect to the oracle
database to fetch the source data.
Relational Connection
Ftp Connection
Queue
Application
The choice of connection you will create, will depend on the type of source and target
systems you want to connect. More often, you would be using relational
connections.
Step 4 – You will return on the previous window. Click on the close button.
Now you are set with the relational connection in workflow manager.
Task Developer
Worklet Designer
Workflow Designer
Task Developer – Task developer is a tool with the help of which you can create
reusable objects. Reusable object in workflow manager are objects which can be
reused in multiple workflows. For Example, if you have created a command task in
task developer, then you can reuse this task in any number of workflows.
The role of Workflow designer is to execute the tasks those are added in it. You can
add any no of tasks in a workflow.
Command task
Session task
Email task
Email task - With the help of email task you can send email to defined recipients
when the Integration Service runs a workflow. For example, if you want to monitor
how long a session takes to complete, you can configure the session to send an email
containing the details of session start and end time. Or, if you want the Integration
Service to notify you when a workflow completes/fails, you can configure the email
task for the same.
Step 4 – To configure the task, double click on the command task icon and it will
open an "edit task window". On the new edit task window
Step 5 – On the command editor box, enter the command "mkdir C:\guru99" (this is
the windows command to create a folder named "guru99") and select OK.
Afther this step you will return to the edit tasks window and you will be able to see the
command you added in to the command text box.
When you create a workflow, it does not consist of any tasks. So, to execute any task
in a workflow you have to add task in it.
Step 4 - To add command task that we have created in Task developer to the workflow
desinger
Step 5 - Select the "link task option" from the toolbox from the top menu. (The link
task option links various tasks in a workflow to the start task, so that the order of
execution of tasks can be defined).
Step 6 – Once you select the link task icon, it will allow you to drag the link between
start task and command task. Now select the start task and drag a link to the
command task.
Now you are ready with the workflow having a command task to be executed.
Once the workflow is executed, it will execute the command task to create a folder
(guru99 folder) in the defined directory.
Session Task
A session task in Informatica is required to run a mapping.
Without a session task, you cannot execute or run a mapping and a session task can
execute only a single mapping. So, there is a one to one relationship between a
mapping and a session. A session task is an object with the help of which Informatica
gets to know how and where to execute a mapping and at which time. Sessions cannot
be executed independently, a session must be added to a workflow. In session object
cache properties can be configured and also advanced performance optimization
configuration.
Step 4 – A window for selecting the mapping will appear. Select the mapping which
you want to associate with this session, for this example select "m_emp_emp_target"
mapping and click OK Button.
Step 5 – After that, click on "Done" button
Step 12 – Double click on the session object in wokflow manager. It will open a task
window to modify the task properties.
Now your configuration of workflow is complete, and you can execute the workflow.
How to add multiple tasks to a start task
The start task is a starting point for the execution of workflow. There are two ways of
linking multiple tasks to a start task.
1. Parallel
2. Serial
In parallel linking the tasks are linked directly to the start task and all tasks start
executing in parallel at same time.
After linking the session task, the workflow will look like this.
Step 5 – Start the workflow and monitor in the workflow monitor.
Step 2 – Confirmation dialogue box will appear in a window, select yes option
The link between the start task and session task will be removed.
Step 3 – Now again go to top menu and select the link task option from the toolbox
Workflow Variable
Workflow variables allows different tasks in a workflow to exchange information with
each other and also allows tasks to access certain properties of other tasks in a
workflow. For example, to get the current date you can use the inbuilt variable
"sysdate".
Most common scenario is when you have multiple tasks in a workflow and in one task
you access the variable of another task. For example, if you have two tasks in a
workflow and the requirement is to execute the second task only when first task is
executed successfully. You can implement such scenario using predefined variable in
the workflow.
When you execute this workflow, the command task executes first and only when it
succeeds then only the session task will get executed.
Workflow Parameter
Workflow parameters are those values which remain constant throughout the run.
once their value is assigned it remains same. Parameters can be used in workflow
properties and their values can be defined in parameter files. For example, instead of
using hard coded connection value you can use a parameter/variable in the connection
name and value can be defined in the parameter file.
Parameter files are the files in which we define the values of mapping/workflow
variables or parameters. There files have the extension of ".par". As a general standard
a parameter file is created for a workflow.
Advantages of Parameter file
[folder_name.WF:Workflow_name]
$Parameter_name=Parameter_value
We will be creating a parameter file for the database connection "guru99" which we
assigned in our early sessions for sources and targets.
When we execute the workflow, the workflow picks the parameter file looks for the
value of its paramters/variables in the parameter file and takes those values.
Workflow monitor is a tool with the help of which you can monitor the execution of
workflows and task assigned to the workflow.
In workflow monitor you can,
Navigator window- shows the monitored repositories, folders & integration service
Output window – displays the messages from integration services and repository
Properties window – displays the details/properties about tasks and workflows
Time window – displays the progress of the running tasks & workflows with timing
details.
In the workflow monitor tool, you will see the repositories and associated integration
services on the left side. Under the status column, you will see whether you are
connected or disconnected to integration service. If you are in the disconnected mode,
you won't see any running workflows. There is a time bar which helps us to determine
how long it took a task to execute.
Task View
Task view displays the workflow runs in report format, and it is organized by workflow
runs. It provides a convenient approach to compare workflow runs and filter details of
workflow runs.
Workflow run list – Shows the list of workflow runs. It contains folder, workflow,
worklet, and task names. It displays workflow runs in chronological order with the
most recent run at the top. It displays folders and Integration Services
alphabetically.
Status message - Message from the Integration Service regarding the status of the
task or workflow.
Node - Node of the Integration Service executed the task.
Start time - The time at which task or workflow started.
Completion time – The time at which task or workflow completed the execution.
Status - Shows status of the task or workflow, whether the workflow started,
succeeded, failed or aborted.
To switch from Gantt chart to Task view or vice versa, click on the respective button as
shown in the screenshot to change the mode.
Step 3 – Here you can view the currently running workflow, which is having status as
"running".
Step 4 – Once the workflow execution completes, it status would change to
succeeded/failed along with start and end time details.
Step 6 – Click on each of the menu of the properties window to view specific details.
Here we chose "Task Details" to view. It will display all the details like Instance Name,
Task Type, Start Time, Integration Service Name, etc.
Task details -
Source and target statistics gives the details of source and target. For example, how
many rows are fetched from the source and how many rows are populated in the
target the current throughput, etc
In the following screen, 14 records are fetched from the source, and all 14 are
populated in the target table.
Applied rows signify how many records Informatica had tried to update or insert
the target
Affected rows signify how many numbers of applied rows were actually
succeeded.Here all 14 rows are successfully loaded in the target, so the count is
same for both.
Rejected rows signify how many rows are dropped due to target constraint or
other issues.
In this tutorial, you have learned how to open and monitor the workflows and tasks
using workflow monitor.
Chapter 7: Debug Mappings & Session
Objects
The debugger is a very handy utility in Informatica. When we execute the mapping,
Informatica executes the mapping based on the transformation logic defined.
Its execution is similar to batch mode execution where we cannot see the intermediate
results, or how the data is getting modified from transformation to transformation.
Many a times we get the data in the target which is not as we expected. With the help
of debugger, we can analyze how the data is moving within the mapping. We
can analyze the data row by row in a debugger.
The use of debugger requires a session instance, we can either reuse an existing
session which was created earlier to run the mapping, or we can create a new debug
session instance. Since during debugging the data will be fetched from the source, so
we have the option of what to do with that data.
Target window
Instance window
In target window, the target record which is going to be inserted into the target table is
shown. In instance window, you can select any of the transformation instances of the
mapping. After that, all the ports of that transformation are visible in this window, and
you can analyze the data, how it is getting transformed.
Our source data for the "emp" table looks like this.
For the ename "SCOTT", using debugger we will check that what value for the "sal"
(salary) column is fetched in the Informatica.
When you use the existing session, Informatica will use the earlier session which you
created for this mapping. When you select the create debug session instance, a new
session instance will be created for debugging.
In this example, we will select existing reusable session instance. The integration
service will be the default.
Step 7 - The debugger has started, but it hasn't fetched any rows from the source.
During debugging it fetches data from source row by row. To start fetch first row from
the source
Press F10 button (shortcut for going to next instance, instead of going to mapping
option then debugger option)
After moving to next instance, the data in instance window will change to next record.
Step 10 - Keep pressing F10 button, till you reach to the required record of "SCOTT"
employee. Once you reach that point, the instance window will be like as shown below
This is the instance, which we were looking for. So in the instance window, we can
analyze the data. Whether it is coming correctly, or any value is missing.
In the source data, the details of the "SCOTT" employee were like job = "ANALYST",
manager id = "7566", sal "3000."
And in the debugger, for the employee "SCOTT" we have analyzed all these details, and
they are matching with the source data, so we are assured that the records were
fetched correctly by source qualifier.
For example if we define deptno=20 in a breakpoint, then whenever for a record if this
condition is true, then the debugger will stop to that instance of data.
1. Go to mappings menu
2. Select debugger option
3. Select "edit breakpoints" option
Step 2 – You will see an "edit breakpoint" window as shown below
Step 3 – In the breakpoint window, click on the add button
Step 4 – In the new window
1. In port drop down, select "ENAME" option ( as we are debugging for record
ename='SCOTT')
2. In the value box, enter value "SCOTT."
3. Select OK button
Step 8 – Now start the debugger as mentioned in the previous topic
Go to mapping menu -> select debugger option -> select start debugger -> select
session instance
Once the Informatica is done with going through all the instances, the debugger is
stopped automatically. If you want to analyze the data again, you have to restart the
debugger again.
Tracing in Informatica
Whenever you execute a session for a mapping, a session log is generated. It consists
of the details of the execution like
Inside the transformations of a mapping, you can configure and set how much details
Informatica writes inside the session log.
There are four levels which can be set for tracing inside the transformations.
Normal In normal mode, Informatica writes about the transformation initialization and status
information, errors encountered and summarizes sessions result.
Terse In Terse Mode, Informatica writes only about initialization information and error details. Terse
mode writes minimum details into the session log.
Verbose In Verbose Initialization mode, Informatica writes the log details similar to normal mode and in
initializa addition also writes about the data and index files used and transformation statistics.
tion
Verbose In Verbose data, Informatica writes log similar to verbose initialization but in addition also
data writes every data which passes through the transformation.
Note: By default the tracing level is set to normal, it is recommended to use verbose
data mode only when you are debugging the mapping. The terse mode can be used to
improve the performance, as it writes the minimum details. So the overhead of writing
log will be minimum.
1. Open workflow monitor and select the session which was executed in the last
step
2. Click on "session log" option to open the session log for the mapping
This will open session log window for the mapping
The session log provides the detail about how your session got executed. It provides
the timing details when execution started and stopped. It provides the basic idea about
the performance. It mentions which database connection you are using, what
parameter file you are using. It also summarizes about the source and targets by
mentioning how many source records were fetched, how many records were loaded
into the target, etc.
Step 6 – Scroll down in the log, and you can see additional log details captured
including the data records.
In this way, you can set the tracing level in mappings to capture the additional details
for debugging.
In this tutorial, you have learned how to set, configure and execute the debugger and
how to set tracing level in mappings. These options provide you the ability to debug
the mappings.
A session object is a task, just like other tasks that we create in workflow manager.
Any session you create must have a mapping associated with it.
A session can have a single mapping at a time and once assigned, it cannot be changed.
To execute a session task, it must be added to a workflow.
A session can be a reusable object or non-reusable. When you create a session in task
developer, then it can be reused, but when you create a session in workflow designer,
then it is non-reusable.
Properties Of Session
Using the properties of the session you can configure various characteristics of the
session like pre and post SQL scripts, log file name and path, memory properties, etc.
You can also override mapping properties in the session properties. In this section, we
will discuss the following important properties of the session.
Step 2) Double click on the session icon inside Task Developer to open edit task
window.
Step 3) Inside the "Edit Task" window clicks on the properties tab.
Step 4) In properties tab, it will show the properties of the session
Treat Source Rows As Property
This property allows you to define how the source data affects the target table. For
example, you can define that the source record should be inserted or deleted from the
target.
Insert
Update
Delete
Data-driven
When this property is set to insert, the source data will be marked to be inserted.
It means the data will only be inserted.
When the property is set to update, the target data will be updated by the source
data. For updating of data primary key needs to be defined in the target table.
When property is set to delete the source data which is already present in the
target will be deleted from the target table. For this property to execute and apply
the changes, the primary key should be defined in the target table.
With the property set to data driven, the Informatica checks what source records
are marked. If in a mapping the source records are marked as insert then records
will be inserted into the target. If records are marked as an update in the mapping,
then the records will be updated in the target. So what operation will be performed
at the target depends on how records are handled inside the mapping.
How To Make Treat Source Rows – Delete
Step 1 -
1. In the property tab of the session task, select "Delete" option in "Treat Source
Rows as"
2. Select OK Button
1. For the EmpNo column, select key type as "primary key" from the scroll down
menu and
2. Select OK button.
Step 4 – Save the changes in Informatica and execute the workflow for this mapping.
When you execute this mapping, the source records which are already present
in the target will get deleted.
For example, if you are inserting 20,000 records in a target table, and you define
commit interval as 5,000, then after every 5,000 insertions of records in the target, a
commit operation will be performed.
Session Log File Name & Session Log File
directory
Configure this property to modify
Memory Properties
Memory properties give us the flexibility to fine tune the memory allocated to the
Informatica for performance optimizations. When there are high bottleneck and
performance is poor then you can try to improve the performance using the memory
properties.
To configure memory properties click on the "config object" tab of the edit task
window. It will open another window where you can configure the changes.
In this section, you can configure the memory properties. For example, default buffer
block size, sequential buffer length, etc. Changes to this properties will determine how
much memory should be allocated to Informatica services for their operation.
Log options
In this property section, you can configure the log properties of the session. You can
set the no for how many no of logs you want to save for a session, session log file max
size.
Error Handling
In this section, you can configure the error properties for the session.
Using Stop on errors you can configure after how many errors the session has to be
stopped.
Using override tracing you can override the mapping tracing levels.
You can also configure the behaviour of the session for various errors encountered for
example stored procedure error, pre-post SQL error, etc.
Using SQL query property, you can override the SQL for the source. You can also
override the source table name in this section.
Target Properties
In this section, you can configure the details of the target. You can define whether
target load has to be a bulk load or a normal mode.
In bulk load, the performance gain is achieved as during the load there are no redo log
buffers managed by the database.
On the other hand, normal load is slower as compared to bulk load, but in case of
failure database recovery is possible.
You can also define the property to truncate the target table before populating it. It
means before loading any records in the target, the target table will be truncated, and
then the load will be performed. This property is useful when we create mappings for
stage load.
We can also define target table pre SQL and post SQL. Pre SQL is the piece of SQL
code which will be executed before performing insert in the target table, and post SQL
code will be executed after the load of target table is completed.
Success or Failure Of Session Task
When you have multiple sessions inside a workflow, then there can be a scenario
where one or more session fails. In such condition, there comes a question of what
would be the status of the workflow because you are having a workflow in which few
tasks have failed, and few task got succeeded. To handle such conditions, Informatica
provides the option to set this failure specific property inside the workflow. To
configure such behaviour –
Step 2 – Double click on the command task, this will open edit task window
1. Select the general tab
2. Select the check box against property "fail parent if this task fails"
3. Select Ok Button
Step 3 – Double click on the session task, this will open edit task window
When you execute this workflow after making the above changes if any of the tasks
fails the workflow status will be made as failed. so you can identify that during the
execution of your workflow some of its task has failed.
Chapter 8: Transformations
Transformations are the objects in Informatica which creates, modifies or passes data
to the defined target structures (tables, files or any other target).
The purpose of the transformation in Informatica is to modify the source data as per
the requirement of target system. It also ensures the quality of the data being loaded
into the target.
For example, performing tax calculation based upon source data, data cleansing
operation, etc. In transformations, we connect the ports to pass data to it, and
transformation returns the output through output ports.
Classification of Transformation
Transformation is classified into two categories, one based on connectivity, and other
based on the change in no of rows. First we will look the transformation based on
connectivity.
Connected Transformations
Unconnected Transformations
The connected transformations are preferred when for every input row,
transformation is called or is expected to return a value. For example, for the zip codes
in every row, the transformation returning city name.
The unconnected transformations are useful when their functionality is only required
periodically or based upon certain conditions. For example, calculation the tax details
if tax value is not available.
Active Transformations
Passive Transformations
Active Transformations are those who modifies the data rows and the number of input
rows passed to them. For example, if a transformation receives ten number of rows as
input, and it returns fifteen number of rows as an output then it is an active
transformation. The data in the row is also modified in the active transformation.
Passive transformations are those who does not change the number of input rows. In
passive transformations the number of input and output rows remain the same, only
data is modified at row level.
In the passive transformation, no new rows are created, or existing rows are dropped.
Filter Transformation
Using the filter transformation, we can filter the records based on the filter condition.
Filter transformation is an active transformation as it changes the no of records.
For example, for loading the employee records having deptno equal to 10 only, we can
put filter transformation in the mapping with the filter condition deptno=10. So only
those records which have deptno =10 will be passed by filter transformation, rest
other records will be dropped.
Step 4 – The filter transformation will be created, Select "Done" button in the create
transformation window
Step 5 – In the mapping
1. Drag and drop all the Source qualifier columns to the filter transformation
2. Link the columns from filter transformation to the target table
Step 6 – Double click on the filter transformation to open its properties, and then
In this way, you can filter the source records using filter transformation.
1. It will confirm the data you have selected are correct and ready for loading into
the target table
2. Click on OK button to process further
Save the mapping (using ctrl+s Shortcut) and execute the workflow, after execution
only the selected columns will be loaded into the target.
In this way, you can override in source qualifier what columns needs to be fetched
from the source & this is the only way to override what specific columns will be
brought inside the mapping.
1. Source Filter – Using the source filter property you can filter the number of
source records. For example, you want to fetch only the employees of deptno 10,
then you can enter the filter condition deptno=10 in source filter property and
execute the data.
2. Number for sorted ports – In source qualifier transformation, you can also
sort the input records based on the ports number. So when the data is passed on
to the transformations inside the mapping, it will read the port number and sort
the data accordingly.
As data can be sorted based on a single or multiple ports, you have to give the
number of ports which will be used in sorting. If you give value as 1, then only
empno data will be sorted. If you give value as 2 then on empno and ename on
both columns data will be sorted.
3. Select Distinct – you can fetch only distinct records from the source using this
property. When you select the select distinct option, only distinct combination of
source data will be fetched by source qualifier.
Aggregator Transformation
The aggregator transformation performs aggregate calculations like sum, average, etc.
For example, if you want to calculate the sum of salaries of all employees department
wise, we can use the Aggregrator Transformation.
The aggregate operations are performed over a group of rows, so a temporary
placeholder is required to store all these records and perform the calculations.
For this, aggregator cache memory is used. This is a temporary main memory which is
allocated to the aggregator transformation to perform such operations.
In this example, we will calculate the sum of salaries department wise. For this, we
require a new column to store this sum. So, first of all, we will prepare a new column.
Step 1 – Create a new database target table, for example, say "sum_sal_deptwise",
using the below script. You will see the new database target table is created under
Target folder in next step.
In order to create new mapping, we need source table (EMP) and target table
(sum_sal_deptwise) both in mapping designer for that we need to
1. Import the target table "sum_sal_deptwise" in the mapping.
2. Import the source table "emp".
1. From the Source Qualifier, delete the columns empno, ename, job, mgr, hiredate
& comm so leaving only the columns deptno and sal.
2. Create a new aggregator transformation using the toolbox menu as shown in
screen shot. When you click on the aggregator icon, a new aggregator
transformation will be created.
Step 4 - Drag and drop SAL & DEPTNO columns from source qualifier (SQ_EMP) to
the aggregator transformation
Step 5 – Double click on the aggregator transformation to open its properties, and
then
Now save the mapping and execute it after creating a new session for this mapping.
The target table would contain the sum of salaries department wise. In this way, we
can use aggregator transformation to calculate aggregate results.
Router Transformation
Similar to filter transformation the router transformation is also used to filter the
source data.
The additional functionality provided beside filtering is that the discarded data
(filtered out data) can also be collected in the mapping, as well as the multiple filter
conditions can be applied to get multiple sets of data.
For example, when filtering the data form deptno =10, we can also get those records
where deptno is not equal to 10. So, router transformation gives multiple output
groups, and each output group can have its own filter condition.
In addition there is also a default group, this default group has those record sets which
doesn't satisfy any of the group conditions. For example, if you have created two
groups for the filter conditions deptno=10 & dept=20 respectively, then those records
which are not having deptno 10 and 20 will be passed into this default group. In short
the data which is rejected by the filter groups will be collected by this default group
and sometimes there can be a requirement to store these rejected data. In such
scenarios, default output group can be useful.
To allow multiple filter condition, the router transformation provides group option.
Step 4 – The router transformation will be created in the mapping, select done option
in the window
Step 5 – Drag and drop all the columns from Source qualifier to router
transformation
Joiner transformation
The joiner transformation provides you the option to create joins in Informatica. The
joins created using joiner transformation are similar to the joins in databases. The
advantage of joiner transformation is that joins can be created for heterogeneous
systems (different databases).
In joiner transformation, there are two sources which we are going to use it for joins.
These two sources are called
Master Source
Detail Source
In the properties of joiner transformation, you can select which data source can be
Master and which source can be detail source.
During execution, the master source is cached into the memory for joining purpose.
So it is recommended to select the source with less number of records as the master
source.
In normal join only matching rows are returned from both the sources.
In this example, we will join emp and dept tables using joiner transformation
Step 1 – Create a new target table EMP_DEPTNAME in the database using the below
script and import the table in Informatica targets.
Step 2 - Create a new mapping and import source tables "EMP" and "DEPT" and
target table which we created in the previous step
Step 4 – Drag and drop all the columns from both the source qualifiers to the joiner
transformation
Step 5 - Double click on the joiner transformation, then in the edit transformation
window
Step 7 –Double click on the joiner transformation to open edit properties window,
and then
Rank Transformation
Rank Transformation performs the filtering of data based on group and ranks.
For example, you want to get ten records of employees having highest salary, such
kind of filtering can be done by rank transformation.
Rank transformation also provides the feature to do ranking based on groups. Like if
you want to get top ten salaried employee department wise, then this grouping can be
done with this transformation.
Rank transformation is an active transformation, as it affects the number of output
rows.
The rank transformation has an output port by which it assigns a rank to the rows.
Our requirement is to load top 3 salaried employees for each department; we will
implement this using rank transformation.
Step 5 – Connect all the ports from source qualifier to the rank transformation
Step 6- Double click on the rank transformation and it will open "edit transformation
window". In this window
For example, you want to assign sequence values to the source records, then you can
use sequence generator. The generated sequence values can be like 5, 10, 15, 20, 25
etc. or 10, 20, 30, 40, 50 etc. depending upon the configured properties of the
transformation.
The sequence generator is also used to generate primary key values. It is a passive
transformation & it does not affect the number of input rows.
CURRVAL
NEXTVAL
Start Value – It is the first value that will be generated by the transformation, the
default value is 0.
Increment by – This is the number by which you want to increment the values.
The default value is 1.
End value – It is the maximum value that the transformation should generate.
Cycle – if this option is set then after reaching the end of the value, the
transformation restarts from the start value.
In this example, we will generate sequence numbers and store in the target.
Step 3 – Create a new mapping and import EMP source and EMP_SEQUENCE target
table
Step 5 - Sequence generator transformation will be created, select the done option
Step 6 - Link the NEXTVAL column of sequence generator to SNO column in target
Step 7 – link the other columns from source qualifier transformation to the target
table
Step 8 – Double click on the sequence generator to open property window, and then
The sno column in the target would contain the sequence numbers generated by the
sequence generator transformation.
In our example, the sequences will be like 1 – Scott, 2 – King, 3 – Adam, 4 – Miller,
etc.
When processing a high volume of data, there can be a situation when to commit the
data to the target.
If a commit is performed too frequently, then it will be an overhead to the system. If a
commit is performed too late then in case of failure there are chances of data loss.
1. TC_CONTINUE_TRANSACTION
In tc_continue_transaction there are no operations performed, the process of
data load continues as it is
2. TC_COMMIT_BEFORE
In tc_commit_before when this flag is found set, a commit is performed before
the processing of current row.
3. TC_COMMIT_AFTER
In tc_commit_after the current row is processed then a commit is performed.
4. TC_ROLLBACK_BEFORE
In tc_rollback_before, rollback is performed first then data is processed to write
5. TC_ROLLBACK_AFTER
In this example, we will commit data to the target when dept no =20 condition is
found true
Step 3 – The transaction control transformation will be created, select done button
Step 4 - Drag and drop all the columns from source qualifier to the transaction
control transformation then link all the columns from transaction control
transformation to the target table
Step 5 – Double click on the transaction control transformation and then in the edit
property window
It means if deptno 20 is found then commit transaction in target, else continue the
current processing.
Step 7 – Select OK in the previous window
Now save the mapping and execute it after creating session and workflows. This
mapping will commit the data to the target whenever department number 20 is found
in the data.
In previous topics, we used joiner transformation to join "emp" and "dept" table to
bring department names.
Step 1 – Create a new mapping with EMP as source and EMP_DEPTNAME as target
Step 4 - Lookup transformation will be created with the columns of DEPT table, now
select done button
Step 5 - Drag and drop DEPTNO column from source qualifier to the lookup
transformation, this will create a new column DEPTNO1 in lookup transformation.
Then link the DNAME column from lookup transformation to the target table.
The lookup transformation will lookup and return department name based upon the
DEPTNO1 value.
Step 6 – Double click on the lookup transformation. Then in the edit transformation
window
Step 7 – Link rest of the columns from source qualifier to the target table
Now, save the mapping and execute it after creating the session and workflow. This
mapping will fetch the department names using lookup transformation.
The lookup transformation is set to lookup on dept table. And the joining condition is
set based on dept number.
Reusable Transformation
A normal transformation is an object that belongs to a mapping and can be used inside
that mapping only. However, by making a transformation reusable it can be re-used
inside several mappings.
Step1 – Open the mapping which is having the transformation, here we are making
the rank transformation reusable.
Step 2 – Double click on the transformation to open edit transformation window.
Then
Normalizer Transformation
Normalizer transformation is a smart way of representing your data in more organized
manner. It is used to convert a single row into multiple rows and vice versa. If in a
single row there is repeating data in multiple columns, then it can be split into
multiple rows. Sometimes we have data in multiple occurring columns. For example
Student Name Class 9 Score Class 10 Score Class 11 Score Class 12 Score
Student 1 50 60 65 80
Student 2 70 64 83 77
In this case, the class score column is repeating in four columns. Using normalizer, we
can split these in the following data set.
Step 1 – Create source table "sales_source" and target table "sales_target" using the
script and import them in Informatica
1. Link the four column of source qualifier of the four quarter to the normalizer
columns respectively.
2. Link store name column to the normalizer column
3. Link store_name & sales columns from normalizer to target table
4. Link GK_sales column from normalizer to target table
Save the mapping and execute it after creating session and workflow. For each quarter
sales of a store, a separate row will be created by the normalizer transformation.
The source data had repeating columns namely QUARTER1, QUARTER2, QUARTER3,
and QUARTER4. With the help of normalizer, we have rearranged the data to fit into a
single column of QUARTER and for one source record four records are created in the
target.
In this way, you can normalize data and create multiple records for a single source of
data.
Lookup Transformation –
Create an index for the column in a lookup table which is used in lookup condition.
Since the lookup table will be queried for looking up the matching data, adding an
index would increase the performance.
If possible, instead of using lookup transformation use join in the database. As
database joins are faster, performance will be increased.
Delete unnecessary columns from the lookup table and keep only the required
columns. This will bring down the overhead of fetching the extra columns from the
database.
Filter Transformation –
Use filter transformation as early as possible inside the mapping. If the unwanted
data can be discarded early in the mapping, it would increase the throughput.'
Use source qualifier to filter the data. You can also use source qualifier SQL
override to filter the records, instead of using filter transformation.
Aggregator Transformation
Filter the data before aggregating it. If you are using filter transformation in the
mapping, then filter the data before using aggregator as it will reduce the
unnecessary aggregation operation.
Limit the no of ports used in the aggregator transformation. This will reduce the
volume of data that aggregator transformation stores inside the cache.
Bring only the required columns from the source. Most of the times not all the
columns of the source table are required, so bring only the required fields by
deleting the unnecessary columns.
Avoid using order by clause inside the source qualifier SQL override. The order by
clause requires additional processing and performance can be increased by
avoiding it.