Data Management Strategy Ver 3
Data Management Strategy Ver 3
As City Power attempt to make Yet, because analytics can be The reality is that in order for City Power Doing so means addressing both
good on the promise of turning “owned” by many players, the to succeed at creating a more insight- the talent capabilities within ES
data into insights, it is key to assess task is inherently more driven organization, data and analytics own team as well as the relative
the competitive landscape for challenging for Data Analytics needs to first take a step back and maturity of analytics within the
analytics within the organisation. department who is tasked with assess where each business unit’s organisation.
Since analytics is a cross-functional managing a bigger part of the data capability sits within each group
enabler, it then concerns the whole analytics agenda within the and how this can be positioned to
organisation. organisation. deliver value in the near and long-term.
Naturally the question maybe - why does analytics requires this level of strategy. Shouldn’t the focus be on simply making the data accessible,
secured and governed and providing the tools for analytics teams and their customers to access that data?
The challenge is that as the organization is increasingly elevating analytics as a differentiating capability through the use of cognitive
applications, automation of functional tasks, and the identification of new business models, there is a gap between boardroom expectations
and what existing teams are structured to deliver. This gap requires to be closed as a priority to ensure mature workflows for continuous
software delivery, human centered design and agile data that is crucial for successfully building data analytics services.
2
INTR O D U C TION
To create an Insight Data Organisation strategy - it is key to first review the current Data and Analytics strategies, Master Data Management, and Data Governance
Frameworks. Workshops are required with the Executive Management, the current analytics teams, ICT Team, the architecture team, and analytics business consumers to
understand the current impact of analytics, data pain points and future vision. In addition, a current state maturity assessment is required within each dimension (people,
process, data and technology). In parallel to this, a proof of value (POV) can be executed whereby some of the assumptions in the strategy can be tested using data, next
generation technology and analytics processes. The proof of value will in turn identify analytics insights and enable the team to incorporate key learnings back into the
strategy and roadmap.
The case for change is real and requires that action is taken now to deal with the data pain points in order to achieve the benefits of an analytics driven culture of innovation
and agile decision making. With the right data and analytics capabilities in place City Power will be able to:
• Make insight data driven decisions in areas such as tariffs, revenue and risk etc.
• Increase proactive maintenance - improve operational efficiency such as end-to-end unplanned maintenance processing time;
• Reduce losses – through combining of granular grid and consumer data to isolate and efficiently eliminate non-technical losses
• Improve workforce productivity
• Improve the understanding of our customers to improve their services experience – thus making it easy for customers to interact with us;
• Generate long term shareholder value
3
Current Data Stack assessment results
The first step is to assess where does City Power play on ‘The Analytics Stack,’ i.e. the end-to-end capabilities from data sourcing up through to the generation of insights
for business decision .
The As-Is assessments that were conducted a while back (2014) provided the following insights from a people, process and technology landscape perspective. This
capability assessment results highlighted data management issues and gaps within City Power. – assessment results are represented in the table below:
Data Governance Data Structure Data Architecture Master Data & Metadata Data Quality Data Security
Process Data Policies Data Taxonomy Data Sizing, Storage, and Data Definitions Data Quality Rules and Policies Compliance and Security
Movement Architecture Policies
Data Standards Logical Data Models Master Data
Business Data Ownership Business Process Flows Data retention and Metadata Data Cleansing Standards Local, National, and
Deletion Policies international Laws
Data Workflow Design Standards & Reference Data Compliance Rules
Guidelines
Project Approval & Issues Physical Data Models
People Data Stewards Enterprise Data Solution Architects Data Stewards Data Administration Corporate Security
Architects
Business Data Owners Data Modelers Storage / Technical Business Data Owners Data Quality Services Team Auditors
Architects
Management Industry Groups Business Function Owners Data Governance Compliance Department
Committee (DGC)
DBAs DBAs
Technology Data Rules Library Data Modeling Tools Archiving Tools Master Data Management Data Profiling, Quality, Security Software Access
Tools Enrichment and Monitoring Rights Management
Automated Notifications (Workflow)
Data Modeling Libraries Storage management and Reference Data Architecture Tools
ETL Tools Data Audit Trails
Hardware
Data Quality & Metadata Design / CASE Tools Technical Architecture Metadata Repository / Audit Reports
(as enablers) Technology
4
Results of the Assessment
1. The data management policy requires approval from all the relevant stakeholders.
2. Data management capabilities exist, but are provided in silos to areas (or projects) that have current priority.
3. Only certain parts of the data management capabilities are provided to certain projects depending on the requirement. One project will develop architectures
and another will execute a data cleaning exercise. Insufficient data management is enforced on projects and therefore little to no cohesive data management
efforts is executed.
4. Because of a lack of a single data knowledgebase (repository), knowledge retention is difficult and a lot of rework and duplication of effort occurs.
5. Data ownership and stewardship is lacking and, where it exists, it is self-appointed with no real accountability.
6. Data management does not form part of the Demand Planning, SCM or any other project lifecycle processes and it is therefore difficult to enforce on initiated
projects.
7. Data governance in City Power is focused on solving audit issues without the required structures and access to relevant systems and data to enable
governance (i.e. no data owners or stewards, etc.).
8. No data management tools are agreed upon and tools are used based on availability, combined with user usage limitations.
9. The organisation’s information management maturity is reactive using a “fire fighting” approach to solving problems
5
Data management capabilities required
This capabilities are required to understand and manage the data assets to support and drive City Power’s initiatives
6
ENTERPRISE ANALYTICS
7
Analytics Reference Model
The future state capabilities required to enable Analytics as a competitive
differentiator are illustrated in the analytics reference model below:
Data
Data Quality
Quality Data stewardship Data
Data governance
governance
Prioritization Project assessment Supplier management Technology innovation Financial
Financial Management
Management Policies
Policies &
& processes
processes
remediation improvement
improvement
• GIS, SAP PM
• Asset & Maintenance Operational
Operational reporting
reporting Asset
Asset &
& Maintenance
Maintenance reporting
reporting Prescriptive
Prescriptive Geospatial
Geospatial
• Procurement Analytics
• Warehouse Tariff
Statutory
Statutory reporting
reporting Tariff &
& Metering
Metering reporting
reporting Statistical
Statistical Visualizations
Visualizations
• Revenue & Billing
• HR & Transformation
Risk
Risk reporting
reporting Revenue
Revenue &
& Billing
Billing reporting
reporting Machine
Text
Text Analytics
Analytics Machine Learning
Learning
Currently CP has not
tapped into many benefits
Discovery
of analytics
Technology
Technology
discovery Analytic
Analytic discovery
discovery Data
Data discovery
discovery Self
Self service
service Data
Data profiling
profiling Model
Model prototyping
prototyping Data
Data rule
rule reverse
reverse engineering
engineering
discovery
To take advantage of the opportunities presented by data, City Power must first take a holistic view of the supporting architecture. The image below represents the
reference architecture containing the core structural components required. This reference architecture will be used to review the current state and plan the future state
analytics and BI technology capabilities
Infrastructure,
ANALYTICS VISUALISATION platform &
Descriptive, diagnostic, predictive, and prescriptive analytical User access to information in alternate ways to ease understanding software services
Secure infrastructure, platforms, and software as a service
Digitization and Standard processes for Vertical digitization, standardized and Horizontal integration of processes Fully digitized, integrated partner
collaboration partly in place harmonized internal processes and and data flows with customers and ecosystem with self-optimized,
integration of
data flows within the company; external partners, intensive data use virtualized processes, focus on core
vertical and limited integration competency; decentralized
through full integration across the
horizontal value network. autonomy. Near real-time access to
chains extended set of operative
information
Analytical capabilities mainly Analytical capabilities supported by Central BI system consolidating all Central use of predictive analytics
Data & Analytics for real-time optimization and
based on semi-manual data central business intelligence (BI) relevant internal and external
as core capability extracts. Selected monitoring data information sources, some predictive automated event handling with
system Isolated, not standardized
processing, no event management decision support systems with some intelligent database and self-learning
analytics. Specific decision support
predictive analytics algorithm enabling impact analysis
and event management systems
and decision support
Fragmented IT architecture in- Homogeneous IT architecture in- Common IT architectures in partner
Agile IT and Data house. No data architecture in- Single Data Lake with external data
house. Connection between different network. Interconnected single data
architecture place integration functionalities and flexible
data cubes developing. lake with high-performance
organisation. with secure data
architecture
exchange
10
City Power benefits to an accelerated data analytics approach
to Big Data and Analytics
An accelerated approach to Data and ICT architecture modernization can deliver significant value to the business
• Improve effectiveness • Improve the usefulness and granularity of demand-side management and demand response
programs in terms of better customer segmentation and other benefits
• Improve asset management in an asset-intensive industry.
• Lead to better grid operations management in extreme weather, including reduced outage times,
cost savings from better SAIFA and SAIDI indexes, and fewer dissatisfied customers
• Ease of providing customer- satisfaction analytics and customer-journey optimization
• Improve efficiency • Improve revenue collection and reduce collection-risk and bad debt
CP benefits to • Lead to reduced energy theft and other non-technical losses – easily pinpoints energy deviations
an accelerated (nontechnical loss).
data analytics • Facilitate the use of geospatial intelligence to visualize grid operations
approach • Provides concurrency and high throughput for excessive data processing
• Reduce risk of • Unable insight decision making through predictive modeling and customer segmentation.
modernization efforts • Easy identification of technical faults in meters (technical loss).
• Establish a database of previous inspections and meter data to develop and train
analytical models.
• Grid-defection and churn-risk prediction
• Ease the strain being placed on traditional business intelligence (BI) and analytic solutions from the
exponential growth of data
• Enable technology • Optimize investment in core IT for new technologies that will allow for increased data insights
evolution initiatives to grow the business
11
Data Analytics plans
Since Data Analytics is a cross-functional enabler, it then concerns the whole organisation - the Data Analytics plans still require
deep-dive workshops across the organisation to understand the current impact of data analytics, data pain points, future vision of
data analytics and expectations of each Group.
The 1st phase on the data strategy relies heavily on having a skilled strong analytical team - the analytics strategy and tools need
to go hand-in-hand with enhancement of the business analytics team that is responsible for business analytics. This is a key
component of sustainable Business Analytics to create and foster a group of employees who have the skills to provide and maintain
the analytical applications needed by the business. There are many cases of successful analytical projects that have failed long-
term because no thought was given to how to sustain the work after the initial project work is completed. The enhancement of the
business analytics team is a program intended to brings expertise, best practices and implementations to meet the business
objectives. As the unit is expected to support business sustainability by providing the analytical skills needed, education and
training to business users as well alignment of services with the strategic goals of the organization it is critical that skilled
personnel are acquired.
12
Engineering Operations – data focused plans
GIS Data – using available grid topology data from GIS to efficiently gain new insights into the grid and achieve more
accurately targeted actions
Preparing GIS data to meet the demands of data analytics
Identifying the best use cases to generate the most value from GIS data
Incorporating GIS data into a variety of models to provide tools and insight which support complex network operations
Exploring the development of algorithms to automate aspects of engineering processes including network planning and
monitoring compliance with best practice
Enhancing of the Customer-to-Network Link
Data alignment between SAP PM , GIS and Forcelink Systems
Low Voltage Network Data – leveraging the new data generated from IoT deployment to the grid edge to gain deeper insights into the end to end
network and support a more extensive set of use cases
Extracting and utilizing data from smart meters in low voltage networks to allow more complete grid awareness and extend analytics benefits to the whole
grid
Expanding coverage of the grid through the deployment of sensors and power quality meters
Processing large quantities of real-time data captured in complex grid topologies
Extracting valuable performance insights from the grid edge and empowering control rooms to monitor low voltage networks
Outage Management – exploiting data from network systems and advanced metering infrastructure to improve responses to network faults and
minimize downtime
Developing software to compile and analyze data collected at multiple points in the network to identify problems or likely causes of faults
Filtering data from multiple grid domains, of varying quality, to ensure reliable analytics performance
Responding proactively to results to restore power quickly during an outage or make preventative repairs outside of routine maintenance
Developing software to ingest data and generate intuitive graphical output, providing clear insight for operations teams
Analytics on Plant Out of Service Duration
13
Engineering Services – data focused plans
Real-time Data – establishing a real-time data management and analysis framework to process data in the order of milliseconds to support daily
network activity - extending CP’s real-time data capacity and capability to allow instantaneous response from operations and maintenance
Developing data architecture capable of processing vast quantities of data generated in real time
Creating accessible dashboards and monitoring systems fed by real-time data for use by asset management and network control teams
Measuring the impact of real-time data for better grid performance
Defining the challenges and opportunities posed by gathering and processing data in high resolution for use in daily network activity
Grid Planning Data – identifying the optimal range of data sources and analytics models to support accuracy in long range grid planning and
investment decision making
Developing situational awareness of the grid and enhancing this by overcoming data quality issues including those in geospatial data
Developing a “digital twin” model for the connectivity and topology of a low voltage network
Identifying underperforming spots within the grid
Using data on consumption and grid capacity to inform efficient future development of the network
Producing models for likely network requirements which incorporate a vast number of variables
Developing projections for multiple energy scenarios while accounting for the potentially unpredictable uptake of new technologies
Increasing the depth of analytics to generate prescriptive insight into your grid planning strategy
Real-time Data – establishing a real-time data management and analysis framework to process data in the order of milliseconds to support daily network
activity - extending CP’s real-time data capacity and capability to allow instantaneous response from operations and maintenance
Developing data architecture capable of processing vast quantities of data generated in real time
Creating accessible dashboards and monitoring systems fed by real-time data for use by asset management and network control teams
Measuring the impact of real-time data for better grid performance
Defining the challenges and opportunities posed by gathering and processing data in high resolution for use in daily network activity
14
Metering Services – data focused plans
Smart Meter Data – extracting deep value from smart meter data whilst delivering a capability to work with other forms of data
Leveraging AMI to deliver advanced business insight to network operators
Building an infrastructure to manage smart meter data which is interoperable with a variety of sources of network data
Identifying the challenges of integrating and processing high data volumes
Prioritizing high-value use cases to add value for network operators today and in the future
Non-Technical Loss Use Case –combining granular grid and consumer data to isolate and efficiently eliminate non-technical losses
Optimizing revenue protection by locating non-technical losses and moving more quickly to curtail them
Using a combination of historical inspection results and smart meter data to train a supervised machine learning model to predict where losses are
located
Automatically extracting important features related to demand such as the identification of drops in consumption and estimating the amount of
energy which will be recovered to guide future inspections
Enabling a targeted approach to guide investigations into energy theft, protect revenue, and reduce bills for law-abiding customers
Debt Settlement – effectively processing data from a variety of energy market participants for a more accurate and timely debt settlement
process
Using data from each grid domain to make the debt settlement process more precise and quick to complete
Developing the infrastructure to access and prepare data streams from multiple organizations
Combining the relevant data and performing calculations more quickly to shorten the timetable for final settlement and move to more precise settlement
periods
Increasing the level of automation in the debt settlement process while improving accuracy and reducing costs for users
15
Finance – data focused plans
Accounting analytics
Accounting change scenario and impact analysis - Are we investing in the right opportunities based on capital and/or revenue, and how do our business
decisions affect our key value drivers?
Financial statement business analysis,
Accounting compliance analysis
16
H u m a n R e s o u r c e & Tr a n s f o r m a t i o n – d a t a f o c u s e d p l a n s
Active Performance Enhancement – identifying the data sources and analytics models that produce KPIs which maximize performance
efficiency and extend equipment lifespan
Business driven talent analytics and Integrated talent management work planning
Employee performance analytics - Using data from workforce performance productivity- Focused use of top performers, analyze the impact to
retention, and how effective is the structure management
Data privacy and compliance
Enhance employee utilization - resource optimization including resource utilization
17
Business Sustainability– data focused plans
Active Performance Enhancement – identifying the data sources and analytics models that produce KPIs which maximize performance
efficiency and extend equipment lifespan
Leveraging more accurate asset data to identify critical metrics, actively monitor asset life-cycle, and respond accordingly to fluctuations in
performance
Installing monitoring capabilities to generate precise data on more parameters pertaining to asset condition
Analysing data from a large sample of assets to minimize the impact of anomalous behavior on predictions
Identifying the key conditions and predictors which govern asset life-cycle, degradation, and failure
Moving from routine to targeted maintenance, informing changes to increase asset life-cycle, and optimizing grid upkeep
18
Enterprise Support – data analytics plans
Security/Intelligence Extension enhances traditional security solutions by analyzing all types and sources of under-
leveraged data
Enhanced Intelligence & Surveillance Insight - Find associations, uncover patterns and facts, maintain currency of information
Real-time Cyber Attack Prediction & Mitigation Crime prediction & protection - Discover new threats early, Detect known complex
threats, Take action in real-time
Reduce Customer Churn - Gather criminal evidence, prevent criminal activities, proactively apprehend criminals, customer
Retention
Cyber Security – identifying the points of vulnerability specific to big data infrastructure and determining appropriate prevention, detection, recovery and
response strategies to mitigate risks
Identifying, detecting and diagnosing threats to the integrity and availability of data including malicious data injections
Building methods for dynamic risk assessment, impact analysis and resilience
Managing threats arising from a combination of the human, physical and cyber aspects of systems
Balancing the need to integrate more systems with minimizing the attack surface for cyber threats
Evaluating new approaches to dealing with the inherent security weaknesses of legacy systems
Exploring vulnerabilities associated with the use of Machine Learning algorithms
Ensuring the protection of systems containing sensitive data to minimize business risk
19
Data Governance Implementation
The 1 Phase
1
ment ment
2
Incre Incre ment
3
Incre
Design
Operate
Phase 3, 4, 5
20
An Initial Data Governance Program Focused on Data Quality
Step 1-Organize Governance and Business Rules
Establish Governance Assess Business Impact Metadata Model for Data Governance
Key Tasks
· Measure information · Define roles and · Select process areas · Define data · Define business
· Validate in-scope KDEs &
management awareness responsibilities specification standards entity rules and
existing business rules
· Identify data users attribute definitions in
·Confirm executive sponsor · Select in-scope Key Data · Define data collection metadata repository
· Gather additional business
Elements (KDEs) · Identify and document standards
rules
· Select in-scope subject areas entities and KDEs · Define business
· Develop Data Governance · Define reporting rules for KDEs in
· Determine business impact
· Identify system Data Quality Strategy · Gather existing KDE standards metadata repository
of erroneous or missing data
authority business rules
· Prioritize KDEs by value
Outcomes
By completing Step 1, we will have a complete set of accepted data standards, definitions and business rules that will improve the productivity of the data quality project. We store data standards and
rules for governance within a metadata repository.
• Organize the Data Governance Management by establish the standards, definitions, and business rules for the Key Data Elements (KDEs)
• Select subject areas and agree upon the roles and responsibilities of data stewards
• Document business processes and rules thoroughly. Classify and prioritize KDEs. Ensure that metadata model is defined and ready to handle result sets from profiling
21
An Initial Data Governance Program Focused on Data Quality
Step 2 - Understand Historical Data Issues and Resolve Them
Establish Metrics Profile & Benchmark Process Investigation Process Re-Engineering Data Re- Data
Engineering Archiving
Key Tasks
• Integrate data profiling · Identify data for re- · Identify data for archiving
· Specify metrics categories for · Analyze data profiling · Formally capture erroneous
quality tracking methods into environment results engineering
processes
· Review and validate
· Conduct data profiling · Use profiling results as · Establish best practices archiving processes
· Document target metrics for · Prioritise key processes for
activities down columns, basis of root cause analysis to fix data
each KDE remediation
across tables and between · Put archiving processes
tables · Identify data and business · Recommend process · Standardize, correct, in place
· Establish baseline metrics for process issues match, link and enrich
improvements
benchmarking · Populate metadata repository data
Outcomes
By completing Step 2, Data Governance levels are measurable and will begin to increase by improving processes and fixing historical problems caused by poor Data Governance. The Data
Governance project will establish best practice re-usable services in data profiling, cleansing, and archiving. The result will be increased confidence of data within the organisation.
• Implement a Data Governance solution that will assess the data against the identified rules that were captured in Step 1.
• Report data quality assessment results against defined benchmarks in Data Governance
• Address data quality issues from the past and begin improving processes to prevent them from occurring in the future
• Define the data items that are candidates for archiving that no longer serve a useful purpose
22
An Initial Data Governance Program Focused on Data Quality
Step 3- Data Governance Monitoring and Reporting
Key Tasks
· Establish policies,
· Provide timely result · Introduce Root · Fix interfaces which · Implement · Monitor for data standard
processes, and people to
feedback to users and Cause Analysis as cause system and data communication plan compliance
address data issues
Stakeholders standard practice breaches
· Conduct management · Monitor for business rule
· Provide the means to
· Publish metrics and · Implement proactive · Document and prioritize (BPM’s) training compliance
investigate and remedy data
benchmarks approach to continuous interfaces requiring
issues
quality improvement remediation · Align performance · Monitor for data
objectives with management process
· Resolution can be done
· Ensure that new organisation goals usage compliance
across systems
interfaces adhere to
standards
Outcomes
The outcomes of Step 3 will be the continual monitoring and reporting of Data Governance metrics. Continual Data Governance improvement process will be established and the
importance of Data Governance will be embedded within the culture of the company.
• As part of the data remediation process, integrating and optimizing the processes and techniques defined in Step 2 into the overall data management framework is
key
• Establish monitoring activities so we review data for completeness and accuracy in an ongoing fashion
• Aim to enforce the use of standards and support ongoing data improvement and management processes for continuous improvement
23
Key Requirements
Resources required:
Personnel
Internal – Data Architecture, Data Modelers and Data Analysts
External – Data Management and Analytics expects
System
Data Quality tool,
Data Cleansing tool
Data Architecture tool
Business Process automation tool
24