Professional Documents
Culture Documents
Data Architecture Basics: An Illustrated Guide For Non-Technical Readers
Data Architecture Basics: An Illustrated Guide For Non-Technical Readers
GUIDEBOOK
An Illustrated Guide
for Non-Technical Readers
INTRODUCTION
WHY DATA ARCHITECTURE MATTERS
Data architecture is the foundation of every organization’s data strategy. Just like in a house or
office, the structure determines whether the inhabitants are able to perform their tasks. Hopefully,
it’s with efficiency and joy; but when it comes to data architecture, poor use of space, inconvenient
pathways, security, and clutter can make life difficult.
Designing data architecture is a valuable skill, but it’s not something that everyone is equipped
with, and that’s OK - not everyone needs to be an expert. But data architecture is not just something
for CIOs and data architects either; everyone at data-powered organizations can benefit from
understanding the ways data moves between teams and flows into data projects to yield insights
(as well as where it might get stuck along the way).
In order to ultimately democratize the use of data through a company, each team and role (whether
data architect, analyst, or data scientist) needs to work together. That becomes infinitely easier
when each role understands what the other is doing and how their work affects the whole of the
data machine.
Data architecture is a big umbrella term that encapsulates data storage, computational resources,
and everything in between. All the technology that supports the collection, processing, and
dashboarding of data is included in the architecture.
However, data architecture doesn’t just include objects; it also includes the operational discipline
surrounding data usage and compliance. Data architects can be said to act as data physicians,
diagnosing both the physical maladies in the data pipeline and the organizational habits that are
detrimental to the health of the system.
Translate Monitor
business needs backups and
to technological fault tolerance
solutions
Explore
competing
suppliers Adjust
operations as
the organization
scales
This can manifest as monitoring backups one day or translating jargon the next. Some examples
of data architects’ jobs include:
• Collaborate with a variety of teams (hardware, machine learning, robotics, software, user
experience, etc) to design and maintain an ideal data architecture.
• Exploring where and how process automation can best improve workflows.
• Getting proactive about data and service security by seeking out potential failure points and
attempting to mitigate risks.
COMPUTE RESOURCES
1. Scalability, that is, the ability to handle the
[kəmˈpjut ˈrisɔrsɪz]
storage of more and more data without
sacrificing performance of a system (and The processing capabilities of a
without causing costs to skyrocket). system that are available to perform
computational work (e.g. execute
2. Security, or the standards and
programs, carry out analysis, etc.).
technologies that protect data from
being destroyed or modified (whether
intentionally or not). DATA PARTITIONING
[ˈdætə pɑrˈtɪʃənɪŋ]
DATA REPLICATION
[ˈdeɪtə ˌrɛpləˈkeɪʃən]
A popular cluster-based open-source framework The process by which data is written to the
for distributed storage. Different data processing primary storage location and a replica or
engines or frameworks that can be used on backup location simultaneously.
top of Hadoop include (but are not limited to)
Hadoop MapReduce and Apache Spark.
To take an example that illustrates why scalability Let’s say the model works so well that Bob earns
matters in context, let’s say Bob’s Flower Company enough capital to open a brick-and-mortar
starts collecting data on what customers purchase, location. However, Bob’s data is stored locally,
when, and what online content they engage with. and his model is still run from the desktop, so
Bob creates a recommendation engine with this he can’t access the information in his new store.
information and offers targeted coupons to his Instead, he has to download the new customer
customers based on their preferences. information and add it manually to his local
model. Obviously, this is not an ideal situation
He stores this data on a local server and runs and one that is prone to user error.
the machine learning model from a desktop.
Engagement with the coupons works very well, and This may seem like an exaggeration, but enterprises
these customers, in turn, are more likely to log onto follow Bob’s architecture on a grander scale all the
Bob’s for spur-of-the-moment floral delivery needs. time. Unless organizations are able to anticipate
their future data needs, their data will become a
hindrance, not a boon. Thus, collaboration across
teams and workstreams is critical when designing
data architecture to help reveal as many areas for
improvement or threats as possible.
FACEBOOK AD PEONIES
20%
OFF ROSES
WEBSITE ROSES
JOHN
EMAIL COUPON PEONIES
ROB
SECURITY
MACHINE LEARNING
MODEL
GDPR
BACKUPS
To contextualize with another example, let’s say that Bob wants to more deeply understand what ads
his user Alice responds to, so that he can target her better in the future. We know that she generally likes
peonies, so Bob starts with three ads:
• ·1
·1
• ·2
·2
• ·3
·3
At this scale, it’s easy for Bob to say “the first option Looking at the architecture from the result, in
has peonies, the second daisies,” but with even a order to analyze the images, Bob would need
hundred customers, human analysis at this level to have access to structured data, in the form of
has a high time cost. It also ignores some of the tags associated with each image (e.g., “peony,”
image analysis deep learning can perform to or “city”). These would be generated from deep
extract context clues. So when Alice clicks on the learning image classification, which, depending
third ad the most, Bob could better understand on the size of the image database, can require an
that she might not like sunflowers, but that she architecture with a lot of computational power.
prefers the human element of that particular ad.
COMMUNICATION
GUIDE
Fault Tolerance is the ability to maintain usage of a system during a component failure. A system that has
High Availability is one that has long periods of uptime (and so, obviously, very little downtime).
PRIMARY SCHEDULER
STANDBY SCHEDULER
CLIENT
MANAGER
If an organization’s operations are critical to the health and wellbeing of users, it is obvious that there is
no room for error when it comes to either high availability or fault tolerance. But operations for consumer
and media products face incredible scrutiny, too. The threat of hemorrhaging users is aways business-
critical, and a good defense is driving user trust by creating a highly available, fault-tolerant system.
Data replication can help ensure high availability, and it can also help with fault tolerance. However,
running multiple databases with the exact same information can not only be expensive, but - perhaps
more critically problematic - complicated.
In a paired system, every time a new piece of data is added to the primary database, the same piece of
data needs to be placed in the backup; thus, each byte must be double stored (running at least double
the cost).
10 ©2020 Dataiku, Inc. | www.dataiku.com | [email protected] | @dataiku
More important than the added cost is the added intricacy, which must be carefully considered as it can
add some risk. Synchronous replication happens when writing data to primary storage and the backup
or replica simultaneously. Asynchronous replication means that data is copied to the backup or replica
after it is already written to the primary storage location. Choosing between the two means balancing
riskiness and speed (with synchronous replication being lower risk but slower, and asynchronous
replication being higher risk, but faster).
SYNCHRONOUS ASYNCHRONOUS
GET
RESPONSE
The risk stems from the fact that the system (or mapper) that copies data represents a dangerous single
point of failure. If the mapper goes offline temporarily, the backup data will become outdated, and it will
be challenging to determine which data needs to be updated.
As an example, let’s come back to Bob’s ecommerce site - let’s say he has implemented asynchronous
replication as a way to ensure high availability and fault tolerance. A user is trying to place an order, and
at that very moment, a bad thunderstorm has caused a failure in Bob’s system. The order has thus been
recorded in the primary storage system, but has not been replicated to the backup. The order thus does
not get placed, yet the customer’s card is charged as a result of the data writing to primary storage.
SCALE “OUT”
In a distributed system framework, data is: centralized ones (though it’s important to note that
they are not always faster, as speed also depends
1. Partitioned, meaning split apart into smaller on algorithms and types of calculations being
pieces and stored on different servers.
performed on the data), and they are less prone to
2. But it can also be Replicated, meaning that single points of failure. They also scale well with
those partitioned bits are also copied several
rapidly growing databases.
times. Read more about the intricacies of
replication in the sidebar on high availability
and fault tolerance. Where data is stored can also determine a lot about
how it can be used and ultimately how scalable
Since data is broken into small chunks, the system is. Thus, as always, it’s critical to select
computation is performed in small parts on each a storage system that works with an organization’s
impacted server, thus “sharing the load” and business needs. Data that must cross the Atlantic
improving overall computation accessibility and ocean with each transaction in order to be stored
speed. Distributed systems are often faster than will add some latency, so the question of response
C D
NAME MAPPER
D B
TASK TASK
A C
TRACKER TRACKER
There are many different types of distributed systems that can be implemented in an architecture.
Hadoop is very common, as it is cost-effective, incorporates redundancy to help with fault tolerance, can
handle a variety of data types, and is (relatively) fast.
Hadoop was designed to handle massive amounts of information efficiently. It is also relatively
inexpensive because the idea is that companies can leverage lots of cheap servers instead of
one giant, expensive server. However, it might not be right for a particular organization’s needs if:
• The technical team does not have Hadoop experience (and cannot hire for that experience)
• Performing any adjustments or maintenance on this type of system will be a challenge without
the requisite experience
The theorem states that no distributed data storage system can have more than two of the
following features:
1. Consistency - every read request returns the most recently written information (or error).
2. Availability - every read request receives a non-error response, but the information
returned may be old.
3. Partition Tolerance - even if messages between nodes fail, the system will sustain operations.
AVAILABILITY
TWO
Whichever two functionalities are prioritized will depend on the output, users, and data. For example, if
the output is a content recommendation engine, availability and partition tolerance matter more than
consistency; if advice is a few minutes stale, it will likely still hold. On the other hand, if users are doctors
analyzing medical records, consistency and availability are likely more important.
Nothing beyond physical sensors is actually able to function in real-time, but so long as communication
about the time-sensitive needs of the business is clear, then architectural priorities can respond.
MONITORING
EVALUATION MODELING
ACTION
RESPONSE SCORING TRAINING
DATA
UNDERSTANDING
SCORING/ BUSINESS
EVALUATION UNDERSTANDING
START
Cloud Architecture
PROS CONS
• Rapid scalability means the architecture • Forfeit of data custody means businesses
can keep up with the business must trust their cloud provider to protect
their data
• Global accessibility enables comparisons
between different markets
One critical part to scaling data architecture is making sure that end users (whether they are data
scientists, analysts, or line-of-business people) can interact successfully with the data pipeline.
Data Lineage
JAN
CLICKS WEBSITE
Clearly, there are some pretty fundamental In addition, data science, machine learning,
differences between data consumption for coders and AI platforms like Dataiku are useful because
vs. non-coders: they offload computation, meaning that the tool
inherently scales automatically as infrastructure
scales. Having to worry about scaling the front-end
Non-
Persona Coders tools as well as the back end can make questions
Coders
about data architecture much more complex.
Priorities - Legibility - Flexibility
- Automatic
- Adaptability
updates
Needs of coding
- Sleek user
environment
experience
AAA AUDIT
[ə ə eɪ] [ˈɔdɪt]
Stands for authentication, authorization, The ability to trace and review everything
and audit, which are the three pillars of data that’s been done within the system.
architecture security.
DATA LINEAGE
AUTHENTICATION
[ɔˌθɛntəˈkeɪʃən] [ˈdeɪtə ˈlɪniəʤ]
The security process by which a user or process Covers the entire path data takes from creation
confirms its identity. This can occur in one to storage to analysis. It also incorporates
location (as is the case with SSO) or through who “owns” what data at a given time,
multi-factor authentication. which is critical transparency for compliance
regulations.
AUTHORIZATION
USER PERMISSIONS
[ˌɔθərəˈzeɪʃən]
[ˈjuzər pərˈmɪʃənz]
The security process by which the system gives Enable users to work with a certain set of data.
a user or process the ability to read and write They come in three stages: Read, where a user
data or execute programs within certain parts can consume the data stored in the system;
of a system. Depending on the user’s clearance Write, where a user can modify data; and
level, this may represent a small section of data Execute, where a user can execute commands
relevant to their processes, or could include the and programs that can impact the database
ability to act as an admin and authorize other structure.
users.
Malicious actors could try to compromise the AUTHENTICATION: IS THE USER WHO (S)HE
system, or more likely, an unknowing user SAYS (S)HE IS?
might do so inadvertently, which could result in Authentication is the equivalent of ID verification
temporary loss of service or more permanent at the airport using a driver’s license or passport.
destruction. Another risk is that employees in The purpose is to ensure that the user is who
the organization see data or projects that they (s)he is declaring to be. In data architecture,
shouldn’t have access to - especially in the age authentication happens through something the
of data privacy, it’s critical to protect against user knows (i.e., a password) plus sometimes for
internal data leakage. Security protocols are not even more security, something (s)he has.
put in place to be annoying, but rather to protect
business users and the broader organization from The latter is known as Multi-Factor Authentication.
large data failures or leaks. Using this protocol, users must provide at least two
proofs that they are who they claim to be when
From a data architecture together in the same environment grows, being able to maintaining
perspective, there are three clear records of who adjusts what data is crucial.
basic ways of accessing and
modifying data, each of which
represents a different level In addition, the ability to perform data usage audits may be a
of permissions granted to
necessary part of system requirements depending on the type of data
the user or applications:
Read, Write, and Execute. used and the compliance standards to which an organization is held.
But even if it’s not a legal requirement, it can be helpful as part of good
These permissions are
clustered, so often a user data governance practice to consider implementing audit systems to
will have read and write monitor usage and help ensure security standards are being met.
permissions, but not execute,
or read and execute, but not
write. Understanding at this
granular level the ways users
engage with data is important
because it highlights the
AUTHENTICATION:
balance that needs to be
Admin checks the user’s key
struck between ease and
security.
AUTHORIZATION:
Admin unlocks the door
PROS
It’s worth noting that whatever audit data is being captured, it is best
practice to put it into a separate system immediately. That way, any
bad actors can't modify the evidence that they were making changes
to a system.
As with any complex system, what works for one company doesn’t necessarily work for another -
cookie-cutter layouts cannot support organizational goals. That said, it’s useful to visualize what a data
architecture might look like:
Users interact
via web browsers
or dashboards
r/w/x r/w/x
bit.ly/MLBASICS
Filter on Mr.
45,000+ 450+
ACTIVE USERS CUSTOMERS
Dataiku is the platform for Everyday AI, systemizing the use of data for exceptional
business results. Organizations that use Dataiku elevate their people (whether technical
and working in code or on the business side and low- or no-code) to extraordinary, arming
them with the ability to make better day-to-day decisions with data.