Hands-On Enterprise Application Development with Python: Design data-intensive Application with Python 3

Hands-On Enterprise Application Development with Python

Design data-intensive Application with Python 3

Saurabh Badhwar

BIRMINGHAM - MUMBAI

Hands-On Enterprise Application Development with Python

Copyright © 2018 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Aaron Lazar

Acquisition Editor: Chaitanya Nair

Content Development Editor: Manjusha Mantri

Technical Editor: Ashi Singh

Copy Editor: Safis Editing

Project Coordinator: Prajakta Naik

Proofreader: Safis Editing

Indexer: Mariammal Chettiyar

Graphics: Jisha Chirayil

Production Coordinator: Jyoti Chauhan

First edition: December 2018

Production reference: 1271218

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78953-236-4

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

Packt.com

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.Packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.Packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors

About the author

Saurabh Badhwar is a developer and open source enthusiast who is passionate about improving software performance and scalability. He has been actively contributing to Mozilla Servo and Fedora Project in the fields of release engineering, quality assurance, and community building. He has given talks at various community organized events and universities to onboard and engage new members in the community. Saurabh is currently exploring the microservices architecture and has given various talks on it. His most recent talk on the microservices architecture happened at DevconfCZ 2018. Currently, Saurabh works as an Associate Software Engineer, Performance and Scale Engineering at Red Hat.

About the reviewer

Zhuo Qingliang (also known online as KDr2) works at Beijing Paoding Tech (PAI), a start-up fintech company in China that is dedicated to improving the financial industry by using artificial intelligence technologies. He has over 10 years of experience in Linux, C, C++, Java, Python, and Perl development. He is interested in programming, consulting, and contributing to the open source community.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Hands-On Enterprise Application Development with Python

About Packt

Why subscribe?

Packt.com

Contributors

About the author

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Conventions used

Get in touch

Reviews

Using Python for Enterprise

Technical requirements

Recent developments in Python

Dropping backward compatibility

It's all Unicode

Support for type hinting

Where Python shines

The requirements of enterprise IT

Python in the enterprise ecosystem

Introducing BugZot – a RESTful bug tracker

Gathering requirements before development

Asking for the user requirements

Categorizing the requirements

Prioritizing the requirements

Generating the software requirement specification document

Summary

Questions

Further reading

Design Patterns – Making a Choice

Technical Requirements

Design patterns

Classification of design patterns

Defining the choice of design patterns

Object-oriented Python

The basic OOP principles

Encapsulation

Composition

Inheritance

Multiple inheritance in Python

Method resolution order in multiple inheritance

Utilizing mixins

Abstract base classes

Metaclasses

The Singleton pattern

The __call__ magic method

The Factory pattern

The Model-View-Controller pattern

Controller

Model

View

Summary

Questions

Building for Large-Scale Database Operations

Technical requirements

Database and object relational mappers

Setting up SQLAlchemy

Building optimal database models

Issues with our model definition

Optimizing our models

Taking advantage of indexes

Maintaining database consistency

Utilizing transactions to maintain consistency

Understanding lazy loading versus eager loading

Using relationships

Lazy loading

Eager loading

Optimizing data loading

Utilizing caching

Caching at the database level

Caching at the block level

Using user-level caching

Summary

Questions

Dealing with Concurrency

Technical requirements

The need for concurrency

Concurrency in GUI applications

Concurrency in enterprise applications

Concurrent programming with Python

Concurrency with multithreading

Thread synchronization

Re-entrant locks

Condition variables

Common pitfalls with multithreading

Race conditions

Deadlocks

The story of GIL

Concurrency with multiprocessing

Python multiprocessing module

Synchronizing processes

Summary

Questions

Building for Large-Scale Request Handling

Technical requirements

The problems of accommodating increased concurrency

The multiple options to scale up

Engineering the application for scalability

Controlling the concurrency

Using thread pools for handling incoming connections

Asynchronous programming with AsyncIO

AsyncIO terminology

Event loop

Co-routines

Tasks

Writing a simple Python AsyncIO program

Implementing a simple socket server with AsyncIO

Boosting the application concurrency

Running behind a reverse proxy

Improved security

Improved connection handling

Resource caching

Serving static resources

Summary

Questions

Example – Building BugZot

Technical requirements

Defining the requirements

Entering the development phase

Setting up the development environment

Setting up the database

Setting up the virtual environment

Structuring our project

Initializing the Flask project

Creating the configuration

Developing database models

Migrating the database models

Building the views

Developing the index view

Getting the index view to render

Building the user registration view

Deploying for concurrent access

Setting up Gunicorn

Setting up Nginx as reverse proxy

Establishing communication between Nginx and Gunicorn

Summary

Questions

Building Optimized Frontends

Technical requirements

The need for optimizing frontends

Components of an optimized frontend

What causes frontend issues

Optimizing the frontend

Optimizing resources

Fetching CSS in parallel by avoiding CSS imports

Bundling JavaScript

Utilizing client-side caching

Setting application-wide cache control

Setting request level cache control

Utilizing web storage

Working with local web storage

Working with session storage

Summary

Questions

Writing Testable Code

Technical requirements

The importance of testing

The different kinds of testing

Unit testing

Integration testing

Building an application with testing in mind

Test-driven development

Writing unit tests

Writing unit tests with Python unittest

Writing unit tests with pytest

Let's set up pytest

Writing our first test with pytest

Writing functional tests with pytest

Summary

Questions

Profiling Applications for Performance

Technical requirements

Behind the scenes of performance bottlenecks

Looking at the causes of performance bottlenecks

Probing an application for performance issues

Writing performance benchmarks

Writing our first benchmark

Writing an API benchmark

Doing component-level performance analysis

Measuring slow operations with timeit

Profiling with cProfile

Profiling for memory usage with memory_profiler

Collecting live performance data

Logging performance metrics

Avoiding performance bottlenecks

Summary

Questions

Securing Your Application

Technical requirements

Enterprise application security

The importance of enterprise security

Challenges in system security

Taking a look at the attack vectors

Security issues with native applications

Security issues with web applications

Security anti-patterns

Not filtering user input

Storing Sensitive Data Unencrypted

Ignoring bound-checking

Not keeping the libraries updated

Giving full privileges of the database to a single user

Improving your application's security

Summary

Questions

Further reading

Taking the Microservices Approach

Technical requirements

The shift toward microservices

Monolithic development model versus microservices

Advantages of the microservices architecture

Guidelines for microservice development

Service-level agreements in microservices

Building your first microservices application

The user microservice

The to-do manager service

Service discovery in microservices

Service registry inside microservices

Client-side service discovery

Server-side service discovery

API gateways in microservices

Asynchronous communication in microservices

Message queues for microservices communication

Summary

Questions

Further reading

Testing and Tracing in Microservices

Technical requirements

Testing in the microservices world

Unit testing in microservices

Functionality testing in microservices

Integration testing in microservices

End-to-end testing in microservices

Scalability testing

Challenges in microservices testing

Tracing requests inside microservices

The OpenTracing standard

Implementing tracing inside ToDo manager

Distributed tracing

Benefits of distributed tracing

Summary

Questions

Going Serverless

Technical requirements

The serverless approach to application development

Components of serverless architecture

Backend as a service

Function as a service

The restrictions on state management

Restrictions on execution times

Executing functions inside FaaS

API gateways in the serverless architecture

Understanding the execution of a serverless application

Cold-starting a function

Hot-starting a function

Building our first serverless application

A quick introduction to Apache OpenWhisk

Setting up the development environment

Building our configuration file

Integrating with the GitHub API

Getting the code ready to run with OpenWhisk

Taking the final steps toward deployment

Deploying to OpenWhisk

Understanding the execution of application Inside Openwhisk

Advantages of going serverless

Summary

Questions

Further reading

Deploying to the Cloud

Technical requirements

Deploying enterprise applications

Making a choice about the deployment strategy

The different deployment strategies

Recreated deployments

Rolling deployments

Blue/green deployments

Canary deployments

A/B deployments

Shadow deployments

Making a choice of infrastructure

The traditional infrastructure

Containerized approach toward application packaging

The move toward the cloud

The different types of cloud deployments

Private clouds

Public clouds

Hybrid clouds

Summary

Questions

Enterprise Application Integration and its Patterns

Technical requirements

The need for EAI

Point-to-point integration

Moving towards EAI

The traditional approach to EAI

The introduction of the ESB

Patterns in EAI

Integration patterns

Mediation pattern

Federation pattern

Access patterns

Asynchronous access patterns

Synchronous access patterns

Issues in EAI

Summary

Questions

Microservices and Enterprise Application Integration

Technical requirements

Microservices and the changing EAI landscape

Challenges of traditional EAI in microservices

Point-to-point integration of microservices

Integrating microservices using the ESB

Utilizing API gateways for the integration of microservices

Transformation of the ESB

Rethinking EAI in microservices

Summary

Questions

Assessment

Chapter 1

Answer 1

Answer 2

Answer 3

Answer 4

Answer 5

Chapter 2

Answer 1

Answer 2

Answer 3

Chapter 3

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 4

Answer 1

Answer 2

Answer 3

Answer 4

Answer 5

Chapter 5

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 6

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 7

Answer 1

Answer 2

Answer 3

Chapter 8

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 9

Answer 1

Answer 2

Answer 3

Answer 4

Answer 5

Chapter 10

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 11

Answer 1

Answer 2

Answer 3

Answer 4

Answer 5

Chapter 12

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 13

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 14

Answer 1

Answer 2

Answer 3

Answer 4

Chapter 15

Answer 1

Answer 2

Answer 3

Chapter 16

Answer 1

Answer 2

Answer 3

Other Books You May Enjoy

Leave a review – let other readers know what you think

Preface

Python is a dynamically typed, interpreted language that facilitates the rapid building of applications in various areas of software development, including AI, desktop, and web applications.

With recent advancements in the Python ecosystem and the availability of a huge number of libraries that support high re-usability and enable the compilation of modular code, Python can be used to build applications that can solve an organization's problems. These applications can be developed in short timeframes and, if developed carefully, can allow them to be scaled in a manner that solves the needs of an organization.

Python version 3.7 brings with it several improvements and new features that make application development a breeze. Along with new features, improvements in the language's performance make it a good choice for building scalable, mission-critical applications powered by a choice of different runtimes based on the organization's needs.

Who this book is for

An enterprise application is a critical application that aims to solve the specific business needs of an organization. The requirements of an enterprise application differ greatly from those that are usually needed by an individual. These applications are supposed to provide high performance and scalability to enable the increasing everyday needs of an organization.

Keeping this in mind, this book is intended for developers who have an intermediate knowledge of coding in Python and are willing to dive deeper into building applications that can be scaled up, based on the needs of the organization. The book provides several examples that can be executed on Python 3.7 running on Linux-based distributions, but which also work on other operating systems.

To make the best use of this book, you must have a basic understanding of fundamental operating system concepts, such as process management and multithreading. Beyond this, a basic working knowledge of database systems may be beneficial, although not mandatory.

Developers who are familiar with the different aspects of building applications in Python can get to learn about the tools and techniques that can help them build scalable applications and provide them with an idea of the enterprise application development approach.

What this book covers

This book contains 16 chapters organized into roughly 4 parts: an introduction to Python in enterprises; the testing, performance, and security of enterprise applications; the shift toward microservices; and enterprise application integration.

The first part covers the utilization of Python in enterprise application development and deals with the various aspects related to building a performant and scalable application with Python. This first part consists of seven chapters.

Chapter 1, Using Python for Enterprise Application Development, introduces the capabilities of Python that make it a rich language when it comes to developing enterprise applications and provides information about the changes that have happened in the Python ecosystem and how they have helped in boosting the utilization of Python within the community. 

Chapter 2, Design Patterns – Making a Choice, explains the importance of design patterns in the development of applications and covers some of the commonly used design patterns in application development and how they can be implemented in Python. This chapter also provides guidance on how to choose which design patterns to use while developing the application.

Chapter 3, Building for Large-Scale Data Operations, covers how to build enterprise applications that can handle a large number of concurrent data operations while maintaining the consistency of data. This chapter takes a look at how the data models are developed to allow for scalability and performance and then goes on to provide information about how to make database operations efficient.

Chapter 4, Dealing with Concurrency, introduces us to the different ways in which we can improve the ability of our application to deal with concurrent workloads. For this, the chapter introduces us to various techniques in Python related to the implementation of multiprocessing and multithreading and how we can use them to boost the concurrency-managing capability of the application.

Chapter 5, Building for Large-Scale Request Handling, introduces us to the ways in which we can make our application scale well when it comes to a high number of concurrent users while keeping the response times adequate. We will also cover the different techniques related to maximizing the output from multithreading, utilizing asynchronous operations, and so on.

Chapter 6, Building BugZot, makes use of what we have learned in the preceding chapters to come up with an example application known as BugZot, which provides our dummy organization, Omega Corporation, with a bug tracking system. While building this application, we get to implement the techniques related to building efficient database models, implementing concurrency, and handling the uploading of large files.

Chapter 7, Building Optimized Frontends, takes us through the importance of building application frontends that are responsive and light on resources. This chapter covers various techniques related to the optimization of web-based frontends and how to provide an experience that keeps the user engaged with the application.

With this, we enter the second part of the book, which consists of three chapters aimed at improving the delivery aspects of the application.

Chapter 8, Writing Testable Code, focuses on the points related to the importance of writing code that can be tested easily and how it helps in delivering applications that are stable and robust when deployed in production. This chapter introduces us to a Python-based testing framework that can be used to write unit tests and integration tests and also covers the aspects of automated testing of the code.

Chapter 9, Profiling Application for Performance, introduces us to the tools and techniques that we, as developers, can use to build performance profiling into the core of the application to understand the performance implications of running the applications and allow us to uncover bottlenecks in production.

Chapter 10, Securing Your Applications, covers the different attack vectors that are used by attackers to compromise enterprise data security and then helps us to understand the basics of how we can avoid introducing vulnerabilities into our applications.

The third part of the book includes the following chapters that cover the architecture of application development.

Chapter 11, Taking the Microservices Approach, takes us on a journey of developing applications based on microservice architecture. In this chapter, we will take a look at what exactly microservice architecture is and the advantages it provides. We will then move on to understanding the different components that power microservice architecture and conclude by building a small application based on microservice principles.

Chapter 12, Testing and Tracing in Microservices, goes into how the move to microservices has changed the way we approach application testing, which now comprises the interaction of several small services instead of large components. We take a look at additional steps that might be required in making sure the applications work in the manner they are expected to, before we dive into another concept that allows us to trace the flow of the request in our application as it passes through different services.

Chapter 13, Going Serverless, introduces us to yet another method of building our applications where we, as developers, do not need to care about where the application will run and how it will scale. In this chapter, we will learn about how we can divide our application into functions that can run without our supervision and execute based on a set of events.

Chapter 14, Deploying to the Cloud, introduces us to the different cloud deployment strategies and how they can help us while also providing us with guidance about which type of cloud deployment to choose, based on the requirements of the organization.

The fourth and final part of the book includes the following chapters, which introduce us to the need to integrate different applications inside the enterprise.

Chapter 15, Enterprise Application Integration and their Patterns, takes us through an introduction to why enterprise application integration is necessary and how it was achieved in days gone by, before diving into the concepts of using middleware and an enterprise service bus. This chapter concludes with a discussion of various EAI patterns that are in use.

Chapter 16, Microservices and Enterprise Application Integration, takes us through the differences that the move toward microservices has introduced to the EAI landscape. In this chapter, we will take a look at how the modern concepts of microservices have changed how application integration was achieved in the past and how we can plan ahead for integrating different applications in this modern age. 

To get the most out of this book

Other than being familiar with programming in general, no particular specialist knowledge is expected in order to be able to take advantage of this book.

Odoo is built using Python, so it is a good idea to have a sound knowledge of the language. We also choose to run Odoo in an Ubuntu host (a popular cloud hosting option) and will do some work on the command line, so some familiarity will be beneficial.

To get the most out of this book, we recommend that you find complementary readings on the Python programming language, Ubuntu/Debian Linux operating system, and the PostgreSQL database.

Although we will run Odoo in an Ubuntu host, we will also provide guidance on how to set up our development environment in a Windows system with the help of VirtualBox. Of course, working from an Ubuntu/Debian system is also a possibility.

All the required software is freely available, and instructions on where to find it will be given.

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packt.com.

Select the SUPPORT tab.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Hands-On-Enterprise-Application-Development-with-Python. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: Beyond these three packages, the reader will also require the sqlalchemy package, which provides the ORM we will be using throughout the chapter, and psycopg2, which provides postgres database bindings to allow sqlalchemy to connect to postgres.

A block of code is set as follows:

username = request.args.get('username')

email = request.args.get('email')

password = request.args.get('password')

user_record = User(username=username, email=email, password=password)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

username = request.args.get('username')

email = request.args.get('email')

password = request.args.get('password')

user_record = User(username=username, email=email, password=password)

Any command-line input or output is written as follows:

mkdir ch3 && cd ch3

virtualenv --python=python3 .

source bin/activate

pip install sqlalchemy psycopg2

Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: In the web client, access the Apps top menu and select the Update Apps List menu option.

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit authors.packt.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

Using Python for Enterprise

Python has been around in the world of programming for more than two decades now, and over the years, the language has seen a number of refinements, a growing community, and a lot of production-ready and well-supported libraries. But is Python ready to make a dent in the world of enterprise application development, which has been long dominated by the likes of C++, Java, and .NET, the so-called enterprise-grade languages?

Over the course of this chapter, we will see how Python has evolved over the years and how it is ready to become a serious competitor in the world of enterprise application development.

This chapter will cover the following topics:

Recent developments in Python to enable its growth in enterprise application development

The special use cases where Python shines

The differences between enterprise and general-purpose software

The requirements for developing an enterprise application

Technical requirements

The code listings in this book can be found under chapter01 directory at https://github.com/PacktPublishing/Hands-On-Enterprise-Application-Development-with-Python.

The code samples can be cloned by running the following command:

git clone https://github.com/PacktPublishing/Hands-On-Enterprise-Application-Development-with-Python

The instructions to run the code can be found under the README file in the individual chapter directories.

The code has been tested to run on a system that is running Fedora 28 and Python version 3.6.5, but it should be able to run on any system running Python 3.6.5.

Recent developments in Python

Python is a dynamically typed, interpreted language that was initially well suited for scripting tasks that are boring and repetitive day-to-day tasks. But as the years progressed, the language gained a number of new features and the huge backing of its community, which propelled its development to make it a language that is now well suited to performing tasks that range from very simple applications, such as web scraping, to analyzing large amounts of data for training machine learning models that themselves are written in Python. Let's take a look at some of the major things that have changed over the years and see what the latest release of Python, Python 3, brings to the table.

Dropping backward compatibility

Python as a language has evolved a lot over the years, but despite this fact, a program written in Python 1.0 will still be able to run in Python 2.7, which is a version that was released 19 years after Python 1.0.

Though a great benefit for the developers of Python applications, this backward compatibility of the language is also a major hurdle in the growth and development of major improvements in the language specification, since a great amount of the older code base will break if major changes are made to the language specification.

With the release of Python 3, this chain of backward compatibility was broken. The language in version 3 dropped the support for programs that were written in earlier versions in favor of allowing a larger set of long-overdue improvements to the language. However, this decision disappointed quite a lot of developers in the community.

It's all Unicode

In the days of Python 2, the text data type str was used to support ASCII data, and for Unicode data, the language provided a unicode data type. When someone wanted to deal with a particular encoding, they took a string and encoded it into the required encoding scheme.

Also, the language inherently supported an implicit conversion of the string type to the unicode type. This is shown in the following code snippet:

str1 = 'Hello'

type(str1)        # type(str1) => 'str'

str2 = u'World'

type(str2)        # type(str2) => 'unicode'

str3 = str1 + str2

type(str3)        # type(str3) => 'unicode'

This used to work, because here, Python would implicitly decode the byte string str1 into Unicode using the default encoding and then perform a concatenation. One thing to note here is that if this str1 string contained any non-ASCII characters, then this concatenation would have failed in Python, raising a UnicodeDecodeError.

With the arrival of Python 3, the data types that dealt with text changed. Now, the default data type str which was used to store text supports Unicode. With this, Python 3 also introduced a binary data type, called bytes, which can be used to store binary data. These two types, str and bytes, are incompatible and no implicit conversion between them will happen, and any attempt to do so will give rise to TypeError, as shown in the following code:

str1 = 'I am a unicode string'

type(str1) # type(str1) => 'str'

str2 = bAnd I can't be concatenated to a byte string

type(str2) # type(str2) => 'bytes'

str3 = str1 + str2

-----------------------------------------------------------

Traceback (most recent call last):

File , line 1, in

TypeError: can't concat str to bytes

As we can see, an attempt to concatenate a unicode type string with a byte type string failed with TypeError. Although an implicit conversion of a string to a byte or a byte to a string is not possible, we do have methods that allow us to encode a string into a bytes type and decode a bytes type to a string. Look at the following code:

str1 = '₹100'

str1.encode('utf-8')

#b'\xe2\x82\xb9100'

b'\xe2\x82\xb9100'.decode('utf-8')

# '₹100'

This clear distinction between a string type and binary type with restrictions on implicit conversion allows for more robust code and fewer errors. But these changes also mean that any code that used to deal with the handling of Unicode in Python 2 will need to be rewritten in Python 3 because of the backward incompatibility.

Here, you should focus on the encoding and decoding format used to convert string to bytes and vice versa. Choosing a different formatting for encoding and decoding can result in the loss of important information, and can result in corrupt data.

Support for type hinting

Python is a dynamically typed language, and hence the type of a variable is evaluated at runtime by the interpreter once a value has been assigned to the variable, as shown in the following code:

a = 10

type(a)        # type(a) => 'int'

a = Joe     

type(a)        # type(a) => 'str'

Though dynamic interpretation of the type of a variable can be handy while writing small programs where the code base can be easily tracked, the feature of the language can also become a big problem when working with very large code bases, which spawn a lot of modules, and where keeping track of the type of a particular variable can become a challenge and silly mistakes related to the use of incompatible types can happen easily. Look at the following code:

def get_name_string(name):

    return name['first_name'] + name['last_name']

username = Joe Cruze

print(get_name_string(username))

Let's see what happens if we try to execute the preceding program:

Traceback (most recent call last):

  File , line 1, in

  File , line 2, in get_name_string

TypeError: string indices must be integers

The program exited with a TypeError because we passed a string type to the get_name_string() method and then tried to access the keys inside a string, which is not the correct solution.

With the release of Python 3.5, the community added support for type hinting that was built into the language. This was not an effort to enforce a method, but was rather provided to support the users who may want to use modules that can catch errors that are related to a variable changing its type in between the execution flow.

The general syntax to mark the type of a variable is as follows:

: =

To mark the types in a method, the following syntax can be used:

def method_name(parameter: type) -> return_type:

    # method body

One of the examples of how to use type hinting in the program code is shown in the following code:

from typing import Dict

def get_name_string(name: Dict[str, str]) -> str:

    return name['first_name'] + name['last_name']

username = Joe Cruze