Hands-On Python for DevOps: Leverage Python's native libraries to streamline your workflow and save time with automation

Hands-On Python for DevOps

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Preet Ahuja

Publishing Product Manager: Suwarna Rajput

Book Project Manager: Ashwin Kharwa

Senior Editor: Mohd Hammad

Technical Editor: Nithik Cheruvakodan

Copy Editor: Safis Editing

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Ponraj Dhandapani

DevRel Marketing Coordinator: Rohan Dobhal

First published: March 2024

Production reference: 1160224

Published by Packt Publishing Ltd.

Grosvenor House
11 St Paul’s Square
Birmingham
B3 1RB, UK

ISBN 978-1-83508-116-7

www.packtpub.com

To my parents, who have always supported my choices and encouraged me to become a better person. To all my friends, peers, and colleagues, each of you has meant the world to me and I cannot begin to describe how you have shaped my life.

Contributors

About the author

Ankur Roy is a Solutions Architect at Online Partner AB in Stockholm, Sweden. Prior to this, he worked as a Software Engineer at Genese Solution in Kathmandu, Nepal. His areas of expertise include cloud-based solutions and workloads in a diverse range of fields such as development, DevOps, and security, among others. Ankur is an avid blogger, podcaster, content creator, and contributing member of the Python, DevOps, and cloud computing community. He has completed all the available certifications in Google Cloud and several others in AWS and Azure as well. Moreover, he is an AWS Community Builder.

I want to thank the entire Packt Publishing team for keeping me on track and focused on this book. I would also like to thank every person I told about writing this book for tolerating my exuberance for it.

About the reviewers

Shishir Subedi, an electronics and communications engineering graduate from the Institute of Engineering, Pulchowk Campus, discovered his passion for data science during his academic journey. Transitioning from a full stack Python developer to a part-time instructor at Ambikeshowri Campus, he focused on data mining and artificial intelligence, bridging theory and practice.

Currently a senior software engineer at Genese Solution, he leverages advanced language models to address financial technology challenges. Beyond his role, he contributes to education through training sessions for students and teachers, showcasing his commitment to technical proficiency and educational advancement.

Sagar Budhathoki, a dedicated Python/DevOps engineer, is recognized for his hands-on expertise in Python frameworks, system programming, and cloud computing. With a focus on automating and optimizing mission-critical deployments in AWS, Sagar leverages configuration management, CI/CD, and DevOps processes. His skills extend to Kubernetes deployment, OpenVPN configurations, and cybersecurity. As an AI/ML enthusiast, Sagar brings a comprehensive approach to technology, ensuring efficient, secure, and scalable solutions in the realm of DevOps.

Table of Contents

Preface

Part 1: Introduction to DevOps and role of Python in DevOps

1

Introducing DevOps Principles

Exploring automation

Automation and how it relates to the world

How automation evolves from the perspective of an operations engineer

Understanding logging and monitoring

Logging

Monitoring

Alerts

Incident and event response

How to respond to an incident (in life and DevOps)

Site reliability engineering

Incident response teams

Post-mortems

Understanding high availability

SLIs, SLOs, and SLAs

RTOs and RPOs

Error budgets

How to automate for high availability?

Delving into infrastructure as a code

Pseudocode

Summary

2

Talking about Python

Python 101

Beautiful-ugly/explicit-implicit

Simple-complex-complicated

Flat-nested/sparse-dense

Readability-special cases-practicality-purity-errors

Ambiguity/one way/Dutch

Now or never

Hard-bad/easy-good

Namespaces

What Python offers DevOps

Operating systems

Containerization

Microservices

A couple of simple DevOps tasks in Python

Automated shutdown of a server

Autopull a list of Docker images

Summary

3

The Simplest Ways to Start Using DevOps in Python Immediately

Technical requirements

Introducing API calls

Exercise 1 – calling a Hugging Face Transformer API

Exercise 2 – creating and releasing an API for consumption

Networking

Exercise 1 – using Scapy to sniff packets and visualize packet size over time

Exercise 2 – generating a routing table for your device

Summary

4

Provisioning Resources

Technical requirements

Python SDKs (and why everyone uses them)

Creating an AWS EC2 instance with Python’s boto3 library

Scaling and autoscaling

Manual scaling with Python

Autoscaling with Python based on a trigger

Containers and where Python fits in with containers

Simplifying Docker administration with Python

Managing Kubernetes with Python

Summary

Part 2: Sample Implementations of Python in DevOps

5

Manipulating Resources

Technical requirements

Event-based resource adjustment

Edge location-based resource sharing

Testing features on a subset of users

Analyzing data

Analysis of live data

Analysis of historical data

Refactoring legacy applications

Optimize

Refactor

Restart

Summary

6

Security and DevSecOps with Python

Technical requirements

Securing API keys and passwords

Store environment variables

Extract and obfuscate PII

Validating and verifying container images with Binary Authorization

Incident monitoring and response

Running runbooks

Pattern analysis of monitored logs

Summary

7

Automating Tasks

Automating server maintenance and patching

Sample 1: Running fleet maintenance on multiple instance fleets at once

Sample 2: Centralizing OS patching for critical updates

Automating container creation

Sample 1: Creating containers based on a list of requirements

Sample 2: Spinning up Kubernetes clusters

Automated launching of playbooks based on parameters

Summary

8

Understanding Event-Driven Architecture

Technical requirements

Introducing Pub/Sub and employing Kafka with Python using the confluent-kafka library

Understanding the importance of events and consequences

Exploring loosely coupled architecture

Killing your monolith with the strangler fig

Summary

9

Using Python for CI/CD Pipelines

Technical requirements

The origins and philosophy of CI/CD

Scene 1 – continuous integration

Scene 2 – continuous delivery

Scene 3 – continuous deployment

Python CI/CD essentials – automating a basic task

Working with devs and infrastructure to deliver your product

Performing rollback

Summary

Part 3: Let’s Go Further, Let’s Build Bigger

10

Common DevOps Use Cases in Some of the Biggest Companies in the World

AWS use case – Samsung electronics

Scenario

Brainstorming

Solution

Azure Use Case – Intertech

Scenario

Brainstorming

Solution

Google Cloud use case – MLB and AFL

Scenario

Brainstorming

Solution

Summary

11

MLOps and DataOps

Technical requirements

How MLOps and DataOps differ from regular DevOps

DataOps use case – JSON concatenation

MLOps use case – overclocking a GPU

Dealing with velocity, volume, and variety

Volume

Velocity

Variety

The Ops behind ChatGPT

Summary

12

How Python Integrates with IaC Concepts

Technical requirements

Automation and customization with Python’s Salt library

How Ansible works and the Python code behind it

Automate the automation of IaC with Python

Summary

13

The Tools to Take Your DevOps to the Next Level

Technical requirements

Advanced automation tools

Advanced monitoring tools

Advanced event response strategies

Summary

Index

Other Books You May Enjoy

Preface

Welcome to this book! Let’s talk about the content of this book and what you will learn from it. This book is about two things: DevOps and Python. It is about how these two entities, philosophies, frameworks, or whatever you would like to call them interact with each other.

This book will help you understand Python on a technical level, but also on a conceptual level, including what distinguishes Python from a lot of other languages and what makes it so popular among programmers and others who provide IT solutions.

At the same time, it will give you perspective on how important and useful DevOps is in modern IT infrastructure and how you can implement the concepts of DevOps using Python.

You will learn how to make the hard stuff easy and how to solve problems in a consistent and sustainable way. You will also learn how to insert bits of Python code into your workload to smoothen your problem-solving process.

This book will go beyond just some technical descriptions and processes and will help you make your workflow and work process even better regardless of the tools you are using.

Who this book is for

If you are even remotely concerned with DevOps or developing, you will find this book useful. But there are a few specific personas who may particularly find this book useful:

Developers looking to explore DevOps: Since this is a book that uses a lot of code for DevOps, it is perfect for developers who may want to explore DevOps

DevOps engineers learning Python: This book will help DevOps engineers who are learning Python and may want to try implementing some Python solutions in DevOps

People who like finding solutions: If you’re someone who wants to find solutions to IT problems and don’t have a specific job title, but have a job to do, this book is for you

What this book covers

Chapter 1

, Introducing DevOps Principles, will help you understand the concepts behind DevOps and how they are important in improving the productivity of your workload.

Chapter 2

, Talking about Python, covers the core philosophical principles behind DevOps and how these principles define the approach that you take toward creating a solution.

Chapter 3

, The Simplest Ways to Start Using DevOps in Python Immediately, provides a quick look at Python and the principles behind it, along with how these principles align with the principles of DevOps.

Chapter 4

, Provisioning Resources, explores the easiest ways to use Python such that it could enhance your DevOps workload.

Chapter 5

, Manipulating Resources, covers using Python as a means to provision resources in a sustainable and accurate way for your DevOps workload.

Chapter 6

, Security and DevSecOps with Python, looks at modifying resources that already exist using Python in order to automate updates and mass modify replicated resources.

Chapter 7

, Automating Tasks, explores using Python to automate common DevOps tasks and increase productivity for users by saving time on repetitive tasks.

Chapter 8

, Understanding Event-Driven Architecture, covers using Python as a way to connect different systems to system architectures using event-driven concepts.

Chapter 9

, Using Python for CI/CD Pipelines, looks at using Python for the most common DevOps task of Continuous Integration/Continuous Deployment (CI/CD) and enhancing these CI/CD pipelines.

Chapter 10

, Common DevOps Use Cases in Some of the Biggest Companies in the World, looks at Python in DevOps use cases in the context of some of the biggest companies and workloads provided by the major cloud platforms.

Chapter 11

, MLOps and DataOps, provides a look at the machine learning and big data niches of DevOps and how Python can help enhance these workloads.

Chapter 12

, How Python Integrates with IaC Concepts, explores how Python libraries and frameworks are used to provision resources using infrastructure as code to build and modify DevOps workloads in a standardized way.

Chapter 13

, The Tools to Take Your DevOps to the Next Level, looks at advanced DevOps concepts and tools and how they can be integrated into your workload.

To get the most out of this book

Often in this book, we cover tools and examples of how to use them to increase the productivity of your DevOps workload. You will need at least the version of Python mentioned here to use all the features described in the book. Most of the tasks done on one cloud platform can be done on equivalent services on other platforms.

For the cloud platforms, you will need to set up accounts and billing with the respective services.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Hands-On-Python-for-DevOps

. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/

. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: If you refer to the following diagram, the packet sizes are stored in the packet_sizes array and the timestamps of the packet are stored in the timestamps variable.

A block of code is set as follows:

def packet_handler(packet):print(packet)packet_sizes.append(len(packet))timestamps.append(packet.time)

Any command-line input or output is written as follows:

pip install sphinx

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: With reference to the preceding figure, when you click the Run button shown at the top, you’ll launch a Flask server (a URL that will return some sort of answer when it is called).

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected]

and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata

and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected]

with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com

.

Share Your Thoughts

Once you’ve read Hands-on Python for DevOps, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback

.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781835081167

Submit your proof of purchase

That’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Introduction to DevOps and role of Python in DevOps

This part will cover the basics of DevOps and Python and their relationship. It will also cover a few tricks and tips that could enhance your DevOps workload.

This part has the following chapters:

Chapter 1

, Introducing DevOps Principles

Chapter 2

, Talking about Python

Chapter 3

, The Simplest Ways to Start Using DevOps in Python Immediately

Chapter 4

, Provisioning Resources

1

Introducing DevOps Principles

Obey the principles without being bound by them.

– Bruce Lee

DevOps has numerous definitions, most of which are focused on culture and procedure. If you’ve gotten to the point where you have purchased this book as a part of your journey in the DevOps field, you have probably heard at least about 100 of these definitions. Since this is a book that focuses more on the hands-on, on-the-ground aspect of DevOps, we’ll keep those abstractions and definitions to a minimum, or rather, explain them through actions rather than words whenever possible.

However, since this is a DevOps book, I am obliged to take a shot at this:

DevOps is a series of principles and practices that aims to set a culture that supports the automation of repetitive work and continuous delivery of a product while integrating the software development and IT operation aspects of product delivery.

Not bad. It’s probably incomplete, but that’s the nature of the beast, and that is perhaps what makes this definition somewhat appropriate. Any DevOps engineer would tell you that the work is never complete. Its principles are similar in many ways to the Japanese philosophy of Ikigai. It gives the engineers a purpose; an avenue for improvement on their systems which gives them the same thrill as a swordsman honing their skills or an artist painting their masterpiece. Satisfied, yet unsatisfied at the same time. Zen.

Philosophical musings aside, I believe DevOps principles are critical to any modern software team. To work on such teams, it is better to start with the principles as they help explain a lot of how the tools used in DevOps were shaped, how and why software teams are constructed the way they are, and to facilitate DevOps principles. If I had to sum it