Ansible Cheat Sheet
Ansible Cheat Sheet
Release 1
Dan Poirier
1 Ansible 3
2 AWS 23
3 Bootstrap 25
4 Debian 27
5 Diet 29
6 Django 35
7 Elasticsearch 81
8 Elixir 87
9 Git 93
10 Google APIs 99
11 Haskell 101
12 i3 105
13 IPv6 107
14 Javascript 109
15 LXDE 115
17 MPD 119
19 NPM 123
20 OpenSSL 125
i
22 Postfix 131
23 Postgres 133
24 Python 139
25 Raspberry Pi 151
26 reStructuredText 157
28 Tmux 163
29 Video 165
30 Virtualbox 167
31 YAML 169
ii
Dan’s Cheat Sheets Documentation, Release 1
Contents:
Contents 1
Dan’s Cheat Sheets Documentation, Release 1
2 Contents
CHAPTER 1
Ansible
This is growing into a minimal Ansible reference of sorts, since Ansible’s own docs have nothing like a reference.
• Ansible.
• list of keys that common playbook objects can take.
• Release tarballs
• Ansible documentation for older releases
Blocks
• Blocks doc
• A blog post about blocks
• Blog post with examples
Blocks can be used anywhere a task can (mostly?). They allow applying task keys to a group of tasks without having
to repeat them over and over. They also provide a form of error handling.
Syntax:
block:
- <task1>
- <task2>
when: <condition>
become: true
become_user: <username>
....
[rescue:
- debug: msg="This task runs if there's an error in the block"
- <task2>
...
always:
- debug: msg="I always run"
... more tasks ..
]
3
Dan’s Cheat Sheets Documentation, Release 1
Conditionals
doc
Conditional tasks
See Task.
Configuration
Configuration file
hostfile
This is the default location of the inventory file, script, or directory that Ansible will use to determine
what hosts it has available to talk to:
hostfile = /etc/ansible/hosts
roles_path
The roles path indicate additional directories beyond the ‘roles/’ subdirectory of a playbook project to
search to find Ansible roles. For instance, if there was a source control repository of common roles
and a different repository of playbooks, you might choose to establish a convention to checkout roles in
/opt/mysite/roles like so:
roles_path = /opt/mysite/roles
Additional paths can be provided separated by colon characters, in the same way as other pathstrings:
roles_path = /opt/mysite/roles:/opt/othersite/roles
Roles will be first searched for in the playbook directory. Should a role not be found, it will indicate all
the possible paths that were searched.
4 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Inventory
Inventory directory
Inventory file
Default /etc/ansible/hosts
Change set ANSIBLE_HOSTS in environment
ansible-playbook -i <inventoryfile> ...
set hostfile in configuration
Syntax .ini file, except initial lines don’t need to be in a section
The inventory file is basically a list of hostnames or IP addresses, one per line. Can include port with
hostname:port or address:port.
Ranges: Including [m:n] in a line will repeat the line for every value from m through n. m and n can be numbers or
letters:
[mygroup]
host[1:25]
Host Variables: Can specify per-host options after hostname on the same line. E.g.:
jumper ansible_ssh_port=5555 ansible_ssh_host=192.168.1.50
[group3:children]
group1
group2
[group3:vars]
group3_var1=27
group3_var2="Hello, World"
1.4. Inventory 5
Dan’s Cheat Sheets Documentation, Release 1
Invoking
Ad-hoc
To run an ad-hoc command, use ansible. (But you almost always will want to run a playbook; see below.)
Examples of ad-hoc commands:
$ ansible all -m ping
# as bruce
$ ansible all -m ping -u bruce
# as bruce, sudoing to root
$ ansible all -m ping -u bruce --sudo
# as bruce, sudoing to batman
$ ansible all -m ping -u bruce --sdo --sudo-user batman
$ ansible all -a "/bin/echo hello"
Help:
Usage: ansible <host-pattern> [options]
Options:
-a MODULE_ARGS, --args=MODULE_ARGS
module arguments
-k, --ask-pass ask for SSH password
--ask-su-pass ask for su password
-K, --ask-sudo-pass ask for sudo password
--ask-vault-pass ask for vault password
-B SECONDS, --background=SECONDS
run asynchronously, failing after X seconds
(default=N/A)
-C, --check don't make any changes; instead, try to predict some
of the changes that may occur
-c CONNECTION, --connection=CONNECTION
connection type to use (default=smart)
-f FORKS, --forks=FORKS
specify number of parallel processes to use
(default=5)
-h, --help show this help message and exit
-i INVENTORY, --inventory-file=INVENTORY
specify inventory host file
(default=/etc/ansible/hosts)
-l SUBSET, --limit=SUBSET
further limit selected hosts to an additional pattern
--list-hosts outputs a list of matching hosts; does not execute
anything else
-m MODULE_NAME, --module-name=MODULE_NAME
module name to execute (default=command)
-M MODULE_PATH, --module-path=MODULE_PATH
specify path(s) to module library
(default=/usr/share/ansible/)
-o, --one-line condense output
-P POLL_INTERVAL, --poll=POLL_INTERVAL
set the poll interval if using -B (default=15)
--private-key=PRIVATE_KEY_FILE
use this file to authenticate the connection
-S, --su run operations with su
-R SU_USER, --su-user=SU_USER
6 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Playbooks
Options:
--ask-become-pass ask for privilege escalation password
-k, --ask-pass ask for connection password
--ask-su-pass ask for su password (deprecated, use become)
-K, --ask-sudo-pass ask for sudo password (deprecated, use become)
--ask-vault-pass ask for vault password
-b, --become run operations with become (nopasswd implied)
--become-method=BECOME_METHOD
privilege escalation method to use (default=sudo),
valid choices: [ sudo | su | pbrun | pfexec | runas |
doas ]
--become-user=BECOME_USER
run operations as this user (default=root)
-C, --check don't make any changes; instead, try to predict some
of the changes that may occur
-c CONNECTION, --connection=CONNECTION
connection type to use (default=smart)
-D, --diff when changing (small) files and templates, show the
differences in those files; works great with --check
-e EXTRA_VARS, --extra-vars=EXTRA_VARS
set additional variables as key=value or YAML/JSON
--flush-cache clear the fact cache
--force-handlers run handlers even if a task fails
-f FORKS, --forks=FORKS
specify number of parallel processes to use
(default=5)
-h, --help show this help message and exit
-i INVENTORY, --inventory-file=INVENTORY
specify inventory host path
(default=/etc/ansible/hosts) or comma separated host
list.
-l SUBSET, --limit=SUBSET
further limit selected hosts to an additional pattern
--list-hosts outputs a list of matching hosts; does not execute
anything else
--list-tags list all available tags
1.5. Invoking 7
Dan’s Cheat Sheets Documentation, Release 1
Ansible-pull (doc) is a small script that will checkout a repo of configuration instructions from git, and then run
ansible-playbook against that content.
Assuming you load balance your checkout location, ansible-pull scales essentially infinitely.
Help from ansible-pull 2.0.1.0:
8 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Options:
--accept-host-key adds the hostkey for the repo url if not already added
--ask-become-pass ask for privilege escalation password
-k, --ask-pass ask for connection password
--ask-su-pass ask for su password (deprecated, use become)
-K, --ask-sudo-pass ask for sudo password (deprecated, use become)
--ask-vault-pass ask for vault password
-C CHECKOUT, --checkout=CHECKOUT
branch/tag/commit to checkout. Defaults to behavior
of repository module.
-c CONNECTION, --connection=CONNECTION
connection type to use (default=smart)
-d DEST, --directory=DEST
directory to checkout repository to
-e EXTRA_VARS, --extra-vars=EXTRA_VARS
set additional variables as key=value or YAML/JSON
-f, --force run the playbook even if the repository could not be
updated
--full Do a full clone, instead of a shallow one.
-h, --help show this help message and exit
-i INVENTORY, --inventory-file=INVENTORY
specify inventory host path
(default=/etc/ansible/hosts) or comma separated host
list.
-l SUBSET, --limit=SUBSET
further limit selected hosts to an additional pattern
--list-hosts outputs a list of matching hosts; does not execute
anything else
-m MODULE_NAME, --module-name=MODULE_NAME
Repository module name, which ansible will use to
check out the repo. Default is git.
-M MODULE_PATH, --module-path=MODULE_PATH
specify path(s) to module library (default=None)
--new-vault-password-file=NEW_VAULT_PASSWORD_FILE
new vault password file for rekey
-o, --only-if-changed
only run the playbook if the repository has been
updated
--output=OUTPUT_FILE output file name for encrypt or decrypt; use - for
stdout
--private-key=PRIVATE_KEY_FILE, --key-file=PRIVATE_KEY_FILE
use this file to authenticate the connection
--purge purge checkout after playbook run
--scp-extra-args=SCP_EXTRA_ARGS
specify extra arguments to pass to scp only (e.g. -l)
--sftp-extra-args=SFTP_EXTRA_ARGS
specify extra arguments to pass to sftp only (e.g. -f,
-l)
--skip-tags=SKIP_TAGS
only run plays and tasks whose tags do not match these
values
-s SLEEP, --sleep=SLEEP
sleep for random interval (between 0 and n number of
seconds) before starting. This is a useful way to
disperse git requests
--ssh-common-args=SSH_COMMON_ARGS
1.5. Invoking 9
Dan’s Cheat Sheets Documentation, Release 1
Loops
See http://docs.ansible.com/playbooks_loops.html
Write a task:
- module: args
with_subelements:
- thelist
- fieldname
In other words, it’ll iterate over the first value as a list, call it item.0, then get the list from that value’s field named
‘fieldname’, and iterate over that as well, calling it item.1.
Presumably you could nest this deeper.
Example from the docs. With variables:
---
users:
- name: alice
authorized:
- /tmp/alice/onekey.pub
- /tmp/alice/twokey.pub
- name: bob
10 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
authorized:
- /tmp/bob/id_rsa.pub
Playbook
Playbook directory
Playbook
Templating A playbook is rendered as a Jinja2 template (doc) before processing it, but playbooks should not use
loops and conditionals.
Playbook include
Note that, unlike Task include s, playbook includes cannot set variables.
Play
A dictionary:
hosts: hosta:pattern1:pattern2 # required
vars:
var1: value1
var2: value2
roles:
- <rolename1>
- {role: <rolename2>, var1: value1, tags: ['tag1', 'tag2']}
tags:
- <tag1>
1.7. Playbook 11
Dan’s Cheat Sheets Documentation, Release 1
- <tag2>
remote_user: username
sudo: yes|no
sudo_user: username
tasks:
- <task>
- include: <taskfile>
- include: <taskfile2>
tags: [tag1, tag2]
- <task>
handlers:
- <task>
- include: <taskfile>
- <task>
notify:
- <handler name>
- <handler name>
vars_files:
- <path to external variables file>
- [<path1>, <path2>, ...] (ansible loads the first one found)
- <path to external variables file>
strategy: linear|free
serial: <number>|"<number>%"
Required keys:
hosts A string, containing one or more Host pattern s separated by colons
Optional keys:
handlers list of Handler s and Task include s.
pre_tasks list of Task s and Task include s. These are executed before roles.
roles list of names of Role s to include in the play. You can add parameters, tags, and conditionals:
roles:
- common
- { role: foo_app_instance, dir: '/opt/a', tags: ["bar", "baz"] }
- { role: foo_app_instance, dir: '/opt/b', when: "ansible_os_family == 'RedHat'" }
serial Set how many hosts at a time to run at a time. The default is to run tasks on all of a play’s machines at once.
See also strategy.
strategy How plays are run on multiple hosts. The default is “linear”, where each task is run on up to serial hosts in
parallel, and then Ansible waits for them all to complete before starting the next task on all the hosts.
“free” lets each host run independently, starting its next task as soon as it finishes the previous one, regardless
of how far other hosts have gotten.
tags see Tags.
tasks list of Task s and Task include s. These are executed after the roles.
post_tasks list of Task s and Task include s. These are executed after the tasks.
notify list of names of Handler s to trigger when done, but only if something changed
vars A dictionary defining additional Variables
remote_user user to login as remotely
sudo yes|no
12 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Running a playbook
Roles
Role
A role is a directory with specified contents. The role directory must be in one of the directories on the roles_path and
its name is used to refer to the role elsewhere.
Inside the role’s top-level directory, you might see a tree like this (most of this is optional).
defaults/main.yml variables within will be defined at the lowest priority (can be overridden by variables declared
anywhere else, even inventory variables)
files/ Any copy tasks can reference files in roles/x/files/ without having to path them relatively or absolutely
Any script tasks can reference scripts in roles/x/files/ without having to path them relatively or absolutely
handlers/main.yml handlers listed therein will be added to the play
library/ modules here (directories) will be available in the role, and any roles called after this role
meta/main.yml Role dependencies file
tasks/main.yml Tasks file
Any include tasks can reference files in roles/x/tasks/ without having to path them relatively or absolutely
templates/ Any template tasks can reference files in roles/x/templates/ without having to path them relatively or
absolutely
vars/main.yml variables listed therein will be added to the play. These override almost any other variables except
those on the command line, so this is really better for the role’s “constants” than variables :-)
1.8. Roles 13
Dan’s Cheat Sheets Documentation, Release 1
allow-duplicates yes|no
Defaults to no, preventing the same role from being listed as a dependency more than once. Set to yes if you
want to list the same role with different variables.
Example:
---
dependencies:
- role: role1
- role: role2
varname: value
Dependency dictionary
Required keys:
role
name of role, or quoted path to role file, or quoted repo URL:
role: postgres
role: ‘/path/to/common/roles/foo’
role: ‘git+http://git.example.com/repos/role-foo,v1.1,foo’
role: ‘/path/to/tar/file.tgz„friendly-name’
Optional keys: any parameters for the role - these define Variables
Secrets
1. Have Ansible prompt for it by passing --ask-vault-pass. Most secure, but inconvenient.
2. Put it plaintext in a well-protected file, and pass --vault-password-file <filename>. Most insecure,
but more convenient than the prompt.
3. Write a script or program that outputs the password on stdout, mark it executable, and pass that:
--vault-password-file <path-to-program>. This makes it possible to use a local system key-
chain or something, which might be more secure than the other options. Or worse...
14 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Ways to use it
One approach I’ve used is to have a single encrypted secrets.yml file in my base directory containing all my secret
variables, and another file with very restrictive permissions (and outside of source control) containing my password,
then add these arguments when running ansible:
--extra-vars @secrets.yml --vault-password-file path/to/passfile
The advantage of that is that if I don’t need the secrets, I can leave all that off and Ansible will run fine. (As opposed
to having the encrypted file automatically read by Ansible every time.)
I’m not sure if that will scale, though.
Limitations
• This is symmetric encryption. In other words, anyone with the password can encrypt and decrypt a file.
• All the encrypted files must be encrypted using the same password.
• That means you have to protect the decrypting password (the only password) very carefully, and makes providing
it to Ansible awkward.
Links:
• Vault
• Not logging secrets
• How to upload encrypted file using ansible vault?
• Managing Secrets with Ansible Vault – The Missing Guide (Part 1 of 2)
• Managing Secrets with Ansible Vault – The Missing Guide (Part 2 of 2)
synchronize
The Ansible synchronize module gets its own page because it is a bitch.
(Update: apparently some of these bad behaviors were bugs in Ansible 2.0.0.x, but I’m keeping this page around for
history.)
Let me count the ways:
• By default, it tries to become locally the user you’ve specified using the become_user variable that you have
said you want to become remotely. [Apparently that was a bug in 2.0.0.x and works correctly in 1.9.x and
2.0.1+.]
• Then it does not try to remotely become the user you’ve specified; you have to hack it by setting rsync_path:
"sudo rsync". [I have not tried this again with 2.0.1+.]
• Unlike every other Ansible module, the owner and group options are booleans, not the names or numbers of
users and groups. If true, it’ll try to copy the owner of the local files, but if you want to specify the ownership of
the target files yourself, you’ll have to fix it afterward.
Here’s a working example:
- name: sync source from local directory
synchronize:
dest: "{{ source_dir }}"
src: "{{ local_project_dir }}/"
1.10. synchronize 15
Dan’s Cheat Sheets Documentation, Release 1
delete: yes
rsync_path: "sudo rsync" # Use sudo on the remote system
recursive: true
rsync_opts:
- "--exclude=.git"
- "--exclude=*.pyc"
become: no # stops synchronize trying to sudo locally
Tags
When you apply tags to things, you can then control whether they’re executed by adding command line options.
Plays and tasks have optional tags attributes where you can specify a list of tags. Here are some tagged Task s:
tasks:
- module: parm1=a parm2=b
tags:
- packages
You can also apply tags when invoking a role from a playbook:
roles:
- { role: webserver, port: 5000, tags: [ 'web', 'foo' ] }
16 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
- include: foo.yml
tags: [web,foo]
What tags do
Adding a tag to a play or task says that if ansible is invoked with --tags=x,y,z, that the tagged play or task will
only be executed if at least one of its tags is included in the list of tags from the command line.
Specifying --tags=all is equivalent to the default behavior, where all playbooks and tasks are run regardless of
their tags.
Specifying --tags=tagged runs only things that have some tag, while -tags=untagged runs only things that
have no tag.
You could alternatively invoke ansible with --skip-tags=a,b,c and it will execute all plays and tasks that are
not tagged with a, b, or c.
Presumably --skip-tags=tagged does the opposite of --tags=tagged, and --skip-tags=untagged
does the opposite of --tags=untagged.
If a play or task is tagged always, then it will be executed unless ansible is invoked with skip-tags=always.
Task
Tasks file
Task include
Anywhere there can be a task definition, you can also use a task include:
- include: <path to tasks file> [options]
The path is relative to the Playbook directory, or the file is also searched for in the tasks directory of a role.
[options] is an optional list of additional variable settings, e.g.:
- include: tasks/footasks.yml vara=1 varb=2 varc=3
You can use an expanded syntax with a vars setting to set more complicated values:
- include: wordpress.yml
vars:
wp_user: timmy
some_list_variable:
- alpha
- beta
- gamma
1.12. Task 17
Dan’s Cheat Sheets Documentation, Release 1
Task
doc
A dictionary:
name: string # optional but highly recommended
module: args # required; the "action"
environment: dictionary
remote_user: username
sudo: yes|no
sudo_user: username
otheroption: othervalue # depending on module
tags:
- <tag1>
- <tag2>
Required keys:
name text
modulename options
Optional keys that can be used on any task:
environment dictionary (in YAML, or variable containing dictionary)
ignore_errors if true, continue even if task fails
register <varname> store result of task in <varname>. See also when for some ways to use.
remote_user user to login as remotely
sudo yes|no
sudo_user user to sudo to remotely
tags list of tags to associate with the task
when expression controls whether task is executed doc:
when: <varname>
when: not <varname>
Additional keys might be required and optional depending on the module being used.
Handler
18 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Variables
Variables
Variables file
Variables files
Ansible will look in Inventory directory and Playbook directory for directories named host_vars or group_vars.
Inside those directories, you can put a single Variables file with the same name as a host or group (respectively) and
Ansible will use those Variables definitions.
1.13. Variables 19
Dan’s Cheat Sheets Documentation, Release 1
Or a file named all that will be used for all hosts or groups.
Or you can create a directory with the same name as a host or group and Ansible will use all the files in that directory
as Variables file s.
You can also include vars files from a Play (doc).
Facts
Ansible automatically defines a whole bunch of variables with information about the system that it’s running on (the
system the plays and tasks are running on, not the system you’re controlling ansible from).
You can add to the facts with config files called local facts (doc) though I don’t know how that’s any better than putting
variables in all the other places you can set them...
To see a list of all of the facts that are available about a machine, you can run the “setup” module as an ad-hoc action:
ansible -m setup hostname
This will print out a dictionary of all of the facts that are available for that particular host. And here’s an example.
Ansible Galaxy
Links
• doc
• popular roles and recent activity
• search
Role specification
Ways of installing
Command-line
20 Chapter 1. Ansible
Dan’s Cheat Sheets Documentation, Release 1
Simple file
YAML file
Use a YAML file to provide more control. The YAML file should contain a list of dictionaries. Each dictionary
specifies a role to install. Keys can include:
src (required) a role specification as above. (Since there’s a separate dictionary key for version, I don’t know
whether you can include version here, or if you’re required to list it separately as version.)
path Where to install (directory, can be relative)
version version to install. e.g. master or v1.4.
name install as a different name
scm default git but could say hg and then in src provide a URL to a mercurial repository.
Example:
# install_roles.yml
# from galaxy
- src: yatesr.timezone
# from github
- src: https://github.com/bennojoy/nginx
- src: http://bitbucket.org/willthames/hg-ansible-galaxy
scm: hg
Ad-hoc command
Host pattern
22 Chapter 1. Ansible
CHAPTER 2
AWS
Contents:
S3
Access control
23
Dan’s Cheat Sheets Documentation, Release 1
]
}
You can use s3cmd ls to get a list of the buckets you can access.
24 Chapter 2. AWS
CHAPTER 3
Bootstrap
The grid.
SO what does a class col-SIZE-N mean?
Each SIZE has a BREAKPOINT:
xs: -1 sm: 750px md: 970px lg: 1170px
Call the window width WIDTH.
For a single class col-SIZE-N:
if WIDTH >= BREAKPOINT(SIZE), then
ELEMENT-WIDTH=WIDTH*N/12
display INLINE (same line as previous element if possible)
else
ELEMENT-WIDTH=100%
display BLOCK (element gets its own line)
and so forth - just look at the class with the largest size
NOTE: Since all widths are >= the breakpoint of XS, then if XS is present, the element will ALWAYS be laid out
inline. Though col-xs-12 is pretty much equivalent to not having an XS class, right???????????/
25
Dan’s Cheat Sheets Documentation, Release 1
26 Chapter 3. Bootstrap
CHAPTER 4
Debian
Services
update-rc.d:
• Make a service run in its default runlevels:
update-rc.d <service> defaults
or:
update-rc.d <service> enable
Packages
Alternatives
27
Dan’s Cheat Sheets Documentation, Release 1
Set xdg program to open/browse a directory (DOES NOT WORK) (do NOT use sudo):
xdg-mime default /usr/share/applications/Thunar.desktop x-directory/normal
28 Chapter 4. Debian
CHAPTER 5
Diet
• Breads
• Dense wholegrain breads
• White corn tortillas
• Grain and seed breads
• Fruit Loaf such as Raisin
• Multigrain breads (look for breads where you can see lots of grains)
• Authentic Sourdough bread
• Breakfast Cereals
• Traditional porridge oats
• Muesli*
• Bircher Muesli
• Wholegrain high fibre cereals
• Vegetables
• Sweetcorn
• Silverbeet
• Carrots
• Zucchini
• Peas, frozen or fresh
• Snowpeas
• CarismaTM Potatoes*
• Green Beans
• Broccoli
• Eggplant
• Cauliflower
29
Dan’s Cheat Sheets Documentation, Release 1
• Squash
• Capsicum
• Salad Vegetables
• Celery
• Leeks
• Tomatoes
• Mushrooms – very low carb or no GI rating
• Butternut Pumpkin (lower GI)
• Avocadoes
• Drinks
• Milo®
• Skim Latte
• Sustagen®
• Soy Drinks
• Fruit Smoothies
• Fruit Juice
• Snacks
• Grain & Fruit bars
• Wholegrain crackers
• Nut & Seed bars
• Dried fruit and nuts
• Legumes
• Split Peas; Green or red Lentils
• Baked Beans
• Canned & Dried beans – kidney, cannellini, butter, borlotti, chickpeas
• Spreads
• Fruit Spreads
• Hummus
• Nut butters
• Main Meal Carbs
• Doongara Low GI White rice
• Fresh Noodles – Hokkein, Udon, Rice
• Low GI Brown rice*
• Soba Noodles
• Basmati rice (lower GI)
• Buckwheat
30 Chapter 5. Diet
Dan’s Cheat Sheets Documentation, Release 1
• Decrease or eliminate:
• Sweets
• Alcohol
• Refined carbohydrates:
• White rice
• bread and pasta made from white flour or semolina
• Saturated fats and fried foods:
• high fat meats
• skin on poultry
• sauces and spreads
• Trans fatty acids and hidden fats:
• hydrogenated vegetable oil
• regular fat meats
• lunchmeats
• hot dogs
• fatty snack foods
• Eat more:
• omega 3 fatty acids:
• fatty fish
• salmon
• mackerel
• sardines
• tuna
• trout
• ground flax seed
• flaxseed oil
• soy products
• legumes
• walnuts
• dark leafy green vegetables
• high fiber foods:
• beans
• whole grains
• ground flaxseed
• pumpkin seeds
• rice bran
• oat bran
• fruits and vegetables
• Eat more plant foods: Vegetable proteins such as
• dried beans,
32 Chapter 5. Diet
Dan’s Cheat Sheets Documentation, Release 1
• peas, and
• soy products;
• White poultry, prepared without the skin, is also a good source of protein without a lot of fat content.
34 Chapter 5. Diet
CHAPTER 6
Django
These are just things I always find myself looking up, so I try to make some notes of the most important parts that I
can refer back to.
Contents:
Admin
URLs
List {{ app_label }}_{{ model_name }}_changelist Change {{ app_label }}_{{ model_name }}_change object_id
https://docs.djangoproject.com/en/stable/ref/contrib/admin/#reversing-admin-urls
Create your own templates/admin/base_site.html that comes ahead of the admin’s default one in the templates path.
At least in Django 1.8+, this gives you a “View site” link for free:
% extends “admin/base.html” %}
{% block title %}{{ title }} | {{ site_title|default:_(‘Django site admin’) }}{% endblock %}
{% block branding %} <h1 id=”site-name”><a href=”{% url ‘admin:index’ %}”>{{
site_header|default:_(‘Django administration’) }}</a></h1> {% endblock %}
{% block userlinks %} <a href=”{% url “clear-cache” %}”>Clear cache</a> / {{ block.super }}
{% endblock userlinks %}
Prior to Django 1.8:
{% extends “admin/base.html” %}
{% block title %}{{ title }} | Caktus Admin{% endblock %}
{% block branding %}<h1 id=”site-name”>Caktus Admin</h1>{% endblock %}
{% block nav-global %}
<div style=’display:block; padding:0 1em 0.5em 1em; float:right;’> <a href=’{% url “home”
%}’>Return to Caktus Home</a> | <a href=’{% url “clear-cache” %}’>Clear cache</a>
</div>
35
Dan’s Cheat Sheets Documentation, Release 1
{% endblock %}
Applications
https://docs.djangoproject.com/en/stable/ref/applications/#django.apps.AppConfig
In __init__.py:
# programs/__init__.py
default_app_config = 'programs.apps.ProgramsConfig'
In apps.py:
# programs/apps.py
class ProgramsConfig(AppConfig):
name = 'programs' # required: must be the Full dotted path to the app
label = 'programs' # optional: app label, must be unique in Django project
verbose_name = "Rock ’n’ roll" # optional
def ready():
"""
This runs after all models have been loaded, but you may not
modify the database in here.
post_migrate.connect(`callable`)
Celery
Useful settings
http://docs.celeryproject.org/en/latest/configuration.html
CELERY_ALWAYS_EAGER: If this is True, all tasks will be executed locally by blocking until the task returns.
apply_async() and Task.delay() will return an EagerResult instance, which emulates the API and behavior of Asyn-
cResult, except the result is already evaluated.
That is, tasks will be executed locally instead of being sent to the queue.
CELERY_EAGER_PROPAGATES_EXCEPTIONS: If this is True, eagerly executed tasks (applied by task.apply(),
or when the CELERY_ALWAYS_EAGER setting is enabled), will propagate exceptions.
It’s the same as always running apply() with throw=True.
36 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
CELERY_IGNORE_RESULT: Whether to store the task return values or not (tombstones). If you still want to store
errors, just not successful return values, you can set CELERY_STORE_ERRORS_EVEN_IF_IGNORED.
CELERYD_HIJACK_ROOT_LOGGER: By default any previously configured handlers on the root logger will be re-
moved. If you want to customize your own logging handlers, then you can disable this behavior by setting CEL-
ERYD_HIJACK_ROOT_LOGGER = False.
CELERYBEAT_SCHEDULE: In each task, you can add an ‘options’ dictionary and set ‘expires’ to a number of
seconds. If the task doesn’t run within that time, it’ll be discarded rather than run when it finally gets to a worker. This
can help a lot with periodic tasks when workers or the queue gets hung up for a while and then unjammed - without
this, the workers will have to work through a huge backlog of the same periodic tasks over and over, for no reason.
Example:
CELERYBEAT_SCHEDULE = {
'process_new_scans': {
'task': 'tasks.process_new_scans',
'schedule': timedelta(minutes=15),
'options': {
'expires': 10*60, # 10 minutes
}
},
}
CELERY_DEFAULT_QUEUE: In the absence of more complicated configuration, celery will use this queue name for
everything. Handy when multiple instances of a site are sharing a queue manager:
CELERY_DEFAULT_QUEUE = 'queue_%s' % INSTANCE
django-compressor
django-compressor docs
FIRST read the usage page in the docs, down to the paragraph starting “Which would be rendered something like:”,
to understand at a high-level what compressor does.
For now, don’t read anything else; it can be confusing.
Much of the following information is based on the code, not the current docs, and hopefully will be more accurate.
Cache
For some things, compressor uses the cache named by COMPRESS_CACHE_BACKEND, which defaults to None,
which gives us the default Django cache.
Principles of compression
Whether compressor is processing templates offline ahead of time or at runtime, there are some common principles.
First, if COMPRESS_ENABLED is False, the {% compress %} tag will simply render as its contents; compressor
won’t change anything.
Otherwise, compressor will
1. parse the contents of the tag and figure out which css and javascript files would be included
2. fetch those files (See “accessing the files to be compressed”)
6.4. django-compressor 37
Dan’s Cheat Sheets Documentation, Release 1
Offline
If COMPRESS_OFFLINE is True, compressor expects all uses of {% compress ... %} in templates to have
been pre-processed by running manage.py compress ahead of time, which puts the results in compressor’s offline
cache. If anything it needs at run-time is not found there, things break/throw errors/render wrong etc.
The offline cache manifest is a json file, stored using COMPRESS_STORAGE, in the subdirectory
COMPRESS_OUTPUT_DIR (default: CACHE), using the filename COMPRESS_OFFLINE_MANIFEST (default:
manifest.json).
The keys in the offline cache manifest are generated from the template content inside each compress tag, not the
contents of the compressed files. So, you must arrange to re-run the offline compression anytime your content files
might have changed, or it’ll be serving up compressed files generated from the old file contents.
The values in the offline cache manifest are paths of the compressed files in COMPRESS_STORAGE.
Not offline
If COMPRESS_OFFLINE is False, compressor will look in COMPRESS_STORAGE for previously processed results,
but if not found, will create them on the fly.
Important Advice
Note: If you follow the above advice, you can probably skip reading the rest of this.
Storage
Compressor uses a Django storage class for some of its operations, controlled by the setting COMPRESS_STORAGE.
The default storage is compressor.storage.CompressorFileStorage, which is a subclass of the standard
filesystem storage class. It uses COMPRESS_ROOT as the base directory in the local filesystem to store files in, and
builds URLs by prefixing file paths within the storage with COMPRESS_URL.
38 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
If you change COMPRESS_STORAGE, then ignore anything in the docs about COMPRESS_ROOT and
COMPRESS_URL as they won’t apply anymore (except in a few cases... see exceptions noted as they come up,
below).
For each file to be compressed, compressor starts with the URL from the rendered original content inside the compress
tag. For example, if part of the content is <script src="http://example.com/foo.js"></script>,
then it extracts "http://example.com/foo.js" as the URL.
It checks that the URL starts with COMPRESS_STORAGE’s base_url, or if accessing that fails (quite possible
since base_url is not a standard part of the file storage class API), uses COMPRESS_URL.
Note: This is a place where compressor can use COMPRESS_URL even if it’s not using its default storage.
If the URL doesn’t start with that string, compressor throws a possibly misleading error, “’%s’ isn’t accessible via
COMPRESS_URL (‘%s’) and can’t be compressed”.
Otherwise, compressor tries to come up with a local filepath to access the file, as follows:
• Try to get a local filepath from COMPRESS_STORAGE using .path().
• If that’s not implemented (for example, for remote storages), it tries again using a different code path, though
that appears to end up again going to COMPRESS_STORAGE.
• If it still can’t get a local filepath, throws an error: “’%s’ could not be found in the COMPRESS_ROOT ‘%s’%s”
which is very misleading if you’re not using a storage class that looks at COMPRESS_ROOT.
When we start thinking about how collectstatic and compressor interact, things get really hinky, and the documentation
especially misleading.
Two problems:
1. Compressor looks at COMPRESS_STORAGE to find the source files to compress, when logically you’d expect
it to look at STATICFILES_STORAGE. But collectstatic uses STATICFILES_STORAGE to gather and
store the source files.
2. Compressor tries to hack access to the storage class to find the local path to the files, rather than just asking the
storage class to open the file wherever it is.
This means for all practical purposes, if you want this to work, you have to:
1. Set COMPRESS_STORAGE the same as STATICFILES_STORAGE, COMPRESS_URL the same as
STATIC_URL, and COMPRESS_ROOT the same as STATIC_ROOT. (So why does django-compressor even
have these settings?)
2. Hack whatever storage class STATICFILES_STORAGE is using, if it normally just stores files remotely, to save
a local copy and make it accessible via .path(). There’s an example in the docs.
Data fixtures
Load it again:
python manage.py loaddata data.yaml
Natural keys
https://docs.djangoproject.com/en/stable/topics/serialization/#natural-keys
from django.db import models
class PersonManager(models.Manager):
def get_by_natural_key(self, first_name, last_name):
return self.get(first_name=first_name, last_name=last_name)
class Person(models.Model):
objects = PersonManager()
...
def natural_key(self):
return (self.first_name, self.last_name)
class Meta:
unique_together = (('first_name', 'last_name'),)
Dependencies
If part of the natural key is a reference to another model, then that model needs to be deserialized first:
class Book(models.Model):
name = models.CharField(max_length=100)
author = models.ForeignKey(Person)
def natural_key(self):
return (self.name,) + self.author.natural_key()
natural_key.dependencies = ['example_app.person']
Databases
Performance
From Django 1.6 on, always add CONN_MAX_AGE to database settings to enable persistent connections. 300 is a
good starting value (5 minutes). None will keep them open indefinitely.
BUT - keep in mind that every open connection to Postgres consumes database server resources. So you might want
instead to run pgbouncer locally.
Install/config
40 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
Install:
pip install django-debug-toolbar
settings.py:
DEBUG = True
INTERNAL_IPS = ['127.0.0.1']
INSTALLED_APPS += [
'debug_toolbar',
]
# The order of MIDDLEWARE and MIDDLEWARE_CLASSES is important. You should include
# the Debug Toolbar middleware as early as possible in the list. However, it must
# come after any other middleware that encodes the response’s content, such as
# GZipMiddleware.
MIDDLEWARE = [
'debug_toolbar.middleware.DebugToolbarMiddleware',
] + MIDDLEWARE
urls.py:
from django.conf import settings
from django.conf.urls import include, url
if settings.DEBUG:
import debug_toolbar
urlpatterns += [
url(r'^__debug__/', include(debug_toolbar.urls)),
]
Dokku
This page has information for those who have to set up and maintain a Dokku server. If you’re just using one, you can
ignore this.
Initial install
The Dokku docs recommend setting up a new OS install of a supported operating system, then running the Dokku
install script.
Experience suggests that that approach is more likely to work than trying to install Dokku on a system that has already
had some configuration done for other things.
6.8. Dokku 41
Dan’s Cheat Sheets Documentation, Release 1
Simple hostnames
Managing users
In other words, who can mess with the apps on a dokku server?
The way this currently works is that everybody ends up sshing to the server as the dokku user to do things. To let
them do that, we want to add a public key for them to the dokku config, by doing this (from any system):
$ cat /path/to/ssh_keyfile.pub | ssh dokku ssh-keys:add <KEYNAME>
The <KEYNAME> is just to identify the different keys. I suggest using the person’s typical username. Just remember
there will not be a user of that name on the dokku server.
When it’s time to revoke someone’s access:
$ ssh dokku ssh_keys:remove <KEYNAME>
Files
requirements.txt
There needs to be a requirements.txt file at the top level. If you prefer to keep your requirements somewhere
else, the top-level one can just look like:
-r path/to/real/requirements.txt
42 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
settings
MIDDLEWARE.remove('django.middleware.security.SecurityMiddleware')
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
] + MIDDLEWARE
# Allow all host headers (feel free to make this more specific)
ALLOWED_HOSTS = ['*']
wsgi.py
Procfile
Create Procfile (more) in the top directory. For our simple case, it can just contain one line, starting with ‘‘web: ‘‘ and
containing the command to start gunicorn for our site:
6.8. Dokku 43
Dan’s Cheat Sheets Documentation, Release 1
runtime.txt
Create runtime.txt in the top directory. It only needs one line, e.g.:
python-3.6.1
This has to be specific. E.g. python-3.5.2 or python-3.6.1 might work if the dokku server supports it, but
python-3.5 or python-3.6 probably won’t.
app.json
Create app.json in the top-level project directory. You might see examples on the Interwebs with lots of things
in app.json (because Heroku uses app.json for lots of things), but as of this writing, dokku ignores everything but
scripts.dokku.predeploy and scripts.dokku.postdeploy. Example:
{
"scripts": {
"dokku": {
"predeploy": "python manage.py migrate --noinput"
}
}
}
Note: Dokku automatically runs collectstatic for you, so you don’t need to do that from app.json.
buildpacks
If your app is not pure Python - e.g. if it uses node - you’ll need to override the automatic buildpack detection, because
it only works for a single application type.
Do this by adding a top-level .buildpacks file, containing links to the buildpacks to use:
https://github.com/heroku/heroku-buildpack-nodejs.git
https://github.com/heroku/heroku-buildpack-python.git
https://github.com/heroku/heroku-buildpack-apt
There’s nothing Django-specific about this, but I’m including it just because we probably want to do it on every single
Django deploy.
To install the postgresql plugin, inside your server run (because plugins must be installed as root):
$ sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git
Now you need to create a database, and link the database to the app. You can do this from your own system:
44 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
Now when dokku runs your app, it’ll set an env var to tell it where its DB is, e.g.:
DATABASE_URL=postgres://user:pass@host:port/db
There are built-in commands making it easy to backup and restore databases:
$ ssh dokku postgres:export [db_name] > [db_name].dump
$ ssh dokku postgres:import [db_name] < [db_name].dump
Letsencrypt
Note: Get the site up and running, and accessible from the Internet, first. Let’s Encrypt will not be able to get you a
certificate until then.
There’s nothing Django-specific about this part, but I’m including it just because we probably want to do it on every
single Django deploy.
To add SSL with the Let’s Encrypt plugin (more), first install the plugin by running on the dokku server (plugins must
be installed as root):
$ sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
Then on your system, to configure your app and tell letsencrypt to manage its certs and renew them periodically:
$ ssh dokku config:set --no-restart <appname> [email protected]
$ ssh dokku letsencrypt myapp
$ ssh dokku letsencrypt:cron-job --add <appname>
If we don’t want to figure out how to override the default nginx config to redirect non-SSL requests to SSL, we can
have Django do it with a few settings.
First, set SECURE_SSL_REDIRECT to True. This will tell Django to do the redirect.
SECURE_SSL_REDIRECT = True
This adds a header to all responses, telling any browser that receives it that this site should only be accessed via SSL,
for that many seconds.
Commit, deploy, and make sure the site still works.
6.8. Dokku 45
Dan’s Cheat Sheets Documentation, Release 1
If everything still seems good, bite the bullet, increase SECURE_HSTS_SECONDS to a large number (e.g. 31536000
seconds, or 1 year), commit, deploy, and test again.
Additional info (move to their own files as they’re ready)...
Environment variables
I can’t find docs on what environment variables Dokku sets globally when running apps. A little poking around in one
of my deployed apps showed that there doesn’t seem to be a whole lot beyond what the plugins are providing, but here
are a few that look useful:
DOKKU_DEPLOY_BRANCH
The name of the branch that is running, e.g. master or develop.
and... that’s about it?
Actually, not even that seems to be reliably provided.
Still, just about any Django app is going to be linked to a database, so if you need to detect whether the app is running
under Dokku or Heroku, looking for DATABASE_URL should be good enough.
Static files
We use whitenoise to serve static files from Django. If the site gets incredible amounts of traffic, throw a CDN in
front, but honestly, very few sites actually need that. (If you have a philosophical objection to serving static files from
Django, you can customize the nginx config through Dokku and probably manage to get nginx to do the static file
serving, but I haven’t bothered figuring it out myself.)
Or put your static files on S3.
Django media
If your site requires uploaded files to be persisted, remember that the container running your code is ephemeral and
any changes made to files inside it will vanish at the next deploy.
First, you can use S3 for your media files.
Or you can use persistent storage on Dokku, which is a way of mounting directories from the dokku server inside
the running container where your site code can store and read files that will continue to exist past the lifetime of that
particular container.
Simple hostnames
If the server is set up properly, assuming the server’s domain name is dokkuserver.domain, creating an app
named foo will automatically foo.dokkuserver.domain resolve to the server’s address, and requests with host
foo.dokkuserver.domain to be routed to that app.
If you want to get fancier, http://dokku.viewdocs.io/dokku/configuration/domains/.
Also, note that any requests simply addressed to dokku.me will get routed to the alphabetically first app on the
server, but you can change that: http://dokku.viewdocs.io/dokku/configuration/domains/ or just set up a “00default”
app.
46 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
If requests are being terminated at a load balancer and then proxied to our dokku server, some ng-
inx config customization will be needed so your app can see the actual origin of the requests:
http://dokku.viewdocs.io/dokku/configuration/ssl/#running-behind-a-load-balancer
Run a command
Suppose you want to run something like python manage.py createsuperuser in the app environment?
$ ssh dokku run <appname> python manage.py createsuperuser
will do it.
Suppose you need to run another instance of your app in another way, for example to run celery beat and celery
worker.
Use the Procfile to tell Dokku what processes to run. E.g. if your Procfile is:
web: gunicorn path/to/file.wsgi
With just that, the extra processes won’t run automatically. You can run them by telling Dokku to scale them up, e.g.:
$ ssh dokku ps:scale <appname> beat=1 worker=4
and see what’s actually running (example from another app that only has one process):
$ ssh dokku ps:report <appname>
=====> <appname>
Processes: 1
Deployed: true
Running: true
Restore: true
Restart policy: on-failure:10
Status web.1 true (CID: 03ea8977f37e)
6.8. Dokku 47
Dan’s Cheat Sheets Documentation, Release 1
Since we probably don’t want to have to remember to manually scale these things up and check that they’re running,
we can add a DOKKU_SCALE file to our repo:
web=1
beat=1
worker=4
Secrets
First, if possible, use Dokku plugin integrations for things like databases, Redis, cache, etc. They automatically set
environment variables in each app that your settings can read, so you don’t have to manage different settings for each
environment. See each plugin’s doc, of course, for more on that.
The way to handle other secrets for each environment is to set them as config on the app, which will add them as
environment variables, again so your settings can read them at runtime.
The downside is that the secrets aren’t being managed/versioned etc for us... but I think we can handle the few we’ll
need by manually keeping them in Lastpass or something.
Here are the docs for setting config.
Before setting any config, note that by default, making any change to the config triggers a new deploy. If you’re not
ready for that, include --no-restart in the command, as these examples will do.
To set config variables:
$ ssh dokku config:set <appname> --no-restart VAR1=VAL1 VAR2=VAL2 ...
To remove a variable:
$ ssh dokku config>unset <appname> --no-restart VAR1
Get all the settings in a single line, handy for use in shell commands or to set on another app:
$ ssh dokku config <appname> --shell
VAR1=VAL1 VAR2=VAL2
$ export $(ssh dokku config <appname> --shell)
$ ssh dokku config:set <appname1> $(ssh dokku config <appname2> --shell)
Get all the settings in a format handy to save in a file for later sourcing in a local shell:
$ ssh dokku config <appname> --export
export VAR1=VAL1
export VAR2=VAL2
$ ssh dokku config <appname> --export >appname.env
$ . appname.env
Note: you can also set config vals globally - just change <appname> to --global in any of these commands.
Logs
Dokku collects stdout from your processes and you can view it with the dokku logs command.
48 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
nginx logs are similarly stored on the server and can be accessed using dokku nginx:access-log <appname>
or dokku nginx:error-log <appname>.
Dokku event logging can be enabled with dokku events:on and then viewed with dokku events. This shows things
like deploy steps.
Note: this doesn’t apply to your main project repo. That can be private and Dokku doesn’t care, because you’re
pushing it directly from your development system to the Dokku server.
But if your requirements include references to private git repos, then you’ll need to arrange for Dokku to get access to
those repos when it’s pip installing your requirements.
Docs, such as they are...
I think the upshot is:
• Create a new ssh key for deploying
• Add it to the repo on Github (or whatever) as an authorized deploy key (TODO: Link to github docs on that)
• Drop a copy of the public key file into /home/dokku/.ssh/ on the Dokku system (with appropriate permis-
sions)
The docs
By default, dokku deploys when the app’s master branch on the dokku server is updated. There are (at least) two ways
to deploy a branch other than master.
1. Push your non-master local branch to the master branch on the server:
$ git push dokku <local branch>:master
but that requires you to always remember that if you have apps that are always supposed to use a different branch than
master.
2. Configure your app so the default branch is different, by changing the config value DOKKU_DEPLOY_BRANCH:
$ ssh dokku config:set --no-restart <app-name> DOKKU_DEPLOY_BRANCH=<some-branch>
This seems like a more useful approach. Just “git push dokku some-branch” every time you want to deploy your app.
Suppose you have local development and sometimes you want to push to staging and sometimes to production. Maybe
you have other apps too.
The key is to set up a different git remote for each remote app. E.g.:
$ ssh dokku app:create staging
$ ssh dokku config:set --no-restart staging DOKKU_DEPLOY_BRANCH=develop
6.8. Dokku 49
Dan’s Cheat Sheets Documentation, Release 1
If you need to completely override the nginx config, you’ll need to provide an nginx config file template.
Luckily, much customization can be done just by providing snippets of configuration for nginx to include after it’s
base config file.
To do this, arrange for the snippets to get copied to /home/dokku/<appname>/nginx.conf.d/ during de-
ployment, probably in a pre- or post-deploy script.
Logging to papertrail
https://github.com/darklow/dokku-sentry
SSH into a random instance. This assumes that you have copied the SSH private key into your $HOME/.ssh direc-
tory:
(bringfido)$ eb ssh staging
Check the overall status of the environment, or detailed info about each instance:
(bringfido)$ eb status -v staging
(bringfido)$ eb health staging
50 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
https://docs.djangoproject.com/en/stable/topics/email/
API
Attachments
Email backends/handlers
https://docs.djangoproject.com/en/stable/topics/email/#email-backends
For development:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
For real:
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
In-memory backend - The ‘locmem’ backend stores messages in a special attribute of the django.core.mail module.
The outbox attribute is created when the first message is sent. It’s a list with an EmailMessage instance for each
message that would be sent:
EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'
This backend is not intended for use in production – it is provided as a convenience that can be used during development
and testing.
ADMINS A tuple that lists people who get code error notifications:
(('John', '[email protected]'), ('Mary', '[email protected]'))
6.10. Email 51
Dan’s Cheat Sheets Documentation, Release 1
Fabric
Sample tasks:
@task
def db_backup():
"""
Backup the database to S3 just like the nightly cron job
"""
require('environment')
require('instance', provided_by='instance')
manage_run("dbbackup --encrypt")
def db_exists(dbname):
"""
Return True if a db named DBNAME exists on the remote host.
"""
require('environment', provided_by=SERVER_ENVIRONMENTS)
output = sudo('psql -l --pset=format=unaligned', user='postgres')
dbnames = [line.split('|')[0] for line in output.splitlines()]
return dbname in dbnames
@task
def db_dump(file):
"""
Dump an instance's database to a remote file.
Example:
dumps to staging_iraq.dump
"""
require('environment', provided_by=SERVER_ENVIRONMENTS)
require('instance', provided_by='instance')
remote_file = file
if files.exists(file):
if not confirm("Remote file {file} exists and will be overwritten. Okay?"
.format(file=remote_file)):
abort("ERROR: aborting")
# Don't need remote DB user and password because we're going to run pg_dump as user postgres
sudo('pg_dump --format=custom --file={outputfile} {dbname}'
.format(dbname=env.db_name, outputfile=remote_file),
user='postgres')
print("Database from {environment} {instance} has been dumped to remote file {file}"
.format(environment=env.environment, instance=env.instance, file=remote_file))
@task
def local_restore(file):
"""
Restore a local dump file to the local instance's database.
:param file:
:return:
52 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
"""
# Find out the local DB settings
import sys
sys.path[0:0] = ['.']
from cts.settings.local import DATABASES
DB = DATABASES['default']
assert DB['ENGINE'] == 'django.contrib.gis.db.backends.postgis'
dbname = DB['NAME']
owner = DB['USER'] or os.getenv('USER')
local('dropdb {dbname} || true'.format(dbname=dbname), shell="/bin/sh")
local('createdb --encoding UTF8 --lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8 --template=templa
local('sudo -u postgres pg_restore -Ox -j4 --dbname={dbname} {file}'.format(dbname=dbname, file=f
@task
def db_restore(file):
"""
Restore a remote DB dump file to a remote instance's database.
Example:
renamed = False
restored = False
if not files.exists(file):
abort("Remote file {file} does not exist".format(file=file))
try:
if db_exists(env.db_name):
# Rename existing DB to backup
db_backup = '{dbname}_bak'.format(dbname=env.db_name)
if db_exists(db_backup):
if confirm("There's already a database named {db_backup}. Replace with new backup?"
.format(db_backup=db_backup)):
sudo('dropdb {db_backup}'.format(db_backup=db_backup),
user='postgres')
else:
abort("ERROR: There's already a database named {db_backup}. "
"Restoring would clobber it."
.format(db_backup=db_backup))
sudo('psql -c "ALTER DATABASE {dbname} RENAME TO {db_backup}"'
.format(dbname=env.db_name, db_backup=db_backup),
user='postgres')
renamed = True
print("Renamed {dbname} to {db_backup}".format(dbname=env.db_name, db_backup=db_backup))
6.11. Fabric 53
Dan’s Cheat Sheets Documentation, Release 1
remote_file = file
print("Database for {environment} {instance} has been restored from remote file {file}"
.format(environment=env.environment, instance=env.instance, file=remote_file))
finally:
if renamed and not restored:
print("Error occurred after renaming current database, trying to rename it back")
if db_exists(env.db_name):
# We already created the new db, but restore failed; delete it
sudo('dropdb {dbname}'.format(dbname=env.dbname), user='postgres')
sudo('psql -c "ALTER DATABASE {db_backup} RENAME TO {dbname}"'
.format(dbname=env.db_name, db_backup=db_backup),
user='postgres')
print("Successfully put back the original database.")
Forms
• https://docs.djangoproject.com/en/stable/topics/forms/
• https://docs.djangoproject.com/en/stable/ref/forms/api/
A basic form:
from django import forms
class ContactForm(forms.Form):
subject = forms.CharField(max_length=100)
54 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
return render_to_response('contact.html', {
'form': form,
})
https://docs.djangoproject.com/en/stable/topics/forms/modelforms/
A model form:
from django.forms import ModelForm, ValidationError
def clean_fieldname(self):
if 'fieldname' in self.cleaned_data:
data = self.cleaned_data['fieldname']
if not /valid/:
raise ValidationError("msg")
return data
def clean(self):
data = self.cleaned_data
if not /valid/:
raise ValidationError("msg")
return data
# Create a form to edit an existing Article, but use POST data to populate the form.
a = Article.objects.get(pk=1)
f = ArticleForm(request.POST, instance=a)
f.save()
6.12. Forms 55
Dan’s Cheat Sheets Documentation, Release 1
<table>
<fieldset>
{% if form.non_field_errors %}
<tr><td colspan="2">{{ form.non_field_errors }}</td></tr>
{% endif %}
{% for field in form %}
<tr{% if field.field.required %} class="required"{% endif %}>
<th style="text-align: left"><label for="{{ field.id_for_label }}">{{ field.label }}:
<td>{% if field.errors %}
{{ field.errors }}<br/>
{% endif %}
{{ field }}
or even
56 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
Read-only form
# Note that a new BoundField is constructed on the fly when you access
# form[name], so any data we want to persist long enough for the template
# to access needs to be on the "real" field. We just use the BoundField
# to get at the field value.
field.readonly = True
field.display_value = display_value
{# input fields #}
{% if field.field.readonly %}
<span class="form-control">{{ field.value }}</span>
{% else %}
<input type="{% block input_field_type %}text{% endblock %}" class="form-control" id="{{ field.id
{% endif %}
{# select fields #}
{% if field.field.readonly %}
<span class="form-control">{{ field.field.display_value }}</span>
{% else %}
<select class="form-control" id="{{ field.id_for_label }}" name="{{ field.html_name }}" placehol
{% for val, label in field.field.widget.choices %}
<option value="{{ val }}"{% if field.value|stringformat:'s' == val|stringformat:'s' %} select
{% endfor %}
</select>
{% endif %}
urlpatterns = [
...
url(r'^$', TemplateView.as_view(template_name='home.html'), name='home'),
]
Logging
Default
Here’s what Django uses (around 1.7, anyway) if you don’t configure logging:
# Default logging for Django. This sends an email to the site admins on every
# HTTP 500 error. Depending on DEBUG, all other log records are either sent to
# the console (DEBUG=True) or discarded by mean of the NullHandler (DEBUG=False).
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse',
},
'require_debug_true': {
'()': 'django.utils.log.RequireDebugTrue',
},
},
'handlers': {
'console': {
'level': 'INFO',
'filters': ['require_debug_true'],
'class': 'logging.StreamHandler',
},
'null': {
'class': 'logging.NullHandler',
},
'mail_admins': {
'level': 'ERROR',
'filters': ['require_debug_false'],
58 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
'class': 'django.utils.log.AdminEmailHandler'
}
},
'loggers': {
'django': {
'handlers': ['console'],
},
'django.request': {
'handlers': ['mail_admins'],
'level': 'ERROR',
'propagate': False,
},
'django.security': {
'handlers': ['mail_admins'],
'level': 'ERROR',
'propagate': False,
},
'py.warnings': {
'handlers': ['console'],
},
}
}
Console
Development
For local development, we want lots of output saved to a log file in case we need to look back at a problem, but no
emailing of exceptions and such.
In settings:
LOG_DIR = os.path.join(PROJECT_ROOT, '..', 'log')
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': { # Log to stdout
6.15. Logging 59
Dan’s Cheat Sheets Documentation, Release 1
'level': 'INFO',
'class': 'logging.StreamHandler',
},
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': os.path.join(LOG_DIR, 'django_debug.log',
}
},
'root': { # For dev, show errors + some info in the console
'handlers': ['console'],
'level': 'INFO',
},
'loggers': {
'django.request': { # debug logging of things that break requests
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
Or how about:
LOGGING = { ‘version’: 1, ‘disable_existing_loggers’: False, ‘formatters’: {
‘simple’: { ‘format’: ‘%(name)-20s %(levelname)-8s %(message)s’,
},
}, ‘handlers’: {
‘console’: { # Log to stdout ‘level’: ‘INFO’, ‘class’: ‘logging.StreamHandler’, ‘format-
ter’: ‘simple’,
},
}, ‘root’: { # For dev, show errors + some info in the console
‘handlers’: [’console’], ‘level’: ‘INFO’,
},
}
Staging
60 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': { # Rotate log file daily, only keep 1 backup
'level': 'DEBUG',
'class': 'logging.handlers.TimedRotatingFileHandler',
'filename': os.path.join(LOG_DIR, 'django_debug.log',
'when': 'd',
'interval': 1,
'backupCount': 1,
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler'
},
},
# EMAIL all errors (might not want this, but let's try it)
'root': {
'handlers': ['mail_admins'],
'level': 'ERROR',
},
'loggers': {
'django.request': {
'handlers': ['file'],
'level': 'INFO',
'propagate': True,
},
},
}
Production
Mark says: for production I like to log to syslog which can then be shipped elsewhere without changing the application
(https://docs.python.org/2/library/logging.handlers.html#logging.handlers.SysLogHandler ?)
@Scottm and I have been talking about making that more common: log to syslog, ship to Logstash, monitor via
Kibana http://www.elasticsearch.org/overview/kibana/
getting Nginx to log to syslog is kind of a pain you basically have to get syslog to monitor the file and ship it Logstash
+ Kibana looks much easier to manage/configure than Graylog2
the plan was to add it to Ona but that isn’t done yet (as of Aug 28, 2014) CCSR was/is using Graylog2 Minidam does
syslog –> Loggly libya is using logstash -> graylog (in addition to sentry)
Example
Here’s what we’ve got set up for Django logging on one project. This sends everything level INFO and higher to a
local log file and a Graylog instance. Anything ERROR and higher is emailed to admins and sent to a Sentry instance,
which can send more notifications.
In environment:
6.15. Logging 61
Dan’s Cheat Sheets Documentation, Release 1
SENTRY_DSN: http://long_hex_string:long_hex_string@hostname:9000/3
Requirements:
raven==3.6.1
Settings:
INSTALLED_APPS = (
...
'raven.contrib.django.raven_compat', # Sentry logging client
...
}
CELERY_SEND_TASK_ERROR_EMAILS = True
62 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
'port': 12201,
'filters': ['static_fields', 'django_exc'],
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
'include_html': False,
'filters': ['require_debug_false'],
},
'sentry': {
'level': 'ERROR',
'class': 'raven.contrib.django.raven_compat.handlers.SentryHandler',
},
},
'root': {
# graylog (or any handler using the 'django_exc' filter ) should be last
# because it will alter the LogRecord by removing the `request` field
'handlers': ['file', 'mail_admins', 'sentry', 'graylog'],
'level': 'WARNING',
},
'loggers': {
# These 2 loggers must be specified, otherwise they get disabled
# because they are specified by django's DEFAULT_LOGGING and then
# disabled by our 'disable_existing_loggers' setting above.
# BEGIN required loggers #
'django': {
'handlers': [],
'propagate': True,
},
'py.warnings': {
'handlers': [],
'propagate': True,
},
# END required loggers #
# The root logger will log anything WARNING and higher, so there's
# no reason to add loggers here except to add logging of lower-level information.
'libya_elections': {
'handlers': ['file', 'graylog'],
'level': 'INFO',
},
'nlid': {
'handlers': ['file', 'graylog'],
'level': 'INFO',
},
'register': {
'handlers': ['file', 'graylog'],
'level': 'INFO',
},
'bulk_sms': {
'handlers': ['file', 'graylog'],
'level': 'INFO',
},
}
}
#
# our_filters.py
#
6.15. Logging 63
Dan’s Cheat Sheets Documentation, Release 1
import logging
class QuotelessStr(str):
"""
Return the repr() of this string *without* quotes. This is a
temporary fix until https://github.com/severb/graypy/pull/34 is resolved.
"""
def __repr__(self):
return self
class StaticFieldFilter(logging.Filter):
"""
Python logging filter that adds the given static contextual information
in the ``fields`` dictionary to all logging records.
"""
def __init__(self, fields):
self.static_fields = fields
class RequestFilter(logging.Filter):
"""
Python logging filter that removes the (non-pickable) Django ``request``
object from the logging record.
"""
def filter(self, record):
if hasattr(record, 'request'):
del record.request
return True
Django doc
• In settings.py, add:
LOGIN_URL = '/login/'
• In urls.py, add:
urlpatterns += patterns('',
(r'^login/$', django.contrib.auth.views.login),
(r'^logout/$', django.contrib.auth.views.logout),
)
• On each view function where users should be logged in before using, add the decorator:
64 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
@login_required
def myview(...)
Migrations
Data migration:
def no_op(apps, schema_editor):
pass
class Migration(migrations.Migration):
...
operations = [
migrations.RunPython(create_types, no_op),
]
For example, earlier migrations refer to models in apps that no longer exist.
The simplest thing is to start from an existing database so you don’t have to migrate.
If you need to start from scratch, you should be able to:
syncdb --all
migrate --fake
NGINX
Files
Just add a new file to /etc/nginx/sites-eanbled for each site, making sure server_name is set correctly
in each.
Redirecting to SSL
6.17. Migrations 65
Dan’s Cheat Sheets Documentation, Release 1
server_name DOMAIN;
access_log PATHo_access.log;
error_log PATH_error.log;
return 301 https://DOMAIN$request_uri;
}
Proxying to gunicorn
Serve static and media files with nginx, and proxy everything else to Django:
upstream django {
server unix:/tmp/PATH fail_timeout=0;
}
server {
listen *:443 ssl; # add spdy here too if you want
listen [::]:443 ssl;
server_name DOMAIN;
ssl_certificate PATH.crt;
ssl_certificate_key PATH.key;
access_log PATH_access.log;
error_log PATH_error.log;
root PATH;
location /media {
alias PATH;
}
location /static {
alias PATH;
}
location / {
client_max_body_size 500M;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_redirect off;
proxy_buffering on;
proxy_intercept_errors on;
proxy_pass http://django;
}
# See https://www.trevorparker.com/hardening-ssl-in-nginx/
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-RSA-AES1\
28-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM\
-SHA384:kEDH+AESGCM:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES\
256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SH\
A256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SH\
A384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SH\
A256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:\
!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:10m;
66 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
Permissions
Assuming you have an application with an app_label foo and a model named Bar, to test for basic permissions you
should use:
• add: user.has_perm(‘foo.add_bar’)
• change: user.has_perm(‘foo.change_bar’)
• delete: user.has_perm(‘foo.delete_bar’)
Permissions are more commonly referred to by a string, of the form “app_label.codename”. E.g., if
user.has_perm(’myapp.codename’): do something.
Confusingly, the Permission model has no app_label field. It uses content_type__app_label for that.
Warning: This means all permissions need a content type, whether it makes sense for that permission to be
applied to a particular model or not.
Default permissions
https://docs.djangoproject.com/en/stable/topics/auth/default/#default-permissions
6.19. Permissions 67
Dan’s Cheat Sheets Documentation, Release 1
For every model in an installed app, Django automatically creates three permissions: applabel.add_modelname, ap-
plabel.change_modelname, and applabel.delete_modelname, where the modelname is lowercased.
https://docs.djangoproject.com/en/stable/ref/models/options/#permissions
You can ask Django to create more permissions for a model:
class Meta:
permissions = [
('codename', 'verbose name'),
]
When the table is created during syncdb, Django will create the additional Permission objects too.
Warning: Prior to Django 1.7, such permissions were only created when Django creates the model’s table during
syncdb. Adding permissions to the model definition later does nothing (unless you drop the table and force
Django to create it again).
You can programmatically force Django to create additional Permissions with code like:
from django.db.models import get_models, get_app
from django.contrib.auth.management import create_permissions
Best practices
• Plan not to give users specific permissions, except when you have to make an exception to your usual policies.
• Design groups with useful sets of permissions.
• Plan to add users to the appropriate groups depending on their roles.
• Provide a way to ensure the groups continue to have the permissions you want.
Fixtures aren’t a bad way to provide initial data, but setting them up for automatic loading is deprecated with Django
1.7 and will go away with Django 2.0. Instead, load them from a data migration. This is better in some ways anyway,
because the migration will use the same version of the models that the fixtures were written for at the time. (Though,
this doesn’t matter so much for Permissions and Groups, which we don’t really expect to change their schemas...)
Add utility methods like this, maybe in accounts/utils.py or equivalent:
def permission_names_to_objects(names):
"""
Given an iterable of permission names (e.g. 'app_label.add_model'),
return an iterable of Permission objects for them. The permission
must already exist, because a permission name is not enough information
to create a new permission.
"""
result = []
for name in names:
app_label, codename = name.split(".", 1)
# Is that enough to be unique? Hope so
68 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
try:
result.append(Permission.objects.get(content_type__app_label=app_label,
codename=codename))
except Permission.DoesNotExist:
logger.exception("NO SUCH PERMISSION: %s, %s" % (app_label, codename))
raise
return result
def get_all_perm_names_for_group(group):
# Return the set of permission names that the group should contain
def create__or_update_groups():
for group_name, perm_names in GROUP_PERMISSIONS.iteritems():
group, created = Group.objects.get_or_create(name=group_name)
perms_to_add = permission_names_to_objects(get_all_perm_names_for_group(group))
group.permissions.add(*perms_to_add)
if not created:
# Group already existed - make sure it doesn't have any perms we didn't want
to_remove = set(group.permissions.all()) - set(perms_to_add)
if to_remove:
group.permissions.remove(*to_remove)
https://docs.djangoproject.com/en/stable/topics/auth/default/#authentication-data-in-templates
{% if user.is_authenticated %} {% if perms.applabel %} {# user has any permissions in app applabel #}
{% if ‘applabel’ in perms %} {# same as above %} {% if perms.applabel.change_thing %} {# user has
‘change_thing’ permission in app applabel #} {% if ‘applabel.change_thing’ in perms %} {# same as
above #}
Field lookups
exact, iexact, contains, icontains, startswith, istartswith, endswith, iendswith, in, gt, gte, lt, lte, range, year, month,
day, week_day, hour, minute, second, isnull, search, regex
Suppose you have a query that needs to run against a table or tables with many millions of rows. Maybe you need
to operate on a couple million of them. What are the do’s and don’t’s of a Django query that will not pessimize
performance (time and memory use)?
• Don’t bother with .iterator(), it downloads the whole result and then iterates over it. It does not do what many
of us think/thought it did (use a database cursor to pull down the results only as you work through them)
• Do limit the query ([start:end]) and run it repeatedly in reasonable sized batches, to avoid downloading too big
a chunk
• Do use .only() and its kin to minimize how much of each record is downloaded to what you need
• Do not order_by() unless you must - it forces the DB to collect the results of the whole query first so it can sort
them, even if you then limit the results you retrieve
• Same for .distinct().
queryset.model
Combining querysets
Given two querysets over the same model, you can do things like this:
queryset = queryset1 & queryset2
queryset = queryset1 | queryset2
queryset = queryset1 & ~queryset2
(similar to Q objects)
Custom QuerySets
class BaseManager(....):
....
class MyModel(models.Model):
objects = BaseManager.from_queryset(CustomQuerySet)()
Custom Lookups
Adding to the kwargs you can pass to filter and exclude etc.
Custom Lookups
Security
70 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
Django
(django-secure appears to be abandoned. Last change was in 2014, and it doesn’t load under Django 1.11/Python 3.6.)
-Best practice: install django-secure and run manage.py checksecure to make sure all the right settings
are enabled.-
See also OWASP.
Admin
SSH
SSL
Django security
djangocon 2011 Paul McMillan
http://subversivecode.com/talks/djangocon-us-2011
HSTS
django needs better password hash (SHA1 not broken but very fast)
OpenID much more secure against password cracking (because cracker won’t have passwords)
password reset strings can be eventually worked out with a timing attack (if you have a long time and a fast connection)
same for which userids exist on the site
you should do rate limiting:
mod_evasive (apache) HttpLimitReqModule (nginx)
do NOT use random.Random() for security functions, not cryptographically secure; use random.SystemRandom()
instead e.g.:
from random import SystemRandom as random
xxxx random.choice(yyyy)...
6.21. Security 71
Dan’s Cheat Sheets Documentation, Release 1
Be very careful with pickle, it’ll execute anything in the pickled data when you unpickle it
BOOK: The web application hacker’s handbook (new version coming out soon (as of 9/8/2011))
SITE: lost.org? (not sure I heard that right)(no I didn’t)
Settings
Baumgartner suggests symlinking the desired one (e.g. dev.py or deploy.py) to local.py and hard-coding that in man-
age.py.
Greenfelds suggest... (FILL THIS IN)
12-factor says there should only be one settings file, and any values that vary by deploy should be pulled from the
environment. See Env vars.
Secret key
(https://github.com/django/django/blob/master/django/core/management/commands/startproject.py#L26)
Env vars
You can load them into Django using dotenv. Pop open manage.py. Add:
import dotenv
dotenv.read_dotenv()
Or in a settings file:
SECRET_KEY = os.environ.get("SECRET_KEY")
72 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
Templates
Example usage:
{% set foo="bar" a=1 %}
...
Hello everyone, foo is {{ foo }} and a is {{ a }}.
register = template.Library()
@register.simple_tag(takes_context=True)
def set(context, **kwargs):
context.update(kwargs)
return ''
register = template.Library()
is rendered as
6.23. Templates 73
Dan’s Cheat Sheets Documentation, Release 1
# Parse out the arg1=foo arg2=bar ... arguments from the arg list into a dictionary.
# kwargs will have "arg1" etc as keys, while the values will be
# template thingies that can later be rendered using different contexts
# to get their value at different times.
kwargs = token_kwargs(args, parser)
# If there are any args left, we have a problem; this tag only
# accepts kwargs.
if args:
raise TemplateSyntaxError("%r only accepts named kwargs" % tag_name)
# Open in new tab unless otherwise told (by setting target to something else).
if 'target' not in kwargs:
kwargs['target'] = parser.compile_filter('"_blank"')
register.tag('detail_link', do_detail_link)
class DetailLinkNode(template.Node):
"""
Stores info about one occurrence of detail_link in a template.
74 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
)
return result
The normal error message when a view fails rendering a template during testing gives no clue where the error is.
You can get a better idea by temporarily editing your local Django installation. Find the file
django/template/base.py. Around line 194 (in Django 1.8.x), in the __init__ method of the Template
class, look for this code:
self.nodelist = engine.compile_string(template_string, origin)
TODO: would be nice to get a line number too (this just gives a filename, which is often enough in combination with
the error message).
Testing
self.assertEqual(a, b, msg=None)
assert self.client.login(**login_parms)
login(email='[email protected]', password='cleartextpassword')
# http://docs.python.org/library/unittest.html # https://docs.djangoproject.com/en/stable/topics/testing/
from django.test import TestCase
from django.contrib.auth.models import User
class XxxTest(TestCase):
def setUp(self):
self.user = User.objects.create_user('tester', '[email protected]', 'testpass')
def test_something(self):
response = self.client.get(show_timemap)
self.assertEqual(response.status_code, 200)
self.assertEqual(response.context['lat'], '123.123')
self.assertEqual(response.context['lon'], '456.456')
self.assertContains(response, "some text")
self.assertNotContains(response, "other text")
6.24. Testing 75
Dan’s Cheat Sheets Documentation, Release 1
self.assertIsNone(val)
self.assertIsNotNone(val)
self.assertIn(thing, iterable) # Python >= 2.7
self.assertNotIn(thing, iterable) # Python >= 2.7
(The name attachment here is not relevant; use whatever name your file-processing code expects.)
Note that if you wish to use the same file handle for multiple post() calls then you will need to manually reset the file
pointer between posts. The easiest way to do this is to manually close the file after it has been provided to post(), as
demonstrated above.
You should also ensure that the file is opened in a way that allows the data to be read. If your file contains binary data
such as an image, this means you will need to open the file in rb (read binary) mode.
# setup.py:
...
setup(
...
test_suite="runtests.runtests",
...
)
# runtests.py:
#!/usr/bin/env python
import os
import sys
if not settings.configured:
settings.configure(
DATABASES={
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:',
}
},
INSTALLED_APPS=(
'selectable',
),
SITE_ID=1,
SECRET_KEY='super-secret',
ROOT_URLCONF='selectable.tests.urls',
76 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
def runtests():
TestRunner = get_runner(settings)
test_runner = TestRunner(verbosity=1, interactive=True, failfast=False)
args = sys.argv[1:] or ['selectable', ]
failures = test_runner.run_tests(args)
sys.exit(failures)
if __name__ == '__main__':
runtests()
Translation
URLs
reverse
redirect
6.25. Translation 77
Dan’s Cheat Sheets Documentation, Release 1
Examples
2. By passing the name of a view and optionally some positional or keyword arguments; the URL will be reverse
resolved using the reverse() method:
def my_view(request):
...
return redirect('some-view-name', foo='bar')
By default, redirect() returns a temporary redirect. All of the above forms accept a permanent argument; if set
to True a permanent redirect will be returned:
def my_view(request):
...
object = MyModel.objects.get(...)
return redirect(object, permanent=True)
https://docs.djangoproject.com/en/stable/topics/http/urls/
from django.conf.urls.defaults import patterns, include, url
from django.contrib import admin admin.autodiscover()
urlpatterns = patterns('',
(r'^polls/', include('polls.urls')),
url(r'^admin/', include(admin.site.urls)),
)
urlpatterns = patterns('polls.views',
(r'^, 'index'),
(r'^(?P<poll_id>\d+)/, 'detail'),
(r'^(?P<poll_id>\d+)/results/, 'results'),
(r'^(?P<poll_id>\d+)/vote/, 'vote'),
url(r'^(?P<poll_id>\d+)/foo/, 'fooview', name='app-viewname'),
)
Misc to file:
Avoid circular imports:
78 Chapter 6. Django
Dan’s Cheat Sheets Documentation, Release 1
6.26. URLs 79
Dan’s Cheat Sheets Documentation, Release 1
80 Chapter 6. Django
CHAPTER 7
Elasticsearch
Documentation
• Elasticsearch Reference
• Elasticsearch Definitive Guide
Contents:
Indices
Create an index
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"type1" : {
"_source" : { "enabled" : false },
"properties" : {
"field1" : { "type" : "string", "index" : "not_analyzed" }
}
}
}
}
Index settings
TBD
81
Dan’s Cheat Sheets Documentation, Release 1
Mappings
Mappings API
Mappings reference
Intro to Mappings in the Guide
Types and Mappings in the Guide
Example mapping for a tweet doctype:
{
"tweet" : {
"properties" : {
"message" : {
"type" : "string",
"store" : true,
"index" : "analyzed",
"null_value" : "na"
},
"user" : {
"type" : "string",
"index" : "not_analyzed",
"norms" : {
"enabled" : false
}
},
"postDate" : {"type" : "date"},
"priority" : {"type" : "integer"},
"rank" : {"type" : "float"}
}
}
}
82 Chapter 7. Elasticsearch
Dan’s Cheat Sheets Documentation, Release 1
Multi fields
Object type
Analysis
Analysis reference
Analysis and Analyzers in the Guide
An analyzer consists of:
• Zero or more CharFilters
• One Tokenizer
• Zero or more TokenFilters
Analysis can be configured when creating an index with the top-level analysis key in the API argument. Example:
analysis :
analyzer :
standard :
type : standard
stopwords : [stop1, stop2]
myAnalyzer1 :
type : standard
7.1. Documentation 83
Dan’s Cheat Sheets Documentation, Release 1
Built-in analyzers
Custom analyzers
84 Chapter 7. Elasticsearch
Dan’s Cheat Sheets Documentation, Release 1
type : html_strip
escaped_tags : [xxx, yyy]
read_ahead : 1024
7.1. Documentation 85
Dan’s Cheat Sheets Documentation, Release 1
86 Chapter 7. Elasticsearch
CHAPTER 8
Elixir
Basic types
Examples:
iex> 1 # integer
iex> 0x1F # integer (31)
iex> 0x1010 # integer (10)
iex> 0o777 # integer (511)
iex> 1.0 # float
iex> 1.0e-10 # float
iex> true # boolean
iex> false # boolean
iex> :atom # atom / symbol
iex> "elixir" # string
iex> [1, 2, 3] # list
iex> {1, 2, 3} # tuple
iex> is_atom(false) # true
iex> is_atom(:false) # true
iex> is_boolean(true) # true
iex> is_boolean(:true) # true
iex> is_integer(1.0) # false
iex> is_float("foo")
iex> is_number(1.0) # true
Arithmetic
87
Dan’s Cheat Sheets Documentation, Release 1
Boolean expressions
Strings
Double quoted literals are strings (single quoted literals are char lists, not strings).
Use <> to concatenate, e.g. "hello" <> " world"
You can interpolate:
iex> "hellö #{:world}"
"hellö world"
Functions
Note: Functions in Elixir are identified by name and by number of arguments (i.e. arity). Therefore, is_boolean/1
identifies a function named is_boolean that takes 1 argument. is_boolean/2 identifies a different (nonexistent) function
with the same name but different arity.:
88 Chapter 8. Elixir
Dan’s Cheat Sheets Documentation, Release 1
iex> is_boolean(true)
true
iex> is_boolean(1)
false
BUT you can’t call a function using its full name and arity, you have to leave off the arity:
iex> is_boolean/1(1)
** (SyntaxError) iex:2: syntax error before: '('
Anonymous functions
Anonymous functions are closures and can access variables that were in scope when they were defined.
Variables assigned inside a function do not affect the surrounding environment, though:
iex> x = 42
42
iex> (fn -> x = 0 end).()
0
iex> x
42
Lists
Literal lists are written with square brackets. Values can be a mix of any types:
iex> length [1, 2, true, 3]
4
Lists are concatenated using ++/2 and can be “subtracted” using --/2:
The “head” of a list is like Lisp’s car but is accessed using the hd/1 function. Similarly, the “tail” is the cdr and
you get it with tl/1:
iex> list = [1,2,3]
iex> hd(list)
1
iex> tl(list)
[2, 3]
A list of small integers is printed by Elixir as a single-quoted “string” - but it’s not a string, it’s a list of chars:
iex> [11, 12, 13]
'\v\f\r'
iex> [104, 101, 108, 108, 111]
'hello'
Introspection
90 Chapter 8. Elixir
Dan’s Cheat Sheets Documentation, Release 1
Description
This is a string: a UTF-8 encoded binary. It's printed surrounded by
"double quotes" because all UTF-8 encoded codepoints in it are printable.
Raw representation
<<104, 101, 108, 108, 111>>
Reference modules
String, :binary
Tuples
Literal tuples are written with curly brackets {1, :ok, true}. Access any element with elem/2 using 0-
indexing, get the length with tuple_size/1, and return a new tuple with an element changed using put_elem/3:
iex> elem({:ok, "hello"})
"hello"
iex> tuple_size({:ok, "hello"})
2
iex> put_elem({:ok, "hello"}, 1, "world"})
{:ok, "world"}
8.9. Tuples 91
Dan’s Cheat Sheets Documentation, Release 1
92 Chapter 8. Elixir
CHAPTER 9
Git
Fetching
Update/create local branch origin/master from remote origin’s branch master with default configuration:
git fetch origin master
Update/create local branch ‘tmp’ from remote origin’s branch master (but only updates if fast-forward is possible):
get fetch origin master:tmp
Peek at an artitrary remote’s branch by pulling it into a (temporary) local branch, then check its log. The temporary
local branch will eventually be garbage collected:
git fetch git://git.kernel.org/pub/scm/git/git.git maint
git log FETCH_HEAD
93
Dan’s Cheat Sheets Documentation, Release 1
http://stackoverflow.com/questions/1683531/how-to-import-existing-git-repository-into-another
Cleaning
Prune branches that have been merged and are no longer upstream:
http://devblog.springest.com/a-script-to-remove-old-git-branches
Prune branches that track remote branches that no longer exist (http://kparal.wordpress.com/2011/04/15/git-tip-of-the-
day-pruning-stale-remote-tracking-branches/):
$ git remote prune origin --dry-run
$ git remote prune origin
Pulls
Aliases
cb = checkout -b
cd = checkout develop
co = checkout
gd = !git fetch origin && git checkout develop && git pull origin develop
gm = !git fetch origin && git checkout master && git pull origin master
# push -f
pf = push -f
94 Chapter 9. Git
Dan’s Cheat Sheets Documentation, Release 1
# Find the common ancestor of HEAD and develop and show a diff
# from that to HEAD
dd = "!git diff $(git merge-base develop HEAD)"
# Find the common ancestor of HEAD and master and show a diff
# from that to HEAD
dm = "!git diff $(git merge-base master HEAD)"
Submodules
Suppose you have branch A and branch B, which branched off of master at various times, and you want to create a
branch C that contains the changes from both A & B.
According to Calvin: checkout the first branch, then git checkout -b BRANDNEWBRANCH. then rebase it on the
second.
(SEE DIAGRAMS BELOW)
Example:
# Start from master
git checkout master
git pull [--rebase]
# rebase A on master
git checkout A
9.7. Submodules 95
Dan’s Cheat Sheets Documentation, Release 1
# rebase B
git checkout B
git rebase -i master
# merge B into C
git checkout C
git merge B
# I think???
# Review before using, and verify the result
Rebase A on master:
master
/
o - o - o - o - o - o - o <--- A
\
o - o - o <--- B
Rebase B on master:
master
/
o - o - o - o - o - o - o - o <--- A
|\
| o - o - o <--- N (includes A)
\
o - o - o <--- B
96 Chapter 9. Git
Dan’s Cheat Sheets Documentation, Release 1
On N, merge B:
master
/
o - o - o - o - o - o - o - o <--- A
|\
| o - o - o - o - o - o <--- N (includes A and B)
\
o - o - o <--- B
98 Chapter 9. Git
CHAPTER 10
Google APIs
Google libraries
Newer libraries
OAuth 2.0
Google APIs client library for Python (beta)
99
Dan’s Cheat Sheets Documentation, Release 1
Haskell
By analogy to Python...
Imports
Math
101
Dan’s Cheat Sheets Documentation, Release 1
Expressions
Lists
Functions
i3
My i3 bindings
“Ŵ” is the “windows” key
Change what we see:
Ŵ-<NUMBER>: switch to workspace NUMBER on whatever monitor it's
attached to.
Ŵ-Control-1: Only use built-in laptop display
Ŵ-Control-2: Use built-in laptop display, and external display
positioned to its right
Ŵ-Control-3: Use built-in laptop display, and external display
positioned above it
Ŵ-<n>: Switch to workspace <n> (need not already exist)
(if workspace <n> is on another screen, it'll switch
that screen to workspace <n>, not your current screen)
Ŵ-<n> where <n> is the current workspace: Switch back to previous
workspace (So you can just do Ŵ-1 (look at screen) Ŵ-1
and be back where you started)
Focus:
Ŵ-j, Ŵ-<left> left
Ŵ-k, Ŵ-<down> down
Ŵ-l, Ŵ-<up> up
Ŵ-;, Ŵ-<right> right
Move things:
Ŵ-Control-<ARROW>: Move current workspace to another monitor.
Ŵ-Shift-Number: Move current window to another workspace
(need not already exist)
Shift-<FOCUS COMMAND>: Move current window within workspace
Layouts:
Ŵ-e default (splith/splitv), repeat to toggle
splith/splitv
Ŵ-s stacked
Ŵ-w tabbed
Ŵ-f fullscreen (toggle)
Ŵ-S-<spc> float (toggle)
Ŵ-mouse1-drag move floating
105
Dan’s Cheat Sheets Documentation, Release 1
Start/end things:
Ŵ-return: open new terminal
Ŵ-D: open dmenu at top to enter a command (output invisible,
use to start new graphical programs)
Ŵ-S-q kill window
Control I3:
Ŵ-S-c reload I3 config
Ŵ-S-r restart I3
Ŵ-S-e kill I3 (logout)
Resizing:
Ŵ-mouse2-drag stretch or shrink window
Screen capture:
<Printscreen> - capture whole screen
Shift-<Printscreen> - select a rectangle or window (?)
Control-<Printscreen> - capture currently focused window
IPv6
Localhost
Localhost is ::1
Any addr
With port
Adding ”:<portnumber>” to an IPv6 address would be ambiguous. The solution is to put “[]” around the address part,
e.g. [::1]:8000.
You need the [] here too, at least if you’re using a hexadecimal IPv6 address. (Even without a port number.):
http://[fe80::b746:6473:e65f:5dd4]/foo/bar
http://[fe80::b746:6473:e65f:5dd4]:8000/foo/bar
Ping
Use ping6:
ping6 ::1
Django
107
Dan’s Cheat Sheets Documentation, Release 1
It does NOT appear possible to have the dev server listen on both IPv4 and IPv6, at least not in Django 1.8. (But I’m
sure you could put nginx in front to do that for you.)
Prefix/L: fd
Global ID: 442da008f4
Subnet ID: cf4f
Combined/CID: fd44:2da0:08f4:cf4f::/64
IPv6 addresses: fd44:2da0:08f4:cf4f:xxxx:xxxx:xxxx:xxxx
If you have multiple locations/sites/networks, you should assign each one a different "Subnet ID", bu
Try to find an “inet6 addr” that isn’t “Scope:Link” or “Scope:Host”; you want “Scope:Global”.
Better way - if you think you have real IPv6 connectivity to the internet, go to google.com and search for “what’s my
IP”. If you’re connecting to Google over IPv6, Google will tell you your IPv6 address (which might be a NATting
router, I suppose - I don’t know if IPv6 still does that?) But that won’t help if you’re just running IPv6 locally for
testing.
Javascript
Contents:
Javascript Builtins
Arrays
109
Dan’s Cheat Sheets Documentation, Release 1
See if none, some, or all elements pass a test (ES5+) (every, some):
all_passed = arr.every(function(value, index, arr){ return bool;}[, thisArg])
some_passed = arr.some(function(value, index, arr){ return bool;}[, thisArg])
none_passed = !arr.some(...)
Create a new array by applying a function to each element of an existing array (ES5+) (map):
arr2 = arr1.map(function(value, index, arr){}[, thisArg])
Create a new array only including elements that pass a test (ES5+ (MDN):
arr2 = arr1.filter(function(value, index, arr){}[, thisArg])
Note: For objects that you’re just using as dictionaries, for...in is perfectly good. Just be careful not to use it on
instances of anything more complicated.
For more complicated objects, here’s how you get just properties that are directly on the object (not inherited from a
prototype). Traditional:
for (var key in object) {
if (object.hasOwnProperty(key)) {
value = object[key]
# ... do stuff ...
}
}
Is a key in a dictionary?
dict.hasOwnProperty(key)
String operations
String, endsWith, includes, indexOf, join, match, replace, search, split, startsWith, substr, substring
arr = str.split(sep=(str|regex)[, limit])
str = arr.join(sep)
index = str.indexOf(substr[, startIndex]) # returns -1 if not found
sub = str.substr(firstIndex[, length]) # firstIndex can be negative to count back from end
sub = str.substring(firstIndex[, lastIndex+1])
str2 = str.replace(regexp|substr, newSubStr|function)
bool = str.startsWith(str2)
bool = str.includes(str2)
bool = str.endsWith(str2)
[matchstr, groups...] = str.match(regexp) # returns null if doesn't match
[matchstr, groups...] = str.search(regexp) # returns null if doesn't match entire string
Contains: haystack.indexOf(needle) != -1
Timer
setTimeout:
window.setTimeout(func, delay, param1, param2, ...);
Javascript Events
Event object
• currentTarget: the element that has the handler attached that is currently running
• target: the element that received the event initially
Promises
MDN docs
quoting from there:
A Promise is in one of these states:
• pending: initial state, not fulfilled or rejected.
• fulfilled: meaning that the operation completed successfully.
• rejected: meaning that the operation failed.
A promise that is not pending is “settled”.
You can create a Promise with new, passing it an executor function:
At any time after the promise is created, you can call then on it and pass in a success handler, a failure handler, or
both:
p.then(on_success) p.then(undefined, on_failure) p.then(on_success, on_failure)
If the promise is still pending, the appropriate handler will be called once it is settled. If it’s settled already, the handler
will be calleed immediately (or ASAP, anyway).
The on_success handler is called with the resolved value from the promise, while the on_failure handler is
called with the error value from the promise.
then() returns a new promise, so you can chain calls:
p.then(on_success).then(another_on_success).then(a_third_on_success)
If the on_success handler returns a new promise, that promise will be the return value from then() (or an equivalent
promise, anyway).
The handlers will be called in order. If one fails, then the promise returned by that then() call will be rejected. “Fail”
can mean raises an exception.
Many async functions nowadays return promises.
Pausing
// USE a promise
promise.then(function(result) {
// use result here
}, function(error) {
// do something with error here
});
// CHAINING
// Note that 'then' returns a new promise
promise.then(function(val1) {
return val1 + 2;
}).then(function(val2) {
val2 is here val1 + 2!!!
});
// MORE CHAINING
promise.then(function(result){
// do something, then return a new promise
return promise2;
}).then(function(result2) {
// this isn't invoked until promise2 resolves, and it gets promise2's result.
...
}
Javascript Syntax
Spread Operator
Spread operator
Given function and args:
function myFunction(x, y, z) { }
var args = [0, 1, 2];
Traditional:
myFunction.apply(null, args);
ES2015:
myFunction(...args);
myFunction(1, ...[2, 3]);
Caution: The ES2016 ... operator is NOT the same as * in a Python function call. ... basically splices the array
it’s applied to into the list at the place where it’s used. It can be used repeatedly, and in any combination with other
unnamed arguments. Python’s * can only be used to extend the list of unnamed arguments at the end.
LXDE
115
Dan’s Cheat Sheets Documentation, Release 1
Logitech Harmony
Install Concordance
• Run ‘lsusb’ to see what devices are already attached to your computer.
• Plug in your remote
• Run ‘lsusb’, looking for the device that wasn’t there before. E.g. my Harmony One produced this:
Bus 001 Device 021: ID 046d:c121 Logitech, Inc.
117
Dan’s Cheat Sheets Documentation, Release 1
• Now (using sudo as needed) create a new file, /etc/udev/rules.d/custom-concordance.rules, substituting in the
values for your own remote:
ATTRS{idVendor}=="046d", ATTRS{idProduct}=="c121", MODE="666"
• Test
– Plug the remote in
– Run “concordance -i”. It should print out information about the attached remote
MPD
Fixing on Ubuntu
On Ubuntu, mpd is installed broken. You’ll have to go read this page and do some work to fix things so mpd will
actually work on Ubuntu. Sigh.
Important: To play anything, you must first get it into the current playlist. Even if you just want to play one thing.
Playlist commands
Playing things
Status:
playlist [-f <format>] List songs in current playlist. See “man mpc” for format string syntax.
current Show what’s playing right now
119
Dan’s Cheat Sheets Documentation, Release 1
Control:
play [<position>] Start playing the item at <position> in the playlist. Deafult is number 1.
pause Pause playing
stop Stop playing
next Stop playing the current entry from the current playlist and start playing the next one.
prev Reverse of next.
Suppose you have a playlist in the database already - e.g. a file “/var/lib/mpd/playlists/WCPE.m3u” that you’ve created
earlier.
Now you want to play that playlist.
$ mpc clear
$ mpc lsplaylists
WUNC
WCPE
Favorites
$ mpc load WCPE
loading: WCPE
$ mpc play
volume: 96% repeat: on random: off single: off consume: off
loading: WCPE
http://audio-ogg.ibiblio.org:8000/wcpe.ogg
[playing] #1/1 0:00/0:00 (0%)
volume: 96% repeat: on random: off single: off consume: off
Django:
pip install mysqlclient
DATABASES['default'] = {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'dbname',
'USER': 'username',
}
In the client:
mysql> SELECT user, host from mysql.user; # List existing users
mysql> CREATE USER 'username' IDENTIFIED BY 'plaintextpassword'; # Create user with password
mysql> CREATE USER 'username'@'localhost'; # no password, can only connect locally
mysql> SHOW DATABASES;
mysql> CREATE DATABASE databasename;
mysql> GRANT ALL ON databasename.* TO "username"@"hostname" IDENTIFIED BY "password";
mysql> FLUSH PRIVILEGES;
mysql> DROP DATABASE databasename;
mysql> DROP USER username;
mysql> EXIT
Bye
Note: default host is ‘%’ which will not let you connect via unix socket, must set password for host ‘localhost’ to
allow that:
121
Dan’s Cheat Sheets Documentation, Release 1
http://dev.mysql.com/doc/refman/5.5/en/resetting-permissions.html
C.5.4.1.3. Resetting the Root Password: Generic Instructions On any platform, you can set the new password using
the mysql client:
Stop mysqld
Restart it with the --skip-grant-tables option. This enables anyone to connect without a password and
$ mysql
mysql> UPDATE mysql.user SET Password=PASSWORD('MyNewPass') WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> EXIT
Make a dump:
mysqldump _dbname_ > dumpfile.sql
mysqldump --result-file=dumpfile.sql _dbname_
Restore a dump:
mysql dbname < dumpfile.sql
Step by step:
$ mysql -u root -p
<ENTER MYSQL ROOT PASSWORD>
mysql> create user 'ctsv2_TR'@'localhost';
mysql> create database ctsv2_TR;
mysql> grant all on ctsv2_TR.* to 'cstv2_TR'@'localhost';
mysql> flush privileges;
mysql> exit
Bye
This is just a matter of piping the dump into the mysql client:
gunzip <ctsv2_TR_20140602.sql.gz | mysql -u ctsv2_TR ctsv2_TR
NPM
source.
123
Dan’s Cheat Sheets Documentation, Release 1
OpenSSL
End-user Functions
Create key
OpenSSL will prompt for the password to use. Algorithms: AES (aes128, aes192 aes256), DES/3DES (des, des3).
Remove passphrase from a key:
openssl rsa -in server.key -out server-without-passphrase.key
Getting Certificates
125
Dan’s Cheat Sheets Documentation, Release 1
-x509 identifies it as a self-signed certificate and -set_serial sets the serial number for the server certificate.
Create a single file that contains both private key and the self-signed certificate:
openssl req -x509 -nodes -days 365 -newkey rsa:1024 \
-keyout myServerCert.pem -out myServerCert.pem \
-subj "/C=US/ST=MA/L=Burlington/CN=myHost.domain.com/[email protected]"
X.509 certificates are usually stored in one of two formats. Most applications understand one or the other, some
understand both:
• DER which is raw binary data.
• PEM which is a text-encoded format based on the Privacy-Enhanced Mail standard (see RFC1421). PEM-format
certificates look something like this:
-----BEGIN CERTIFICATE-----
MIIBrjCCAWwCAQswCQYFKw4DAhsFADBTMQswCQYDVQQGEwJBVTETMBEGA1UECBMK
U29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMQww
:
:
MQAwLgIVAJ4wtQsANPxHo7Q4IQZYsL12SKdbAhUAjJ9n38zxT+iai2164xS+LIfa
C1Q=
-----END CERTIFICATE-----
OpenSSL uses the PEM format by default, but you can tell it to process DER format certificates..
You can specifiy -inform pem if you want to look at a PEM-format certificate
If you have a PEM-format certificate which you want to convert into DER-format, you can use the command:
openssl x509 -in filename.pem -inform pem -out filename.cer -outform der
PKCS12 files
PKCS12 files are a standard way of storing multiple keys and certificates in a single file. Think of it like a zip file for
keys & certificates, which includes options to password protect etc.
Don’t worry about this unless you need it because some application requires a PKCS12 file or you’re given one that
you need to get stuff out of.
Viewing PKCS12 Keystore Contents:
openssl pkcs12 -in filename.p12 -info
If you have two separate files containing your certificate and private key, both in PEM format, you can combine these
into a single PKCS12 file using the command:
openssl pkcs12 -in cert.pem -inkey key.pem -export -out filename.p12
Signing E-mails:
openssl smine -sign -in msg.txt -text -out msg.encrypted -signer blah.crt.pem -inkey blah.key.pem
Verify signature:
if openssl dgst -verify public.key -signature signature.asc ; then echo GOOD; else echo BAD; fi
When setting up a new CA on a system, make sure index.txt and serial exist (empty and set to 01, respectively), and
create directories private and newcert.
Edit openssl.cnf - change default_days, certificate and private_key, possibly key size (1024, 1280, 1536, 2048) to
whatever is desired.
Create CA Certificate:
openssl req -new -x509 -keyout private/something-CA.key.pem \
-out ./something-CA.crt.pem -days 3650
Revoke Certificate:
openssl ca -revoke blah.crt.pem
Use only when you’ve no CA and will only be generating one key/certificate (useless for anything that requires signed
certificates on both ends)
C-c l org-store-link
C-c c org-capture
C-c b org-iswitchb (?)
<TAB> org-cycle
S-<TAB> org-global-cycle
129
Dan’s Cheat Sheets Documentation, Release 1
TBD
Postfix
131
Dan’s Cheat Sheets Documentation, Release 1
Postgres
Snippets
• In psql:
# \d *pattern*
Display definition of table, view, or other relations with name matching the pattern
See http://jacobian.org/writing/pg-encoding-ubuntu/ to fix postgres to default to UTF-8 on ancient Ubuntus. (Destroys
data.)
To set up your user on ubuntu to auth to local postgres automatically https://help.ubuntu.com/community/PostgreSQL
Check replication:
ON master, select * from pg_stat_replication;
(from http://www.dansketcher.com/2013/01/27/monitoring-postgresql-streaming-replication/)
Environment Variables
The following environment variables can be used to select default connection parameter values, which will be used
if no value is directly specified. These are useful to avoid hard-coding database connection information into simple
client applications, for example.
PGHOST behaves the same as host connection parameter.
PGHOSTADDR behaves the same as hostaddr connection parameter. This can be set instead of or in
addition to PGHOST to avoid DNS lookup overhead.
PGPORT behaves the same as port connection parameter.
PGDATABASE behaves the same as dbname connection parameter.
PGUSER behaves the same as user connection parameter.
PGPASSWORD behaves the same as password connection parameter. Use of this environment variable
is not recommended for security reasons (some operating systems allow non-root users to see process
environment variables via ps); instead consider using the ~/.pgpass file (see Section 30.14).
PGPASSFILE specifies the name of the password file to use for lookups. If not set, it defaults to ~/.pgpass
(see Section 30.14).
133
Dan’s Cheat Sheets Documentation, Release 1
How do we do things on Postgres without giving superuser to the user that actually uses the database every day?
The following assumes a Postgres superuser named ‘master’. (Or the RDS ‘master’ user, who has most superuser
privileges.)
In the examples below, for readability I’m omitting most of the common arguments to specify where the postgres
server is, what the database name is, etc. You can set some environment variables to use as defaults for things:
export PGDATABASE=dbname
export PGHOST=xxxxxxxxx
export PGUSER=master
export PGPASSWORD=xxxxxxxxxx
Create user
This is pretty standard. To create user $username with plain text password $password:
export PGUSER=master
export PGDATABASE=postgres
createuser -DERS $username
psql -c "ALTER USER $username WITH PASSWORD '$password';"
Yes, none of the options in -DERS are strictly required, but if you don’t mention them explicitly, createuser asks you
about them one at a time.
If not on RDS, for the user to actually do something useful like connect to postgres, you might also have to edit
pg_hba.conf and add a line like:
local <dbname> <rolename> md5
to let it connect using host=” (unix domain socket) and provide a password to access <dbname>. You could also put
“all” there to let it access any password it otherwise has auth for. E.g. to allow local connections via both unix socket
and tcp connections to localhost:
local all all md5
host all all 127.0.0.1/32 md5
Create database
• Create it as any other user, so long as the other user is a member, direct or indirect, of the $project_user
role. That suggests that we could add master to that role... need to research that. I think we could do:
export PGUSER=master
psql -c "grant $project_user to master;" postgres
createdb --template=template0 --owner=$project_user $dbname
The question would be: Does master have enough privileges to grant itself membership in another role?
• Finally, you could create it as master when master is not a member of the project_user role. To do that, you’ll
need to create it as master and then modify the ownership and permissions:
export PGUSER=master
createdb --template=template0 $dbname
psql -c "revoke all on database $dbname from public;"
psql -c "grant all on database $dbname to master;"
psql -c "grant all on database $dbname to $project_user;"
If you need to enable extensions etc, do that now (see below). When done, then:
psql -c "alter database $dbname owner to $project_user;"
A superuser could create the database already owned by a specific user, but RDS’s master user cannot.
PostGIS
where $project_user is the postgres user who will be using the database.
(Outside of RDS, only a superuser can use create extension; RDS has special handling for a whitelist of
extensions.)
Hstore
Hstore is simpler, but you still have to use the master user:
export PGUSER=master
psql -c "create extension hstore;"
Create the database as above, including changing ownership to the project user, and enabling any needed extensions.
Then as the project user:
export PGUSER=$project_user
pg_restore --no-owner --no-acl --dbname=$dbname file.dump
Note that you might get some errors during the restore if it tries to create extensions that already exist and that kind of
thing, but those are harmless. It does mean you can’t use --one-transaction or --exit-on-error for the
restore though, because they abort on the first error.
Drop database
When it comes time to drop a database, only master has the permission, but master can only drop databases it owns,
so it takes two steps. Also, you can’t drop the database you’re connected to, so you need to connect to a different
database for the dropdb. The postgres database is as good as any:
export PGUSER=master PGDATABASE=postgres
psql -c "alter database $dbname owner to master;"
psql -c "drop database if exists $dbname;"
(Outside of RDS, a superuser can drop any database. A superuser still has to be connected to some other database
when doing it, though.)
Drop user
This is standard too. Just beware that you cannot drop a user if anything they own still exists, including things like
permissions on databases.:
$ export PGUSER=master
$ dropuser $user
Postgres on RDS
Python
Contents:
Asyncio
What is it
asyncio is a library included in Python 3.5 that supports a programming model where sometimes, operations that
would normally block the thread until some other event happened (like getting a response from a network connection)
instead allow other code to run on that thread while waiting.
asyncio takes a very, very explicit approach to asynchronous programming: only code written in methods flagged as
async can call any code in an asynchronous way.
Which creates a chicken/egg problem: your async methods can only be called by other async methods, so how do you
call the first one?
The answer: you don’t. What you have to do instead is turn over control of the thread to an event loop, after arranging
for the loop to (sooner or later) invoke your async code.
Then once you start the loop running, it can invoke the async code.
What good is it
Note first that you can use threads to accomplish the same things as asyncio in most cases, with better performance.
So what good is asyncio?
For one thing, it leads to more straightforward code than managing multiple threads, protecting data structures from
concurrent access, etc. There’s only one thread and no preemptive multitasking.
If you want to play with async programming in Python, asyncio looks easier to work with and understand than Twisted,
but that’s not a very practical reason.
More significantly, threads won’t scale as well if you need to wait for many, many things at the same time - asyncio
might be somewhat slower, but might be the only way that some tasks can be run at all. Each thread can take 50K of
memory, while a coroutine might take only 3K.
139
Dan’s Cheat Sheets Documentation, Release 1
Event loops
Async code can only run inside an event loop. The event loop is the driver code that manages the cooperative multi-
tasking.
(I think) a typical pattern would be to get or create an event loop, set up some things to be run by it, then start the event
loop running and have it run until the program is finished.
If it’s useful for some reason, you can create multiple threads and run different event loops in each of them. For
example, Django uses the main thread to wait for incoming requests, so we can’t run an asyncio event loop there, but
we can start a separate worker thread for our event loop.
Coroutines
coroutines
• Python distinguishes between a coroutine function and a coroutine object
• Write a coroutine function by putting async in front of the def.
• Only a coroutine function can use await, non-coroutine functions cannot.
• Calling a coroutine function does not execute it, but rather returns a coroutine object. (This is analogous to
generator functions - calling them doesn’t execute the function, it returns a generator object, which we then use
later.)
• To execute a coroutine object, either:
– use it in an expression with await in front of it, or
– schedule it with ensure_future() or create_task().
Example with await:
async def coro_function():
return 2 + 2
coro = coro_function()
# not executed yet; coro is a coroutine, not 4
print(await coro)
# prints "4"
coro = coro_function("example.com")
asyncio.ensure_future(coro)
Of course, usually you wouldn’t split it onto two lines with a temp variable:
asyncio.ensure_future(coro_function("example.com"))
or:
asyncio.get_event_loop().create_task(coro_function("example.com"))
Futures
A future is an object that represents something uncompleted. It makes it easy for code in one place to indicate when
the work is done, and optionally what the result was, and for code elsewhere that was interested in it to find out about
it.
In other words, you can use future objects to manage synchronization more explicitly.
Create one on the fly by calling loop.create_future():
future = loop.create_future()
You can add lots of callbacks. They’ll all be called (one at a time).
The callback receives the future object as an argument. Use functools.partial as usual if you want to pass other
arguments.
When the future is done, mark it done and set its result:
future.set_result(value)
The callbacks can call future.result() to find out what the result was if they care.
Tasks
A Task is a way to arrange for a coroutine to be executed by an event loop, while also providing the caller a way to
find out what the result was.
A task is automatically scheduled for execution when it is created.
There are two ways to do this, which seem equivalent as far as I can tell:
future = loop.create_task(coroutine)
future = asyncio.ensure_future(coroutine[, loop=loop])
Also, if the loop isn’t already running and you just want to run the loop for this one thing, you can now:
loop.run_until_complete(future)
Awaitables
Coroutine objects and future objects are called awaitables - either can be used with await.
Note: You can only invoke an awaitable once; after that, it’s completed, done, it runs no more.
Event loops
Creating/getting one
• To get the current thread’s default event loop object, call asyncio.get_event_loop()
• get_event_loop will not create an event loop object unless you’re on the main thread, and otherwise will raise
an exception if the current thread doesn’t have a default loop set.
• To create a new event loop: new_event_loop()
• To make a loop the default loop for the current thread: set_event_loop(loop)
So, to use an event loop in the main thread, you can just do:
loop = asyncio.get_event_loop()
# use loop....
But to run an event loop in another thread, you would do something like:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# use loop...
You don’t have to set your loop as the thread’s default, though, if you’re willing to pass your loop object to all the
APIs that otherwise use the default loop. But that’s a pain.
Running a loop
If you want a long-running loop that keeps responding to events until it’s told to stop, use loop.run_forever().
If you want to compute some finite work using coroutines and then stop, use loop.run_until_complete(<future or
coroutine>).
Stopping a loop
Use loop.stop().
If you’re doing this from another thread, then you need to use a different method, asyn-
cio.run_coroutine_threadsafe(coro, loop):
future = asyncio.run_coroutine_threadsafe(coroutine, loop)
If you need to call some blocking code from a coroutine, and don’t want to block the whole thread, you can make it
run in another thread using coroutine AbstractEventLoop.run_in_executor(executor, func, *args):
fn = functools.partial(method, *args)
result = await loop.run_in_executor(None, fn)
Sleep
Calling asyncio.sleep(seconds) does not sleep; it returns a coroutine object. When you execute it by invoking it with
await etc, it will complete after <seconds> seconds. So, mostly you’d do:
await asyncio.sleep(10) # pause 10 seconds
Mock
• http://www.voidspace.org.uk/python/mock/mock.html
• https://docs.python.org/3/library/unittest.mock.html
our test.py:
with mock.patch('medley.photos.tasks.galleryimport.urlopen') as urlopen:
urlopen.side_effect = ValueError("Deliberate exception for testing")
Use mock.patch(’python.packagepath.ClassName.method_name:
# medley/photos/models.py
class MedleyPhotoManager(...):
def build_from_url(...):
Test code:
with mock.patch('medley.photos.models.MedleyPhotoManager.build_from_url') as build_from_url:
build_from_url.return_value = None
Python Packaging
here = path.abspath(path.dirname(__file__))
setup(
name='ursonos',
version='0.0.1',
packages=find_packages(),
url='',
license='',
author='poirier',
author_email='[email protected]',
description='Urwid application to control Sonos',
install_requires=[
'soco',
'urwid'
]
)
adds all files matching *.txt or *.rst that are anywhere under subdir/path. Finally:
prune examples/sample?/build
should be obvious.
Wheels
You can also permanently set the --universal flag in “setup.cfg” (e.g., see sampleproject/setup.cfg)
[bdist_wheel]
universal=1
Upload
The docs recommend twine but I haven’t had much luck getting it working, so...
python setup.py sdist bdist_wheel upload
Pip
instead of just:
pip freeze > requirements.txt
Danger: beware of git urls being replaced by egg names in the process.
Timezones in Python
Key points
• Install the pytz package to provide actual time zones. Python doesn’t come with them.
• There are two kinds of datetime objects in Python, and you need to always know which you’re working with:
• naive - has no timezone info. (datetime.tzinfo is None)
• aware - has timezone info (datetime.tzinfo is not None)
• There will always be some things you want to do with datetimes that are just inherently ambiguous. Get used to
it.
Given a formatted time string with timezone, end up with a datetime object
It would be nice if strptime could just parse this and give you an aware datetime object, but no such luck:
>>> fmt = "%a, %d %B %Y %H:%M:%S %Z"
>>> datetime.datetime.strptime(s, fmt)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.py", line 32
(data_string, format))
ValueError: time data 'Tue, 3 July 2012 14:11:03 -0400' does not match format '%a, %d %B %Y %H:%M:%S
>>> fmt = "%a, %d %B %Y %H:%M:%S %z"
>>> datetime.datetime.strptime(s, fmt)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/_strptime.py", line 31
(bad_directive, format))
ValueError: 'z' is a bad directive in format '%a, %d %B %Y %H:%M:%S %z'
That is assuming we know exactly how long the timezone string was, but we might not. Try again:
>>> last_space = s.rindex(' ')
>>> last_space
25
>>> datetime.datetime.strptime(s[:last_space], fmt)
Now, we need to figure out what that timezone string means. Pick it out:
>>> tzs = s[last_space+1:]
>>> tzs
'-0400'
We could have a timezone name or offset, but let’s assume the offset for now. RFC 2282 says this is in the format
[+-]HHMM:
>>> sign = 1
>>> if tzs.startswith("-"):
... sign = -1
... tzs = tzs[1:]
... elif tzs.startswith("+"):
... tzs = tzs[1:]
...
>>> tzs
'0400'
>>> sign
-1
Unfortunately, we can’t just plug that offset into our datetime. To create an aware object, Python wants a tzinfo object
that has more information about the timezone than just the offset from UTC at this particular moment.
So here’s one of the problems - we don’t KNOW what timezone this date/time is from, we only know the current offset
from UTC.
So, the best we can do is to figure out the corresponding time in UTC, then create an aware object in UTC. We know
this time is 240 minutes less than the corresponding UTC time, so:
>>> import time
>>> time_seconds = time.mktime(dt.timetuple())
>>> time_seconds -= 60*minutes
>>> utc_time = datetime.datetime.fromtimestamp(time_seconds, pytz.utc)
>>> utc_time
datetime.datetime(2012, 7, 3, 22, 11, 3, tzinfo=<UTC>)
And there we have it, an aware datetime object for that moment in time.
Virtualenv
http://www.doughellmann.com/docs/virtualenvwrapper:
switch to //envname2//:
workon //envname2//
XML in Python
Like this:
def format_element(element):
"""
Given an etree Element object, return a string with the contents
formatted as an XML file.
"""
tree = ElementTree(element)
bytes = StringIO()
tree.write(bytes, encoding='UTF-8')
return bytes.getvalue().decode('utf-8')
add to top:
import logging
handler = logging.StreamHandler()
root_logger = logging.getLogger('')
root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(handler)
Warnings
Pylint
Running it:
python /usr/bin/pylint --rcfile=.pylintrc -f colorized aremind.apps.patients | less -r
Raspberry Pi
Here are detailed steps to get a Raspberry PI running entirely off a USB drive (after booting from its SD card).
Some sources:
• Move root to USB drive: http://magnatecha.com/using-a-usb-drive-as-os-root-on-a-raspberry-pi/
http://elinux.org/Transfer_system_disk_from_SD_card_to_hard_disk
• UUIDs: http://liquidat.wordpress.com/2013/03/13/uuids-and-linux-everything-you-ever-need-to-know/
Detailed log:
• Start with 4 GB SD card
• Copy 2013-07-26-wheezy-raspbian.img onto it:
sudo dd if=path/2013-07-26-wheezy-raspbian.img of=/dev/mmcblk0 bs=1M
• Partition USB drive:
– Partition 1: system root: 16 GB
– Partition 2: Swap: 1 GB
– Partition 3: /usr/local: rest of drive
• Format the partitions too (ext4)
– sudo mkfs.ext3 -q -m 0 -L ROOT /dev/sdb1
– sudo mkswap -q -L SWAP120 /dev/sdb2
– sudo mkfs.ext3 -q -m 0 -L LOCAL /dev/sdb3
• Boot off 4 GB SD card
• ssh to its IP address (get from router or whatever):
ssh pi@[IP ADDRESS]
password: raspberry
• Copy current root partition to USB drive (see blog post mentioned above to make sure you’re using the right
partitions):
sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=4M
• Resize:
151
Dan’s Cheat Sheets Documentation, Release 1
• Mount:
sudo mount /dev/sda1 /mnt
• Edit config files and change current root partition to the new root UUID in fstab, and /dev/sda1 in cmdline.txt
(cmdline.txt doesn’t support UUID, darn)
– vi /mnt/etc/fstab:
UUID=9c7e2035-df9b-490b-977b-d60f2170889d / ext4 defaults,noatime,async 0 1
– vi /boot/cmdline.txt (http://elinux.org/RPi_cmdline.txt)
• Umount /mnt:
sudo umount /mnt
• Umount:
sudo umount /mnt
Crashplan on Raspberry PI
edit /usr/local/crashplan/bin/CrashPlanEngine and find the line that begins with FULL_CP= (its around the start case)
add /usr/share/java/jna.jar: to the begining of the string. This fixes the cryptic “backup disabled — not available” error.
I ended up with:
FULL_CP=”/usr/share/java/jna.jar:$TARGETDIR/lib/com.backup42.desktop.jar:$TARGETDIR/lang”
• This will backup by default to /usr/local/var/crashplan. We’ll fix that next. But for now, reboot and make sure
crashplan is now running:
sudo reboot ... wait, log back in ... /usr/local/crashplan/bin/CrashPlanEngine status
Nope, it’s stopped. Did I miss a step?
From a crashplan log (/usr/local/crashplan/log/engine_output.log):
Exiting!!! java.lang.UnsatisfiedLinkError: /usr/local/crashplan/libjtux.so: /usr/local/crashplan/libjtux.so:
cannot open shared object file: No such file or directory (Possible cause: can’t load IA 32-bit .so on a
ARM-bit platform).
But that could well be from the initial attempt to start, before we replaced the .so file.
Deleted log file, tried “sudo service crashplan start” again, checked status again... it’s running. So maybe it’s just not
being started at boot? The installer implied it would be...
There’s a rc2.d/S99crashplan which is promising. Try booting again, and wait a sec?
Nope, after several minute, still stopped.
Try this:
sudo update-rc.d crashplan defaults reboot
Yes! It’s running.
now continue with my “Backing up to ReadyNAS using Crashplan on RaspberryPI” simplenote.
This assumes the setup from “USB drive with RaspberryPI” and “Crashplan on Raspberry PI” have already been done.
We have Crashplan running on our Raspberry Pi. We want it to just backup to a share on our ReadyNAS
with no complicated configuration within Crashplan. It looks like a good way to do that might be to just make
/usr/local/var/crashplan be a mount from the NAS. Let’s try that.
Get NFS client working on Raspberry Pi (https://help.ubuntu.com/community/SettingUpNFSHowTo#NFS_Client)
• sudo apt-get install rpcbind nfs-common
• sudo update-rc.d rpcbind enable
• sudo update-rc.d nfs-common enable
• sudo reboot (to make sure it’s all started)
Enable a share for this on the ReadyNAS:
• Login to ReadyNAS admin web
• Go to Services/Standard File Protocols
• Enable NFS and click Apply if it wasn’t enabled already
• go to Shares/Add shares
• enter name=pibackup and a suitable description, turn off Public Access
• Click Apply
• Go to Shares/Share listing
• Next to pibackup, click the NFS icon
• Under Write-enabled hosts, add the Pi’s IP address
Mount it on the Pi for testing:
• sudo showmount -e 10.28.4.2
• sudo mkdir /mnt/pibackup
• sudo mount 10.28.4.2:/c/pibackup /mnt/pibackup
• sudo umount /mnt/pibackup
Now arrange to mount it where crashplan is going to do backups:
• sudo service crashplan stop (for safety)
• sudo -e /etc/fstab
10.28.4.2:/c/pibackup /usr/local/var/crashplan nfs soft,intr,rsize=8192,wsize=8192
• sudo service crashplan start
Configure Crashplan and try it out. First, we’ll do a small backup from here to Crashplan servers, just for sanity.
• Arrange to run the desktop client connected to the Raspberry Pi server (see Crashplan on Headless Boxes)
• Login with your Crashplan account if needed (first time).
• In Settings, give this computer a unique name (poirpi?)
• In Backup/files, select some small set of files to backup - maybe /etc (after showing hidden files)
• In Backup/Destinations, next to CrashPlan Central, click Start Backup. (Enter Archive encryption key if
prompted)
• Wait until the backup is complete
• Exit the desktop
Now, let’s get another computer to backup to here (which is what we really want).
• Run CrashPlanDesktop locally on the source computer
• Go to Settings/Backup tab
• Find a small backup set and select it.
• under Destinations, click Change...
• Find the new Raspberry Pi and click its checkbox
• Click OK
• Click Save
• Go to Backup
• Check the status of the backup of that backup set on the new computer.
reStructuredText
Contents:
.. toctree::
:maxdepth: 2
keys
api
file2.rst
---------
.. _morestuff:
More stuff
==========
To show code::
157
Dan’s Cheat Sheets Documentation, Release 1
.. autoclass:: healthvaultlib.healthvault.HealthVaultConn
:members:
http://techblog.ironfroggy.com/2012/06/how-to-use-sphinx-autodoc-on.html
Various code blocks:
.. code-block:: bash|python|text
:linenos:
:emphasize-lines: 1,3-5
# Hi
# there
# all
# you
# coders
# rejoice
1 # Hi
2 # there
3 # all
4 # you
5 # coders
6 # rejoice
You can include an ‘HTML link‘_ like this and the definition can go nearby or at the bottom of the page:
.. _HTML link: http://foo.bar.com/
Or you can just write ‘HTML link <http://foo.bar.com>‘_ all in one place.
http://sphinx-doc.org/markup/inline.html#ref-role
Link to a filename in this set of docs using :doc:‘Any text you want </path/to/page>‘ or just
:doc:‘path‘.
Don’t include the ”.rst” on the end of the filename. Relative filenames work too. But it’s better to use :ref:, see next.
You can define an anchor point, which Sphinx calls a label. Put this above a section header:
.. _my-label:
My Section
----------
Now from somewhere else, you can write :ref:‘my-label‘ and it’ll be rendered as “My Section” and will link
to the section. If you want some other text in the link, you can write :ref:‘any text <my-label>‘ instead.
159
Dan’s Cheat Sheets Documentation, Release 1
Salt Stack
https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html
In the Jinja2 context, pillar is just a dictionary, so you can use the usual Python dictionary methods, e.g.:
{% for user, uid in pillar.get('users', {}).items() %}
{{user}}:
user.present:
- uid: {{uid}}
{% endfor %}
If you have nested data, it can be easier to use salt[’pillar.get’], which accepts a single key parameter like
key1:key2. E.g. if the pillar data looks like:
apache:
user: fred
instead of:
{{ pillar['apache']['user'] }}
161
Dan’s Cheat Sheets Documentation, Release 1
Tmux
Client:
detach current: d
detach any: D
suspend: C-z
Sessions:
rename $
select s
last L
Windows:
Info: i
Create: c
Kill: &
Switch to: '
Choose: w
Rename: ,
Switch to 0-9: 0-9
Previously selected: l
Next: n
Previous: p
Next window with bell M-n
Pane:
Kill: x
Previous: ;
swap with previous {
swap with next }
Panes:
Split current pane into two:
Top and bottom: -
Left and right: |
163
Dan’s Cheat Sheets Documentation, Release 1
Video
avidemux
GUI tool that’s good for chopping up longer videos into shorter pieces without re-encoding when you need to watch
the video and scan back and forth to find the places you want to split.
makemkv
mkvmerge
CLI only.
Join mkv streams end-to-end without re-encoding. (Inverse of avidemux.)
Split mkv streams at timestamps or chapters without re-encoding.
165
Dan’s Cheat Sheets Documentation, Release 1
Virtualbox
Delete a VM:
VboxManage unregistervm <UUID|VMNAME> [--delete]
167
Dan’s Cheat Sheets Documentation, Release 1
YAML
YAML syntax
A value:
value
A list:
- 1
- 2
- 'a string'
A dictionary:
key1: val1
key2: val2
key3: val3
169
Dan’s Cheat Sheets Documentation, Release 1
A list of dictionaries:
children:
- name: Jimmy Smith
age: 15
- name: Jenny Smith
age 12
• genindex
• modindex
• search
171
Dan’s Cheat Sheets Documentation, Release 1
R
redirect() (built-in function), 77
173