Hy Lisp Python
Hy Lisp Python
This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Setting Up Your Development Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
What is Lisp Programming Style? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Hy is Python, But With a Lisp Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
How This Book Reflects My Views on Artificial Intelligence and the Future of Society
and Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
About the Book Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
A Request from the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Why Lisp? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
I Hated the Waterfall Method in the 1970s but Learned to Love a Bottom-Up Programming
Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
First Introduction to Lisp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Commercial Product Development and Deployment Using Lisp . . . . . . . . . . . . . . . . 21
Hy Macros Let You Extend the Hy Language in Your Programs . . . . . . . . . . . . . . . . . 21
Performing Bottom Up Development Inside a REPL is a Lifestyle Choice . . . . . . . . . . . 22
CONTENTS
Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Simple Multi-layer Perceptron Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Using Keras and TensorFlow to Model The Wisconsin Cancer Data Set . . . . . . . . . . . . 49
Using a LSTM Recurrent Neural Network to Generate English Text Similar to the
Philosopher Nietzsche’s writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Datastores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Sqlite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
PostgreSQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
RDF Data Using the “rdflib” Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Wrap-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
The topics were chosen because of my work experience and the theme of this book is how to increase
programmer productivity and happiness using a Lisp language in a bottom-up development style.
This style relies heavily on the use of an interactive REPL for exploring APIs and writing new code.
I chose the above topics based on my experience working as a developer and researcher. Please note:
you will see the term REPL frequently in this book. REPL stands for Read Eval Print Loop.
Some of the examples are very simple (e.g., the web app examples) while some are more complex
(e.g., Deep Learning and knowledge graph examples). Regardless of the simplicity or complexity of
the examples I hope that you find the code interesting, useful in your projects, and fun to experiment
with.
Preface 2
Depending on which examples you run and experiment with you will also need to install some of
the following libraries:
The Hy language is under active development and it is not unusual for libraries and frameworks
created more than a few months before the current Hy release to break. As a result of this, I have
been careful in the selection of book material to leave out interesting functionality and libraries from
the Hy ecosystem that I feel might not work with new releases. Here we stick with a few popular
Python libraries like Keras, TensorFlow, and spaCy and otherwise we will work with mostly pure
Hy language code in the examples.
I find, in general, that a “bottom-up” approach gets me to working high quality systems faster than
spending too much time doing up front planning and design. The problem with spending too much
up front time on design is that we change our minds as to what makes the most sense to solve a
problem as we experiment with code. I try to avoid up front time spent on work that I will have to
re-work or even toss out.
Usually I use photographs that I take myself for covers of my LeanPub books. Although I have
SCUBA dived since I was 13 years old, sadly I have no pictures of an octopus that I have taken
myself. I did find a public domain picture I liked (that is the cover of this book) on Wikimedia.
Cover Credit: Thanks to Wikimedia user Pseudopanax for placing the cover image in the public
domain.
I enjoy writing and your support helps me write new editions and updates for my books and to
develop new book projects. Thank you!
Acknowledgements
I thank my wife Carol for editing this manuscript, finding typos, and suggesting improvements.
I would like to thank Pascal (Reddit user chuchana) for corrections and suggestions. I would like to
thank Carlos Ungil for catching a typo and reporting it. I would like to thank Jud Taylor for finding
several typo errors.
³https://leanpub.com/u/markwatson
⁴https://markwatson.com/
Introduction to the Hy Language
The Hy programming language⁵ is a Lisp language that inter-operates smoothly with Python. We
start with a few interactive examples that I encourage you to experiment with as you read. Then we
will look at Hy data types and commonly used built-in functions that are used in the remainder of
this book.
I assume that you know at least a little Python and more importantly the Python ecosystem and
general tools like pip.
Please start by installing Hy in your current Python environment:
1 #!/usr/bin/env hy
2
3 (require [hy.contrib.walk [let]])
Line 1 is similar to how we make Python scripts into runnable programs. Here we run hy instead
of python. Line 3 imports the let macro. We will occasionally use let for code blocks with local
variable and function definitions and also for using closures (I will cover closures at the end of this
chapter):
⁵http://docs.hylang.org/en/stable/
Introduction to the Hy Language 6
1 #!/usr/bin/env hy
2
3 (require [hy.contrib.walk [let]])
4
5 (let [x 1]
6 (print x)
7 (let [x 33]
8 (print x)
9 (setv x 44)
10 (print x))
11 (print x))
1
33
44
1
Notice that setting a new value for x in the inner let expression does not change the value bound to
the variable x in the outer let expression.
In the following listing notice how we import other code and libraries in Hy. The special form setv
is used to define variables in a local context. Since the setv statements in lines 3, 5, and 6 are used
at the top level, they are global in the Python/Hy module named after the root name of the source
file.
Introduction to the Hy Language 7
1 $ hy
2 hy 0.18.0 using CPython(default) 3.7.4 on Darwin
3 => (import [bs4 [BeautifulSoup]])
4 => (setv raw-data "<html><body><a href=\"http://markwatson.com\">Mark</a></body></ht\
5 ml>")
6 => (setv soup (BeautifulSoup raw-data "lxml"))
7 => (setv a (.find-all soup "a"))
8 => (print "atags:" a)
9 atags: [<a href="http://markwatson.com">Mark</a>]
10 => (type a)
11 <class 'bs4.element.ResultSet'>
12 => (dir a)
13 ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '\
14 __dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribut\
15 e__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__in\
16 it_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__mul__', \
17 '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__r\
18 mul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '\
19 __weakref__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop'\
20 , 'remove', 'reverse', 'sort', 'source']
Notice in lines 3 and 6 that we can have “-“ characters inside of variable and function names (raw-
data and find-all in this case) in the Hy language where we might use “_” underscore characters in
Python. Like Python, we can use type get get the type of a value and dir to see what symbols are
available for a object.
1 Marks-MacBook:deeplearning $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (defn foo []
4 ... (global x)
5 ... (setv x 1)
6 ... (print x))
7 => (foo)
8 1
9 => x
Introduction to the Hy Language 8
10 1
11 =>
Before executing function foo the global variable x is undefined (unless you coincidentally already
defined somewhere else). When function foo is called, a global variable x is defined and then it equal
to the value 1.
This code will be in a module named test because that is the root source code file name. We might
import the Python code using the following in Python:
1 import test
2
3 print(test.factorial(5))
and we can use the following in Hy to import the Python module test (defined in test.py):
1 (import test)
2
3 (print (test.factorial 5))
1 $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import test)
4 => test
5 <module 'test' from '/Users/markw/GITHUB/hy-lisp-python/test.py'>
6 => (print (test.factorial 5))
7 120
If we only wanted to import BeautifulSoup from the Python BeautifulSoup library bs4 we can
specify this in the import form:
Introduction to the Hy Language 9
We define two functions here. Notice the optional argument anAgent defined in lines 4-5 where we
provide a default value in case the calling code does not provide a value. In the next Python listing
we import the file in the last listing and call the Hy function main on line 4 using the Python calling
syntax.
Hy is the same as Python once it is compiled to an abstract syntax tree (AST).
hy-lisp-python/use_in_python:
1 import hy
2 from get_web_page import main_hy
3
4 main_hy()
What I want you to understand and develop a feeling for is that Hy and Python are really the same
but with a different syntax and that both languages can easily be used side by side.
⁶https://github.com/mark-watson/hy-lisp-python
Introduction to the Hy Language 10
$ python
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
>>> s = '0123456789'
>>> s[2:4]
'23'
>>> s[-4:]
'6789'
>>> s[-4:-1]
'678'
>>>
$ hy
hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
=> (setv s "0123456789")
=> (cut s 2 4)
'23'
=> (cut s -4)
'6789'
=> (cut s -4 -1)
'678'
=>
It also works to use cut with setv to destructively change a list; for example:
On line 2, the expression (enumerate sentence) generates one character at a time from a string.
enumerate operating on a list will generate one list element at a time.
Line 9 shows an example of destructuring: the values in the list vv are tuples (tuples are like lists but
are immutable, that is, once a tuple is constructed the values it holds can not be changed) with two
values. The values in each tuple are copied into binding variables in the list [a b]. We could have
used the following code instead but it is more verbose:
Introduction to the Hy Language 12
Formatted Output
I suggest using the Python format method when you need to format output. In the following repl
listing, you can see a few formatting options: insert any Hy data into a string (line 3), print values
with a specific width and right justified (in line 5 the width for both values is 15 characters), print
values with a specific width and left justified (in line 7), and limiting the number of characters values
can be expressed as (in line 9 the object “cat” is expressed as just the first two characters and the
value 3.14159 is expressed as just three numbers, the period not counting).
$ hy
hy 0.18.0 using CPython(default) 3.7.4 on Darwin
=> (.format "first: {} second: {}" "cat" 3.14159)
'first: cat second: 3.14159'
=> (.format "first: {:>15} second: {:>15}" "cat" 3.14159)
'first: cat second: 3.14159'
=> (.format "first: {:15} second: {:15}" "cat" 3.14159)
'first: cat second: 3.14159'
=> (.format "first: {:.2} second: {:.3}" "cat" 3.14159)
'first: ca second: 3.14'
=>
Notice the calling .format here returns a string value rather than writing to an output stream.
Introduction to the Hy Language 13
1 Marks-MacBook:hy-lisp-python $ pwd
2 /Users/markw/GITHUB/hy-lisp-python
3 Marks-MacBook:hy-lisp-python $ cd webscraping
4 Marks-MacBook:webscraping $ hy
5 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
6 => (import sys)
7 => (sys.path.insert 1 "../nlp")
8 => (import [nlp-lib [nlp]])
9 => (nlp "President George Bush went to Mexico and he had a very good meal")
10 {'text': 'President George Bush went to Mexico and he had a very good meal',
11 ...
12 'entities': [['George Bush', 'PERSON'], ['Mexico', 'GPE']]}
13 => (import [coref-nlp-lib [coref-nlp]])
14 => (coref-nlp "President George Bush went to Mexico and he had a very good meal")
15 {'corefs': 'President George Bush went to Mexico and President George Bush had a ver\
16 y good meal', ... }}}
17 =>
Here I did not install the library nlp_lib.hy using Python setuptools (which I don’t cover in this
book, you can read the documentation⁷) as a library on the system. I rely on relative paths between
the library directory and the application code that uses the library.
On line 6 I am inserting the library directory into the Python system load path so the import
statement on line 8 can find the nlp-lib library and on line 13 can find the coref-nlp-lib library.
Using Closures
Function definitions can capture values defined outside of a function and even change the captured
value as seen in this example (file closure_example.hy in the directory hy-lisp-python/misc):
⁷https://setuptools.readthedocs.io
Introduction to the Hy Language 14
1 #!/usr/bin/env hy
2
3 (require [hy.contrib.walk [let]])
4
5 (let [x 1]
6 (defn increment []
7 (setv x (+ x 1))
8 x))
9
10 (print (increment))
11 (print (increment))
12 (print (increment))
That produces:
2
3
4
Using closures is often a good alternative to object oriented programming for maintaining private
state that only one or a few functions (that are defined inside the closure) are allowed to access and
modify. In the last example the let statement could have defined more than one variable with initial
values and many functions could have been defined to perform various calculations with the values
of these captured variables and/or change the values of captured variables. This effectively hides the
variables defined in the let statement from code outside of the let statement but the functions are
accessible from outside the let statement.
This podcast⁹ in 2015 interviews Hy developers Paul Tagliamonte, Tuukka Turto, and Morten
Linderud. You can see the current Hy contributer list on github¹⁰.
1 $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import [numpy :as np])
4 => (import [matplotlib.pyplot :as plt])
5 =>
6 => (defn sigmoid [x]
7 ... (/ 1.0 (+ 1.0 (np.exp (- x)))))
8 => (sigmoid 0.2)
9 0.549833997312478
10 => (sigmoid 2)
11 0.8807970779778823
12 => (np.array [-5 -2 0 2 5])
13 array([-5, -2, 0, 2, 5])
14 => (sigmoid (np.array [-5 -2 0 2 5]))
15 array([0.00669285, 0.11920292, 0.5, 0.88079708, 0.99330715])
16 =>
The git repository directory hy-lisp-python/matplotlib contains two similar scripts for plotting the
sigmoid and relu functions. Here is the script to plot the sigmoid function:
⁹https://www.pythonpodcast.com/episode-23-hylang-core-developers/
¹⁰https://github.com/hylang/hy/graphs/contributors
Introduction to the Hy Language 16
The generated plot looks like this on macOS (Matplotlib is portable and also works on Windows and
Linux):
Introduction to the Hy Language 17
Sigmoid Function
The add the following to your .profile, .bash_profile, or .zshrc (depending on your shell setup):
Introduction to the Hy Language 18
1 export MPLBACKEND="module://itermplot"
Here we run an example from the last section in a zsh shell (bash, etc. also should work):
The best part of generating inline plots is during interactive REPL-based coding sessions:
Introduction to the Hy Language 19
If you use a Mac laptop to SSH into a remote Linux server you need to install itermplot and set the
environment variable MPLBACKEND on the remote server.
Why Lisp?
Now that we have learned the basics of the Hy Lisp language in the last chapter, I would like to move
our conversation to a broader question of why we would want to use Lisp. I want to start with my
personal history of why I turned to Lisp languages in the late 1970s for almost all of my creative and
research oriented development and later transitioned to also using Lisp languages in production.
Mind Inside Matter” and I learned Lisp on my own time and then, during lunch hour, taught a one
day a week class to anyone at work who wanted to learn Lisp. After a few months of Lisp experience
I received permission to teach an informal lunch time class to teach anyone working in my building
who wanted to to learn Lisp on our DECsystem-10.
Lisp is the perfect language to support the type of bottom-up iterative programming style that I like.
The following listing shows my experiments in a Hy REPL to write the macro all-to-string:
Why Lisp? 22
1 $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (list (map str ["a" 4]))
4 ['a', '4']
5 => (.join " " (list (map str ["a" 4])))
6 'a 4'
7 => (defmacro foo2 [&rest x] x)
8 <function foo2 at 0x10b91b488>
9 => (foo2 1 2 3)
10 [1, 2, 3]
11 => (foo2 1 runpuppyrun 3)
12 Traceback (most recent call last):
13 File "stdin-3241d1d4f129e0da87f331bfe8f9f7aba903073a", line 1, in <module>
14 (foo2 1 runpuppyrun 3)
15 NameError: name 'runpuppyrun' is not defined
16 => (defmacro all-to-string [&rest x] (.join " " (list (map str x))))
17 <function all-to-string at 0x10b91b158>
18 => (all-to-string cater123 22)
19 'cater123 22'
20 => (all-to-string the boy ran to get 1 new helmet)
21 'the boy ran to get 1 new helmet'
22 => (all-to-string the boy "ran" to get 1 "new" helmet)
23 'the boy ran to get 1 new helmet'
24 =>
My first try in line 7 did not work, the macro just returning a function that echos the arguments but
throws an error (line 50) when one of the arguments is a symbol with no definition. The second try
on line 16 works as intended because we are mapping the function str (which coerces any argument
into a string) over the argument list.
When I need to write a new low-level function, I start in a REPL and define variables (with test
values) for what the function arguments will be. I then write the code for the function one line at a
time using these “arguments” in expressions that will later be copied to a Hy source file. Immediately
seeing results in a REPL helps me catch mistakes early, often a misunderstanding of the type or values
of intermediate calculations. This style of coding works for me and I hope you like it also.
Writing Web Applications
Python has good libraries and frameworks for building web applications and here we will use the
Flask library and framework “under the hood” and write two simple Hy Language web applications.
We will start with a simple “Hello World” example in Python, see how to reformulate it in Hy, and
then proceed with more complex examples that will show how to use HTML generating templates,
sessions, and cookies to store user data for the next time they visit your web site. In a later chapter
we will cover use of the SQLite and PostgreSQL databases which are commonly used to persist data
for users in web applications. This pattern involves letting a user login and store a unique token for
the user in a web browser cookie. In principle, you can do the same with web browser cookies but
if a user visits your web site with a different browser or device then they will not have access to the
data stored in cookies on a previous visit.
I like lightweight web frameworks. In Ruby I use Sinatra, in Haskell I use Spock, and when I built
Java web apps I liked lightweight tools like JSP. Flask is simple but capable and using it from Hy is
productive and fun. In addition to using lightweight frameworks I like to deploy web apps in the
simplest way possible. We will close this chapter by discussing how to use the Heroku and Google
Cloud Platform AppEngine platforms.
We will use the Hy macro with-decorator to replace Python code with annotations. Here the
decorator @app.route is used to map a URI pattern with a Python callback function. In the following
case we define the behavior when the index page of a web app is accessed:
Writing Web Applications 25
I first used Flask with the Hy language after seeing a post of code from HN user “volent”, seen in
the file flask_test.hy in the directory hy-lisp-python/webapp that is functionally equivalent to the
above Python code snippet:
1 #!/usr/bin/env hy
2
3 ;; snippet by HN user volent:
4
5 (import [flask [Flask]])
6
7 (setv app (Flask "Flask test"))
8 (with-decorator (app.route "/")
9 (defn index []
10 "Hello World !"))
11 (app.run)
The Hy macro with-decorator macro is used to use Python style decorators in Hy applications.
I liked this example and after experimenting with the code, I then started using Hy and Flask. Please
try running this example to make sure you are setup properly with Flask:
1 <html>
2 <head>
3 <title>Testing Jinja2 and Flask with the Hy language</title>
4 </head>
5 <body>
6 {% if name %}
7 <h1>Hello {{name}}</h1>
8 {% else %}
9 <h1>Hey, please enter your name!</h1>
10 {% endif %}
11
12 <form method="POST" action="/response">
13 Name: <input type="text" name="name" required>
14 <input type="submit" value="Submit">
15 </form>
16 </body>
17 </html>
¹²https://pypi.org/project/Jinja2/
Writing Web Applications 27
Note that in line 6 we are using a Python if expression to check if the variable name is defined in
the current app execution context.
In the context of a running Flask app, the following will render the above template with the variable
name defined as None:
1 (render_template "template1.j2")
We can set values as named parameters for variables used in the template, for example:
I am assuming that you understand the basics or HTML and also GET and POST operations in HTTP
requests.
The following Flask web app defines behavior for rendering the template without the variable name
set and also a HTML POST handler to pass the name entered on the HTML form back to the POST
response handler:
1 #!/usr/bin/env hy
2
3 (import [flask [Flask render_template request]])
4
5 (setv app (Flask "Flask and Jinja2 test"))
6
7 (with-decorator (app.route "/")
8 (defn index []
9 (render_template "template1.j2")))
10
11 (with-decorator (app.route "/response" :methods ["POST"])
12 (defn response []
13 (setv name (request.form.get "name"))
14 (print name)
15 (render_template "template1.j2" :name name)))
16
17 (app.run)
Please note that there is nothing special about the names inside the with-decorator code blocks:
the functions index and response could have arbitrary names like a123 an b17. I used the function
names index and response because they help describe what the functions do.
Open http://127.0.0.1:5000/¹³ in your web browser:
¹³http://127.0.0.1:5000/
Writing Web Applications 28
Flask web app using a Jinja2 Template after entering my name and submitting the HTML input form
Inside a Jinja2 template you can use a simple Python expression to place a session variable’s value
into the HTML generated from a template:
{{ session['name'] }}
(request.cookies.get "name")
inside of a with-decorator form. The value for request is defined in the execution context by Flask
when handling HTTP requests. Here is a complete example of handling cookies in the file cookie_-
test.hy:
1 #!/usr/bin/env hy
2
3 (import [flask [Flask render_template request make-response]])
4
5 (setv app (Flask "Flask and Jinja2 test"))
6
7 (with-decorator (app.route "/")
8 (defn index []
9 (setv cookie-data (request.cookies.get "hy-cookie"))
10 (print "cookie-data:" cookie-data)
11 (setv a-response (render_template "template1.j2" :name cookie-data))
12 a-response))
13
14 (with-decorator (app.route "/response" :methods ["POST"])
15 (defn response []
16 (setv name (request.form.get "name"))
17 (print name)
18 (setv a-response (make-response (render-template "template1.j2" :name name)))
19 (a-response.set-cookie "hy-cookie" name)
20 a-response))
21
22 (app.run)
Writing Web Applications 30
I suggest that you not only try running this example as-is but also try changing the template, and
generally experiment with the code. Making even simple code changes helps to understand the code
better.
1 import hy
2 import flask_test
3 from flask_test import app
4
5 if __name__ == '__main__':
6 # Used when running locally only. When deploying to Google App
7 # Engine, a webserver process such as Gunicorn will serve the app.
8 app.run(host='localhost', port=9090, debug=True)
The Hy app is slightly different than we saw in the last section. On line 6 we specify the location of
static assets and we do not call the run() method on the app object.
I assume that you have some experience with GCP and have the following:
After cloning or otherwise copying this project, you use the command line tools to deploy and test
your Flask app:
python main.py
Please note that everytime you deploy, a new instance is created. You will want to use the GCP
AppEngine console to remove old instances, and remove all instances when you are done.
Going forward
You can make a copy of this example, create a github repo, and follow the above directions as
a first step to creating Hy language application on AppEngine. The Google Cloud Platform has
many services that you can use in your app (using the Python APIs, called from your Hy program),
including:
import hy
import flask_test
from flask_test import app
The Heroku platform will call the run() method on the imported app object because of the settings
in the Heroku Proc file for this project:
Here we are stating to the Heroku platform that we want the production-friendly gunicorn server
to call the run() method on the app object that is defined in the wsgi module (here the module name
is the prefix name of the Python WSGI handler file).
The Hy Flask app has a few changes from earlier examples. All changes are in line 3:
heroku login
heroku create
git push heroku master
If you have your Heroku account setup these commands will deploy this example.
You can look at the Heroku log files for your application using:
You can open this Hello World app in your default web browser using:
heroku open
By default, your Hello World app will run on the free Heroku mode. You should still remove it when
you are done:
Going forward
You can make a copy of this example, create a github repo, and follow the above directions.
To test your Heroku setup locally or for development, you can use:
heroku local
The Heroku platform has a wide variety of supported services, including many third party services
like data services¹⁷ and Heroku and third party addons¹⁸.
Wrap-up
I like to be able to implement simple things simply, without a lot of ceremony. Once you work
through these examples I hope you feel that you can generate Hy and Flask based web apps quickly
and with very little code required.
To return to the theme of bottom-up programming, I find that starting with short low level utility
functions and placing them in a separate file makes reuse simple and makes future similar projects
¹⁷https://www.heroku.com/managed-data-services
¹⁸https://elements.heroku.com/addons
Writing Web Applications 34
even easier. For each language I work with, I collect snippets of useful code and short utilities kept in
separate files. When writing code I start looking in my snippets directory for the language I am using
to implement low level functionality even before doing a web search. When I work in Common Lisp
I keep all low level code that I have written in small libraries contained a single Quicklisp source
root directory and for Python and Hy I use Python’s setuptools library to generate libraries that
are installed globally on my laptop for easy reuse. It is worth some effort to organize your work for
future reuse.
Responsible Web Scraping
I put the word “Responsible” in the chapter title to remind you that just because it is easy (as we will
soon see) to pull data from web sites, it is important to respect the property rights of web site owners
and abide by their terms and conditions for use. This Wikipedia article on Fair Use¹⁹ provides a good
overview of using copyright material.
The web scraping code we develop here uses the Python BeautifulSoup and URI libraries.
For my work and research, I have been most interested in using web scraping to collect text data for
natural language processing but other common applications include writing AI news collection and
summarization assistants, trying to predict stock prices based on comments in social media which
is what we did at Webmind Corporation in 2000 and 2001, etc.
¹⁹https://en.wikipedia.org/wiki/Fair_use
Responsible Web Scraping 36
1 $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import [get-page-data [get-raw-data-from-web]])
4 => (get-raw-data-from-web "http://knowledgebooks.com")
5 b'<!DOCTYPE html><html><head><title>KnowledgeBooks.com - research on the Knowledge M\
6 anagement, and the Semantic Web ...'
7 =>
8 => (import [get-page-data [get-page-html-elements]])
9 => (get-page-html-elements "http://knowledgebooks.com")
10 {'title': [<title>KnowledgeBooks.com - research on the Knowledge Management, and the\
11 Semantic Web </title>],
12 'a': [<a class="brand" href="#">KnowledgeBooks.com </a>, ...
13 =>
This REPL session shows the the function get-raw-data-from-web defined in the previous listing
returns a web page as a string. In line 9 we use a function get-page-html-elements to find all
elements in a string containing HTML. This function is defined in the next listing and shows how to
parse and process the string contents of a web pages. Note: you will need to install the lxml library
for this example (using pip or pip3 depending on your Python configuration):
The following listing of file get_page_data.hy uses the Beautiful Soup library to parse the string
data for HTML text from a web site. The function get-page-html-elements returns names and
associated data with each element in HTML represented as a string (the extra code on lines 20-24 is
just debug example code):
The function get-element-data defined in lines 5-9 accepts as an argument an HTML element object
(as defined in the Beautiful soup library) and extracts data, if available, for text, name, class, and href
values. The function get-page-html-elements defied in lines 11-18 accepts as an argument a string
containing a URI and returns a dictionary (or map, or hashtable) containing lists of all a, h1, h2, and
title elements in the web page pointed to by the input URI. You can modify get-page-html-elements
to add additional HTML element types, as needed.
Here is the output (with many lines removed for brevity):
make data
This should copy the home pages for both web sites to the files:
1 #!/usr/bin/env hy
2
3 (import [get-web-page [get-web-page-from-disk]])
4 (import [bs4 [BeautifulSoup]])
5
6 ;; you need to run 'make data' to fetch sample HTML data for dev and testing
7
8 (defn get-democracy-now-links []
9 (setv test-html (get-web-page-from-disk "democracynow_home_page.html"))
10 (setv bs (BeautifulSoup test-html :features "lxml"))
11 (setv all-anchor-elements (.findAll bs "a"))
12 (lfor e all-anchor-elements
13 :if (> (len (.get-text e)) 0)
14 (, (.get e "href") (.get-text e))))
15
16 (if (= __name__ "__main__")
17 (for [[uri text] (get-democracy-now-links)]
18 (print uri ":" text)))
This simply prints our URIs and text (separated with the string “:”) for each link on the home page.
On line 13 we discard any anchor elements that do not contain text. On line 14 the comma character
at the start of the return list indicates that we are constructing a tuple. Lines 16-18 define a main
function that is used when running this file art the command line. This is similar to how main
functions can be defined in Python to allow a library file to also be run as a command line tool.
A few lines of output from today’s front page is:
Responsible Web Scraping 39
1 #!/usr/bin/env hy
2
3 (import [get-web-page [get-web-page-from-disk]])
4 (import [bs4 [BeautifulSoup]])
5
6 ;; you need to run 'make data' to fetch sample HTML data for dev and testing
7
8 (defn get-npr-links []
9 (setv test-html (get-web-page-from-disk "npr_home_page.html"))
10 (setv bs (BeautifulSoup test-html :features "lxml"))
11 (setv all-anchor-elements (.findAll bs "a"))
12 (setv filtered-a
13 (lfor e all-anchor-elements
14 :if (> (len (.get-text e)) 0)
15 (, (.get e "href") (.get-text e))))
16 filtered-a)
17
18 (defn create-npr-summary []
Responsible Web Scraping 40
In lines 12-15 we are filtering out (or removing) all anchor HTML elements that do not contain text.
The following shows a few lines of the generated output for data collected today:
January 16, 2020 Birds change the shape of their wings far more than
planes. The complexities of bird flight have posed a major design challenge
for scientists trying to translate the way birds fly into robots.
January 16, 2020 The bureau was faulted after the Russian attack on the
2016 election for keeping too much information from state and local
authorities. It says it'll use a new policy going forward.
January 16, 2020 Ukraine's Internal Affairs Ministry says it's asking the
FBI to help determine whether international laws were broken, or "whether it
is just a bravado and a fake information" from a U.S. politician.
Electric Burn: Those Who Bet Against Elon Musk And Tesla Are Paying A Big Price
January 16, 2020 For years, Elon Musk skeptics have shorted Tesla stock, confident \
the electric carmaker was on the brink of disaster. Instead, share value has skyrock\
eted, costing short sellers billions.
TSA Says It Seized A Record Number Of Firearms At U.S. Airports Last Year
The examples seen here are simple but should be sufficient to get you started gathering text data
from the web.
Using the Microsoft Bing Search APIs
You will need to register with Microsoft’s Azure search service to use the material in this chapter. It
is likely that you view search as a manual human-centered activity. I hope to expand your thinking
to considering applications that automate search, finding information on the web, and automatically
organizing information.
export BING_SEARCH_V7_SUBSCRIPTION_KEY=4e97234341d9891191c772b7371ad5b1
$ hy
hy 0.18.0 using CPython(default) 3.7.4 on Darwin
=> (import os)
=> (get os.environ "BING_SEARCH_V7_SUBSCRIPTION_KEY")
'4e97234341d9891191c772b7371ad5b1'
=>
#!/usr/bin/env hy
(import json)
(import os)
(import sys)
(import [pprint [pprint]])
(import requests)
;; Construct a request
(setv mkt "en-US")
(setv params { "q" query "mkt" mkt })
(setv headers { "Ocp-Apim-Subscription-Key" subscription_key })
You can use search hints like “site:wikidata.org” to only search specific web sites. In the following
example I use the search query:
This example generates 364 lines of output so I only show a few selected lines here:
{'_type': 'SearchResponse',
'queryContext': {'originalQuery': 'site:wikidata.org Sedona Arizona'},
...
{'totalEstimatedMatches': 27,
'value': [{'about': [{'name': 'Sedona'}, {'name': 'Sedona'}],
'dateLastCrawled': '2020-05-24T00:04:00.0000000Z',
'displayUrl': 'https://www.wikidata.org/wiki/Q80041',
...
Using the Microsoft Bing Search APIs 44
'https://www.wikidata.org/wiki/Q80041'
key: displayUrl : https://www.wikidata.org/wiki/Q80041
key: language : en
key: name : Sedona - Wikidata
Wrap-up
In addition to using automated web scraping to get data for my personal research, I often use
automated web search. I find the Microsoft’s Azure Bing search APIs are the most convenient to
use and I like paying for services that I use. The search engine Duck Duck Go also provides free
search APIs but even though I use Duck Duck Go for 90% of my manual web searches, when I build
automated systems I prefer to rely on services that I pay for.
Deep Learning
Most of my professional career since 2014 has involved Deep Learning, mostly with TensorFlow
using the Keras APIs. In the late 1980s I was on a DARPA neural network technology advisory panel
for a year, I wrote the first prototype of the SAIC ANSim neural network library commercial product,
and I wrote the neural network prediction code for a bomb detector my company designed and built
for the FAA for deployment in airports. More recently I have used GAN (generative adversarial
networks) models for synthesizing numeric spreadsheet data and LSTM (long short term memory)
models to synthesize highly structured text data like nested JSON and for NLP (natural language
processing). I have 55 USA and several European patents using neural network and Deep Learning
technology.
The Hy language utilities and example programs we develop here all use TensorFlow and Keras
“under the hood” to do the heavy lifting. Keras is a simpler to use API for TensorFlow and I usually
use Keras rather than the lower level TensorFlow APIs.
There are other libraries and frameworks that might interest you in addition to TensorFlow and
Keras. I particularly like the Flux library for the Julia programming language. Currently Python has
the most comprehensive libraries for Deep Learning but other languages that support differential
computing (more on this later) like Julia and Swift may gain popularity in the future.
Here we will learn a vocabulary for discussing Deep Learning neural network models, look at
possible architectures, and show two Hy language examples that should be sufficient to get you used
to using Keras with the Hy language. If you already have Deep Learning application development
experience you might want to skip the following review material and skip to the Hy language
examples.
If you want to use Deep Learning professionally, there are two specific online resources that I
recommend: Andrew Ng leads the efforts at deeplearning.ai²¹ and Jeremy Howard leads the efforts
at fast.ai²². Here I will show you how to use a few useful techniques. Andrew and Jeremy will teach
you skills that may lead a professional level of expertise if you take their courses.
There are many Deep Learning neural architectures in current practical use; a few types that I use
are:
• Multi-layer perceptron networks with many fully connected layers. An input layer contains
placeholders for input data. Each element in the input layer is connected by a two-dimensional
weight matrix to each element in the first hidden layer. We can use any number of fully
connected hidden layers, with the last hidden layer connected to an output layer.
²¹https://www.deeplearning.ai/
²²https://www.fast.ai/
Deep Learning 46
• Convolutional networks for image processing and text classification. Convolutions, or filters,
are small windows that can process input images (filters are two-dimensional) or sequences like
text (filters are one-dimensional). Each filter uses a single set of learned weights independent
of where the filter is applied in an input image or input sequence.
• Autoencoders have the same number of input layer and output layer elements with one or
more hidden fully connected layers. Autoencoders are trained to produce the same output as
training input values using a relatively small number of hidden layer elements. Autoencoders
are capable of removing noise in input data.
• LSTM (long short term memory) process elements in a sequence in order and are capable of
remembering patterns that they have seen earlier in the sequence.
• GAN (generative adversarial networks) models comprise two different and competing neural
models, the generator and the discriminator. GANs are often trained on input images (although
in my work I have applied GANs to two-dimensional numeric spreadsheet data). The generator
model takes as input a “latent input vector” (this is just a vector of specific size with random
values) and generates a random output image. The weights of the generator model are trained
to produce random images that are similar to how training images look. The discriminator
model is trained to recognize if an arbitrary output image is original training data or an image
created by the generator model. The generator and discriminator models are trained together.
The core functionality of libraries like TensorFlow are written in C++ and take advantage of special
hardware like GPUs, custom ASICs, and devices like Google’s TPUs. Most people who work with
Deep Learning models don’t need to even be aware of the low level optimizations used to make
training and using Deep Learning models more efficient. That said, in the following section I am
going to show you how simple neural networks are trained and used.
The following figure shows a simple backpropagation network with one hidden layer. Neurons in
adjacent layers are connected by floating point connection strength weights. These weights start
out as small random values that change as the network is trained. Weights are represented in the
following figure by arrows; in the code the weights connecting the input to the output neurons are
represented as a two-dimensional array.
Each non-input neuron has an activation value that is calculated from the activation values of
connected neurons feeding into it, gated (adjusted) by the connection weights. For example, in
the above figure, the value of Output 1 neuron is calculated by summing the activation of Input
1 times weight W1,1 and Input 2 activation times weight W2,1 and applying a “squashing function”
like Sigmoid or Relu (see figures below) to this sum to get the final value for Output 1’s activation
value. We want to flatten activation values to a relatively small range but still maintain relative
values. To do this flattening we use the Sigmoid function that is seen in the next figure, along with
the derivative of the Sigmoid function which we will use in the code for training a network by
adjusting the weights.
Deep Learning 48
Simple neural network architectures with just one or two hidden layers are easy to train using
backpropagation and I have from scratch code for this several of my previous books. You can
see Java and Common Lisp from-scratch implementations in two of my books that you can read
online: Practical Artificial Intelligence Programming With Java²³ and Loving Common Lisp, or the
Savvy Programmer’s Secret Weapon²⁴. However, here we are using Hy to write models using the
TensorFlow framework which has the huge advantage that small models you experiment with on
your laptop can be scaled to more parameters (usually this means more neurons in hidden layers
which increases the number of weights in a model) and run in the cloud using multiple GPUs.
Except for pendantic purposes, I now never write neural network code from scratch, instead I take
advantage of the many person-years of engineering work put into the development of frameworks
like TensorFlow, PyTorch, mxnet, etc. We now move on to two examples built with TensorFlow.
Deep Learning
Deep Learning models are generally understood to have many more hidden layers than simple multi-
layer perceptron neural networks and often comprise multiple simple models combined together in
series or in parallel. Complex architectures can be iteratively developed by manually adjusting the
size of model components, changing the components, etc. Alternatively, model architecture search
can be automated. At Capital One I used Google’s AdaNet project²⁵ that efficiently searches for
effective model architectures inside a single TensorFlow session. The model architecture used here
is simple: one input layer representing the input values in a sample of University of Wisconsin cancer
data, one hidden layer, and an output layer consisting of one neuron whose activation value will be
interpreted as a prediction of benign or malignant.
The material in this chapter is intended to serve two purposes:
• If you are already familiar with Deep Learning and TensorFlow then the examples here will
serve to show you how to call the TensorFlow APIs from Hy.
²³https://leanpub.com/javaai
²⁴https://leanpub.com/lovinglisp
²⁵https://github.com/tensorflow/adanet
Deep Learning 49
• If you have little or no exposure with Deep Learning then the short Hy language examples will
provide you with concise code to experiment with and you can then decide to study further.
Once again, I recommend that you consider taking two online Deep Learning course sequences. For
no cost, Jeremy Howard provides lessons at fast.ai²⁶ that are very good and the later classes use
PyTorch which is a framework that is similar to TensorFlow. For a modest cost Andrew Ng provides
classes at deeplearning.ai²⁷ that use TensorFlow. I have been working in the field of machine learning
since the 1980s, but I still take Andrew’s online classes to stay up-to-date. In the last eight years I have
taken his Stanford University machine learning class twice and also his complete course sequence
using TensorFlow. I have also worked through much of Jeremy’s material. I recommend both course
sequences without reservation.
• 0 Clump Thickness 1 - 10
• 1 Uniformity of Cell Size 1 - 10
• 2 Uniformity of Cell Shape 1 - 10
• 3 Marginal Adhesion 1 - 10
• 4 Single Epithelial Cell Size 1 - 10
• 5 Bare Nuclei 1 - 10
• 6 Bland Chromatin 1 - 10
• 7 Normal Nucleoli 1 - 10
• 8 Mitoses 1 - 10
• 9 Class (0 for benign, 1 for malignant)
We will use separate training and test files hy-lisp-python/deeplearning/train.csv and hy-lisp-
python/deeplearning/test.csv. Here are a few samples from the training file:
²⁶https://fast.ai
²⁷https://www.deeplearning.ai/
Deep Learning 50
6,2,1,1,1,1,7,1,1,0
2,5,3,3,6,7,7,5,1,1
10,4,3,1,3,3,6,5,2,1
6,10,10,2,8,10,7,3,3,1
5,6,5,6,10,1,3,1,1,1
1,1,1,1,2,1,2,1,2,0
3,7,7,4,4,9,4,8,1,1
1,1,1,1,2,1,2,1,1,0
After you look at this data, if you did not have much experience with machine learning then it might
not be obvious how to build a model to accept a sample for a patient like we see in the Wisconsin
data set and then predict if the sample implies benign or cancerous outcome for the patient. Using
TensorFlow with a simple neural network model, we will implement a model in about 40 lines of
Hy code to implement this example.
Since there are nine input values we will need nine input neurons that will represent the input values
for a sample in either training or separate test data. These nine input neurons (created in lines 9-10
in the following listing) will be completely connected to twelve neurons in a hidden layer. Here,
completely connected means that each of the nine input neurons is connected via a weight to each
hidden layer neuron. There are 9 * 12 = 108 weights between the input and hidden layers. There is
a single output layer neuron that is connected to each hidden layer neuron.
Notice that in lines 12 and 14 in the following listing that we specify a relu activation function while
the activation function connecting the hidden layer to the output layer uses the sigmoid activation
function that we saw plotted earlier.
There is an example in the git example repo directory hy-lisp-python/matplotlib in the file plot_-
relu.hy that generated the following figure:
Deep Learning 51
Relu Function
The following listing shows the use of the Keras TensorFlow APIs to build a model (lines 9-19) with
one input layer, two hidden layers, and an output layer with just one neuron. After we build the
model, we define two utility functions train (lines 21-23) to train a model given training inputs (x
argument) and corresponding training outputs (y** argument), and we also define predict (lines
25-26) using a trained model to make a cancer or benign prediction given test input values (x-data
argument).
Lines 28-33 show a utility function load-data that loads a University of Wisconsin cancer data set
CSV file, scales the input and output values to the range [0.0, 1.0] and returns a list containing input
(x-data) and target output data (y-data). You may want to load this example in a REPL and evaluate
load-data on one of the CSV files.
The function main (lines 35-45) loads training and test (evaluation of model accuracy on data not
used for training), trains a model, and then tests the accuracy of the model on the test (evaluation)
Deep Learning 52
data:
1 #!/usr/bin/env hy
2
3 (import argparse os)
4 (import keras
5 keras.utils.data-utils)
6
7 (import [pandas [read-csv]])
8
9 (defn build-model []
10 (setv model (keras.models.Sequential))
11 (.add model (keras.layers.core.Dense 9
12 :activation "relu"))
13 (.add model (keras.layers.core.Dense 12
14 :activation "relu"))
15 (.add model (keras.layers.core.Dense 1
16 :activation "sigmoid"))
17 (.compile model :loss "binary_crossentropy"
18 :optimizer (keras.optimizers.RMSprop))
19 model)
20
21 (defn train [batch-size model x y]
22 (for [it (range 50)]
23 (.fit model x y :batch-size batch-size :epochs 10 :verbose False)))
24
25 (defn predict [model x-data]
26 (.predict model x-data))
27
28 (defn load-data [file-name]
29 (setv all-data (read-csv file-name :header None))
30 (setv x-data10 (. all-data.iloc [(, (slice 0 10) [0 1 2 3 4 5 6 7 8])] values))
31 (setv x-data (* 0.1 x-data10))
32 (setv y-data (. all-data.iloc [(, (slice 0 10) [9])] values))
33 [x-data y-data])
34
35 (defn main []
36 (setv xyd (load-data "train.csv"))
37 (setv model (build-model))
38 (setv xytest (load-data "test.csv"))
39 (train 10 model (. xyd [0]) (. xyd [1]))
40 (print "* predictions (calculated, expected):")
41 (setv predictions (list (map first (predict model (. xytest [0])))))
Deep Learning 53
1 $ hy wisconsin.hy
2 Using TensorFlow backend.
3 * predictions (calculated, expected):
4 [(0.9759052, 1), (0.99994254, 1), (0.8564741, 1), (0.95866203, 1), (0.03042546, 0), \
5 (0.21845636, 0), (0.99662805, 1), (0.08626339, 0), (0.045683343, 0), (0.9992156, 1)]
Let’s look at the first test case: the “real” output from the training data is a value of 1 and the
calculated predicted value (using the trained model) is 0.9759052. In making predictions, we can
choose a cutoff value, 0.5 for example, and interpret any calculated prediction value less than the
cutoff as a Boolean false prediction and calculated prediction value greater to or equal to the cutoff
value is a Boolean true prediction.
²⁸https://keras.io/examples/lstm_text_generation/
Deep Learning 54
1 pushd ~/.keras/datasets/
2 mv nietzsche.txt nietzsche_large.txt
3 head -800 nietzsche_large.txt > nietzsche.txt
4 popd
The next time you run the example, the Keras example data loading utilities will notice a local copy
and even though the file now is much smaller, the data loading utilities will not download a new
copy.
When I start training a new Deep Learning model I like to monitor system resources using the top
command line activity, watching for page faults when training on a CPU which might indicate that
I am trying to train too large of a model for my system memory. If you are using CUDA and a
GPU then use the CUDA command line utilities for monitoring the state of the GPU utilization.
It is beyond the scope of this introductory tutorial, but the tool TensorBoard²⁹ is very useful for
monitoring the state of model training.
There are a few things that make the following example code more complex than the example using
the University of Wisconsin cancer data set. We need to convert each character in the training data
to a one-hot encoding which is a vector of all 0.0 values except for a single value of 1.0. I am going
to show you a short REPL session so that you understand how this works and then we will look at
the complete Hy code example.
1 $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import [keras.callbacks [LambdaCallback]])
4 Using TensorFlow backend.
5 => (import [keras.models [Sequential]])
6 => (import [keras.layers [Dense LSTM]])
7 => (import [keras.optimizers [RMSprop]])
8 => (import [keras.utils.data_utils [get_file]])
9 => (import [numpy :as np]) ;; note the syntax for aliasing a module name
10 => (import random sys io)
11 => (with [f (io.open "/Users/markw/.keras/datasets/nietzsche.txt" :encoding "utf-8")]
12 ... (setv text (.read f)))
13 => (cut text 98 130)
14 'philosophers, in so far as they '
15 => (setv chars (sorted (list (set text))))
16 => chars
17 ['\n', ' ', '!', '"', "'", '(', ')', ',', '-', '.', '0', '1', '2', '3', '4', '5', '6\
18 ', '7', '8', '9', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', '\
19 K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', '_', 'a', \
20 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r',\
21 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
²⁹https://www.tensorflow.org/tensorboard/
Deep Learning 55
22 => (setv char_indices (dict (lfor i (enumerate chars) (, (last i) (first i)))))
23 => char_indices
24 {'\n': 0, ' ': 1, '!': 2, '"': 3, "'": 4, '(': 5, ')': 6, ',': 7, '-': 8, '.': 9, '0\
25 ': 10, '1': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9': \
26 19, ':': 20, ';': 21, '?': 22, 'A': 23, 'B': 24, 'C': 25, 'D': 26, 'E': 27, 'F': 28,\
27 'G': 29, 'H': 30, 'I': 31, 'J': 32, 'K': 33, 'L': 34, 'M': 35, 'N': 36, 'O': 37, 'P\
28 ': 38, 'Q': 39, 'R': 40, 'S': 41, 'T': 42, 'U': 43, 'V': 44, 'W': 45, 'X': 46, 'Y': \
29 47, '_': 48, 'a': 49, 'b': 50, 'c': 51, 'd': 52, 'e': 53, 'f': 54, 'g': 55, 'h': 56,\
30 'i': 57, 'j': 58, 'k': 59, 'l': 60, 'm': 61, 'n': 62, 'o': 63, 'p': 64, 'q': 65, 'r\
31 ': 66, 's': 67, 't': 68, 'u': 69, 'v': 70, 'w': 71, 'x': 72, 'y': 73, 'z': 74}
32 => (setv indices_char (dict (lfor i (enumerate chars) i)))
33 => indices_char
34 {0: '\n', 1: ' ', 2: '!', 3: '"', 4: "'", 5: '(', 6: ')', 7: ',', 8: '-', 9: '.', 10\
35 : '0', 11: '1', 12: '2', 13: '3', 14: '4', 15: '5', 16: '6', 17: '7', 18: '8', 19: '\
36 9', 20: ':', 21: ';', 22: '?', 23: 'A', 24: 'B', 25: 'C', 26: 'D', 27: 'E', 28: 'F',\
37 29: 'G', 30: 'H', 31: 'I', 32: 'J', 33: 'K', 34: 'L', 35: 'M', 36: 'N', 37: 'O', 38\
38 : 'P', 39: 'Q', 40: 'R', 41: 'S', 42: 'T', 43: 'U', 44: 'V', 45: 'W', 46: 'X', 47: '\
39 Y', 48: '_', 49: 'a', 50: 'b', 51: 'c', 52: 'd', 53: 'e', 54: 'f', 55: 'g', 56: 'h',\
40 57: 'i', 58: 'j', 59: 'k', 60: 'l', 61: 'm', 62: 'n', 63: 'o', 64: 'p', 65: 'q', 66\
41 : 'r', 67: 's', 68: 't', 69: 'u', 70: 'v', 71: 'w', 72: 'x', 73: 'y', 74: 'z'}
42 => (setv maxlen 40)
43 => (setv s "Oh! I saw 1 dog (yesterday)")
44 => (setv x_pred (np.zeros [1 maxlen (len chars)]))
45 => (for [[t char] (lfor j (enumerate s) j)]
46 ... (setv (get x_pred 0 t (get char_indices char)) 1))
47 => x_pred
48 array([[[0., 0., 0., ..., 0., 0., 0.],
49 [0., 0., 0., ..., 0., 0., 0.],
50 [0., 0., 1., ..., 0., 0., 0.], // here 1. is the third character "!"
51 ...,
52 [0., 0., 0., ..., 0., 0., 0.],
53 [0., 0., 0., ..., 0., 0., 0.],
54 [0., 0., 0., ..., 0., 0., 0.]]])
55 =>
For lines 48-54, each line represents a single character one-hot encoded. Notice how the third
character shown on line 50 has a value of “1.” at index 2, which corresponds to the one-hot encoding
of the letter “!”.
Now that you have a feeling for how one-hot encoding works, hopefully the following example will
make sense to you. We will further discuss one-hot-encoding after the next code listing. For training,
we take 40 characters (the value of the variable maxlen) at a time, and using one one-hot encode
a character at a time as input and the target output will be the one-hot encoding of the following
Deep Learning 56
character in the input sequence. We are iterating on training the model for a while and then given a
few characters of text, predict a likely next character - and keep repeating this process. The generated
text is then used as input to the model to generate yet more text. You can repeat this process until
you have generated sufficient text.
This is a powerful technique that I used to model JSON with complex deeply nested schemas and
then generate synthetic JSON in the same schema as the training data. Here, training a model to
mimic the philosopher Nietzsche’s writing is much easier than learning highly structured data like
JSON:
1 #!/usr/bin/env hy
2
3 ;; This example was translated from the Python example in the Keras
4 ;; documentation at: https://keras.io/examples/lstm_text_generation/
5 ;; The original Python file LSTM.py is included in the directory
6 ;; hy-lisp-python/deeplearning for reference.
7
8 (import [keras.callbacks [LambdaCallback]])
9 (import [keras.models [Sequential]])
10 (import [keras.layers [Dense LSTM]])
11 (import [keras.optimizers [RMSprop]])
12 (import [keras.utils.data_utils [get_file]])
13 (import [numpy :as np]) ;; note the syntax for aliasing a module name
14 (import random sys io)
15
16 (setv path
17 (get_file ;; this saves a local copy in ~/.keras/datasets
18 "nietzsche.txt"
19 :origin "https://s3.amazonaws.com/text-datasets/nietzsche.txt"))
20
21 (with [f (io.open path :encoding "utf-8")]
22 (setv text (.read f))) ;; note: sometimes we use (.lower text) to
23 ;; convert text to all lower case
24 (print "corpus length:" (len text))
25
26 (setv chars (sorted (list (set text))))
27 (print "total chars (unique characters in input text):" (len chars))
28 (setv char_indices (dict (lfor i (enumerate chars) (, (last i) (first i)))))
29 (setv indices_char (dict (lfor i (enumerate chars) i)))
30
31 ;; cut the text in semi-redundant sequences of maxlen characters
32 (setv maxlen 40)
33 (setv step 3) ;; when we sample text, slide sampling window 3 characters
Deep Learning 57
77 (sys.stdout.write generated)
78 (for [i (range 400)]
79 (setv x_pred (np.zeros [1 maxlen (len chars)]))
80 (for [[t char] (lfor j (enumerate sentence) j)]
81 (setv (get x_pred 0 t (get char_indices char)) 1))
82 (setv preds (first (model.predict x_pred :verbose 0)))
83 (setv next_index (sample preds diversity))
84 (setv next_char (get indices_char next_index))
85 (setv sentence (+ (cut sentence 1) next_char))
86 (sys.stdout.write next_char)
87 (sys.stdout.flush))
88 (print)))
89
90 (setv print_callback (LambdaCallback :on_epoch_end on_epoch_end))
91
92 (model.fit x y :batch_size 128 :epochs 60 :callbacks [print_callback])
In lines 52-54 we defined a model using the Keras APIs and in lines 56-57 compiled the model using
a categorical crossentropy loss function with an RMSprop optimizer³⁰.
In lines 59-65 we define a function sample that takes a first required argument preds which is a
one-hot predicted encoded character that might look like (maxlen or 40 values):
[2.80193929e-02 6.78635418e-01 7.85831537e-04 4.92034527e-03 . . . 6.62320468e-04 9.14627407e-03
2.31375365e-04]
Now, here the predicted one hot encoding values are not strictly 0 or 1, rather they are small floating
point numbers of a single number much larger than the others. The largest number is 6.78635418e-01
at index 1 which corresponds to a one-hot encoding for a “ “ space character.
If we print out the number of characters in text and the unique list of characters (variable chars) in
the training text file nietzsche.txt we see:
vectors to text when we use a trained model to generate text. It will help to see the dictionaries for
converting characters to indices and then reverse indices to original characters as we saw earlier,
some output removed:
char_indices:
{'\n': 0, ' ': 1, '!': 2, '"': 3, "'": 4, '(': 5, ')': 6, ',': 7, '-': 8, '.': 9, '\
0': 10, '1': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9':\
19,
. . .
'f': 58, 'g': 59, 'h': 60, 'i': 61, 'j': 62, 'k': 63, 'l': 64, 'm': 65, 'n': 66, 'o\
': 67, 'p': 68, 'q': 69, 'r': 70, 's': 71, 't': 72, 'u': 73, 'v': 74, 'w': 75, 'x': \
76, 'y': 77, 'z': 78, 'Æ': 79, 'ä': 80, 'æ': 81, 'é': 82, 'ë': 83}
indices_char:
{0: '\n', 1: ' ', 2: '!', 3: '"', 4: "'", 5: '(', 6: ')', 7: ',', 8: '-', 9: '.', 1\
0: '0', 11: '1', 12: '2', 13: '3', 14: '4', 15: '5', 16: '6', 17: '7', 18: '8', 19: \
'9',
. . .
'o', 68: 'p', 69: 'q', 70: 'r', 71: 's', 72: 't', 73: 'u', 74: 'v', 75: 'w', 76: 'x\
', 77: 'y', 78: 'z', 79: 'Æ', 80: 'ä', 81: 'æ', 82: 'é', 83: 'ë'}
We prepare the input and target output data in lines 43-48 in the last code listing. Using a short
string, let’s look in the next REPL session listing at how these input and output training examples
are extracted for an input string:
1 Marks-MacBook:deeplearning $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (setv text "0123456789abcdefg")
4 => (setv maxlen 4)
5 => (setv i 3)
6 => (cut text i (+ i maxlen))
7 '3456'
8 => (cut text (+ 1 maxlen))
9 '56789abcdefg'
10 => (setv i 4) ;; i is the for loop variable for
11 => (cut text i (+ i maxlen)) ;; defining sentences and next_chars
12 '4567'
13 => (cut text (+ i maxlen))
14 '89abcdefg'
15 =>
So the input training sentences are each maxlen characters long and the next-chars target outputs
each start with the character after the last character in the corresponding input training sentence.
Deep Learning 60
This script pauses during each training epoc to generate text given diversity values of 0.2, 0.5, 1.0,
and 1.2. The smaller the diversity value the more closely the generated text matches the training
text. The generated text is more realistic after many training epocs. In the following, I list a highly
edited copy of running through several training epochs. I only show generated text for diversity
equal to 0.2:
38 the strength of the sense-in the former the sense-in the special and the same man in\
39 the consequently the soul of the superstition of the special in the end to the poss\
40 ible that it is will not be a sort of the superior of the superstition of the same m\
41 an to the same man
Here we trained on examples, translated to English, of the philosopher Nietzsche. I have used similar
code to this example to train on highly structured JSON data and the resulting LSTM bsed model was
usually able to generate similarly structured JSON. I have seen other examples where the training
data was code in C++.
How is this example working? The model learns what combinations of characters tend to appear
together and in what order.
In the next chapter we will use pre-trained Deep Learning models for natural language processing
(NLP).
Natural Language Processing
I have been working in the field of Natural Language Processing (NLP) since 1985 so I ‘lived through’
the revolutionary change in NLP that has occurred since 2014: deep learning results out-classed
results from previous symbolic methods.
I will not cover older symbolic methods of NLP here, rather I refer you to my previous books Practical
Artificial Intelligence Programming With Java³¹, [Loving Common Lisp, The Savvy Programmer’s
Secret Weapon³², and Haskell Tutorial and Cookbook³³ for examples. We get better results using
Deep Learning (DL) for NLP and the library spaCy (https://spacy.io³⁴) that we use in this chapter
provides near state of the art performance. The authors of spaCy frequently update it to use the
latest breakthroughs in the field.
You will learn how to apply both DL and NLP by using the state-of-the-art full-feature library
spaCy³⁵. This chapter concentrates on how to use spaCy in the Hy language for solutions to a few
selected problems in NLP that I use in my own work. I urge you to also review the “Guides” section of
the spaCy documentation³⁶ where examples are in Python but after experimenting with the examples
in this chapter you should have no difficulty in translating any spaCy Python examples to the Hy
language.
If you have not already done so install the spaCy library and the full English language model:
You can use a smaller model (which requires loading “en_core_web_sm” instead of “en” in the
following examples):
1 Marks-MacBook:nlp $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import spacy)
4 => (setv nlp-model (spacy.load "en"))
5 => (setv doc (nlp-model "President George Bush went to Mexico and he had a very good\
6 meal"))
7 => doc
8 President George Bush went to Mexico and he had a very good meal
9 => (dir doc)
10 ['_', '__bytes__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__fo\
11 rmat__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init_\
12 _', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new\
13 __', '__pyx_vtable__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__\
14 setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '_bulk_merg\
15 e', '_py_tokens', '_realloc', '_vector', '_vector_norm', 'cats', 'char_span', 'count\
16 _by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', 'from_disk', 'get_\
17 extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'is_nered', 'is_parsed'\
18 , 'is_sentenced', 'is_tagged', 'lang', 'lang_', 'mem', 'merge', 'noun_chunks', 'noun\
19 _chunks_iterator', 'print_tree', 'remove_extension', 'retokenize', 'sentiment', 'sen\
20 ts', 'set_extension', 'similarity', 'tensor', 'text', 'text_with_ws', 'to_array', 't\
21 o_bytes', 'to_disk', 'to_json', 'user_data', 'user_hooks', 'user_span_hooks', 'user_\
22 token_hooks', 'vector', 'vector_norm', 'vocab']
In lines 3-6 we import the spaCy library, load the English language model, and create a document
from input text. What is a spaCy document? In line 9 we use the standard Python function dir to
look at all names and functions defined for the object doc returned from applying a spaCy model
to a string containing text. The value printed shows many built in “dunder” (double underscore
attributes), and we can remove these:
In lines 23-26 we use the dir function again to see the attributes and methods for this class, but filter
out any attributes containing the characters “__”:
23 => (lfor
24 ... x (dir doc)
25 ... :if (not (.startswith x "__"))
26 ... x)
27 ['_', '_bulk_merge', '_py_tokens', '_realloc', '_vector', '_vector_norm', 'cats', 'c\
28 har_span', 'count_by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', '\
29 from_disk', 'get_extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'is_ne\
30 red', 'is_parsed', 'is_sentenced', 'is_tagged', 'lang', 'lang_', 'mem', 'merge', 'no\
31 un_chunks', 'noun_chunks_iterator', 'print_tree', 'remove_extension', 'retokenize', \
32 'sentiment', 'sents', 'set_extension', 'similarity', 'tensor', 'text', 'text_with_ws\
33 ', 'to_array', 'to_bytes', 'to_disk', 'to_json', 'user_data', 'user_hooks', 'user_sp\
Natural Language Processing 64
The to_json method looks promising so we will import the Python pretty print library and look at
the pretty printed result of calling the to_json method on our document stored in doc:
73 'id': 4,
74 'pos': 'ADP',
75 'start': 27,
76 'tag': 'IN'},
77 {'dep': 'pobj',
78 'end': 36,
79 'head': 4,
80 'id': 5,
81 'pos': 'PROPN',
82 'start': 30,
83 'tag': 'NNP'},
84 {'dep': 'cc',
85 'end': 40,
86 'head': 3,
87 'id': 6,
88 'pos': 'CCONJ',
89 'start': 37,
90 'tag': 'CC'},
91 {'dep': 'nsubj',
92 'end': 43,
93 'head': 8,
94 'id': 7,
95 'pos': 'PRON',
96 'start': 41,
97 'tag': 'PRP'},
98 {'dep': 'conj',
99 'end': 47,
100 'head': 3,
101 'id': 8,
102 'pos': 'VERB',
103 'start': 44,
104 'tag': 'VBD'},
105 {'dep': 'det',
106 'end': 49,
107 'head': 12,
108 'id': 9,
109 'pos': 'DET',
110 'start': 48,
111 'tag': 'DT'},
112 {'dep': 'advmod',
113 'end': 54,
114 'head': 11,
115 'id': 10,
Natural Language Processing 66
The JSON data is nested dictionaries. In a later chapter on Knowledge Graphs, we will want to get the
named entities like people, organizations, etc., from text and use this information to automatically
generate data for Knowledge Graphs. The values for the key ents (stands for “entities”) will be useful.
Notice that the words in the original text are specified by beginning and ending text token indices
(values of head and end in lines 52 to 142).
The values for the key tokens listed on lines 42-132 contains the head (or starting index, ending
index, the token number (id), and the part of speech (pos). We will list what the parts of speech
mean later.
We would like the words for each entity to be concatenated into a single string for each entity and
we do this here in lines 136-137 and see the results in lines 138-139.
I like to add the entity name strings back into the dictionary representing a document and line 140
shows the use of lfor to create a list of lists where the sublists contain the entity name as a single
string and the type of entity. We list the entity types supported by spaCy in the next section.
Natural Language Processing 67
We can also access each sentence as a separate string. In this example the original text used to create
our sample document had only a single sentence so the sents property returns a list containing a
single string:
The last example showing how to use a spaCy document object is listing each word with its part of
speech:
The following list shows the definitions for the part of speech (POS) tags:
• ADJ: adjective
• ADP: adposition
Natural Language Processing 68
• ADV: adverb
• AUX: auxiliary verb
• CONJ: coordinating conjunction
• DET: determiner
• INTJ: interjection
• NOUN: noun
• NUM: numeral
• PART: particle
• PRON: pronoun
• PROPN: proper noun
• PUNCT: punctuation
• SCONJ: subordinating conjunction
• SYM: symbol
• VERB: verb
• X: other
• CARDINAL: any number that is not identified as a more specific type, like money, time, etc.
• DATE
• FAC: facilities like highways, bridges, airports, etc.
• GPE: Countries, states (or provinces), and cities
• LOC: any non-GPE location
• PRODUCT
• EVENT
• LANGUAGE: any named language
• MONEY: any monetary value or unit of money
• NORP: nationalities or religious groups
• ORG: any organization like a company, non-profit, school, etc.
• PERCENT: any number in [0, 100] followed by the percent % character
• PERSON
Natural Language Processing 69
1 (import spacy)
2
3 (setv nlp-model (spacy.load "en"))
4
5 (defn nlp [some-text]
6 (setv doc (nlp-model some-text))
7 (setv entities (lfor entity doc.ents [entity.text entity.label_]))
8 (setv j (doc.to_json))
9 (setv (get j "entities") entities)
10 j)
1 #!/usr/bin/env hy
2
3 (import [nlp-lib [nlp]])
4
5 (print
6 (nlp "President George Bush went to Mexico and he had a very good meal"))
7
8 (print
9 (nlp "Lucy threw a ball to Bill and he caught it"))
1 Marks-MacBook:nlp $ ./nlp_example.hy
2 {'text': 'President George Bush went to Mexico and he had a very good meal', 'ents':\
3 [{'start': 10, 'end': 21, 'label': 'PERSON'}, {'start': 30, 'end': 36, 'label': 'GP\
4 E'}], 'sents': [{'start': 0, 'end': 64}], 'tokens':
5
6 ..LOTS OF OUTPUT NOT SHOWN..
easy example, but often proper nouns that pronouns refer to are in previous sentences and resolving
coreference can be ambiguous and require knowledge of common word use and grammar. This
problem is now handled by deep learning transfer models like BERT³⁷.
In addition to installing spaCy you also need the library neuralcoref. Only specific versions of
spaCy and neuralcoref are compatible with each other. As of July 31, 2020 the following works to
get dependencies and run the example for this section:
Please note that version 2.1.3 of spaCy is older than the default version that pip installs. You might
want to create a new Python virtual environment for this example or if you use Anaconda then use
separate Anaconda environment.
Listing of coref_nlp_lib.hy contains a wrapper for spaCy’s coreference model:
Listing of coref_example.hy shows code to test the Hy spaCy and coreference wrapper:
³⁷https://github.com/google-research/bert
Natural Language Processing 71
1 #!/usr/bin/env hy
2
3 (import [coref-nlp-lib [coref-nlp]])
4
5 ;; tests:
6 (print (coref-nlp "President George Bush went to Mexico and he had a very good meal"\
7 ))
8 (print (coref-nlp "Lucy threw a ball to Bill and he caught it"))
1 Marks-MacBook:nlp $ ./coref_example.hy
2 {'corefs': 'President George Bush went to Mexico and President George Bush had a ver\
3 y good meal', 'clusters': [President George Bush: [President George Bush, he]], 'sco\
4 res': {President George Bush: {President George Bush: 1.5810412168502808}, George Bu\
5 sh: {George Bush: 4.11817741394043, President George Bush: -1.546141266822815}, Mexi\
6 co: {Mexico: 1.4138349294662476, President George Bush: -4.650205612182617, George B\
7 ush: -3.666614532470703}, he: {he: -0.5704692006111145, President George Bush: 9.385\
8 97583770752, George Bush: -1.4178757667541504, Mexico: -3.6565260887145996}, a very \
9 good meal: {a very good meal: 1.652894377708435, President George Bush: -2.554375886\
10 9171143, George Bush: -2.13267183303833, Mexico: -1.6889561414718628, he: -2.7667927\
11 742004395}}}
12
13 {'corefs': 'Lucy threw a ball to Bill and Bill caught a ball', 'clusters': [a ball: \
14 [a ball, it], Bill: [Bill, he]], 'scores': {Lucy: {Lucy: 0.41820740699768066}, a bal\
15 l: {a ball: 1.8033190965652466, Lucy: -2.721518039703369}, Bill: {Bill: 1.5611814260\
16 482788, Lucy: -2.8222298622131348, a ball: -1.806389570236206}, he: {he: -0.57600766\
17 42036438, Lucy: 3.054243326187134, a ball: -1.818403720855713, Bill: 3.0774276256561\
18 28}, it: {it: -1.0269954204559326, Lucy: -3.4972281455993652, a ball: -0.31290221214\
19 294434, Bill: -2.5343685150146484, he: -3.6687228679656982}}}
Anaphora resolution, also called coreference, refers to two or more words or phrases in an input text
refer to the same noun. This analysis usually entails identifying which noun phrases that pronouns
refer to.
Wrap-up
I spent several years of development time during the period from 1984 through 2015 working on
natural language processing technology and as a personal side project I sold commercial NLP
libraries that I wrote on my own time in Ruby and Common Lisp. The state-of-the-art of Deep
Learning enhanced NLP is very good and the open source spaCy library makes excellent use of both
Natural Language Processing 72
conventional NLP technology and pre-trained Deep Learning models. I no longer spend very much
time writing my own NLP libraries and instead use spaCy.
I urge you to read through the spaCy documentation³⁸ because we covered just basic functionality
here that we will also need in the later chapter on automatically generating data for Knowledge
Graphs. After working through the interactive REPL sessions and the examples in this chapter, you
should be able to translate any Python API example code to Hy.
³⁸https://spacy.io/api/doc
Datastores
I use flat files and the PostgreSQL relational database for most data storage and processing needs
in my consulting business over the last twenty years. For work on large data projects at Compass
Labs and Google I used Hadoop and Big Table. I will not cover big data datastores here, rather I will
concentrate on what I think of as “laptop development” requirements: a modest amount of data and
optimizing speed of development and ease of infrastructure setup. We will cover three datastores:
For graph data we will stick with RDF because it is a fairly widely used standard. Google, Microsoft,
Yahoo and Yandex support schema.org³⁹ for defining schemas for structured data on the web. In
the next chapter we will go into more details on RDF, here we look at the “plumbing” for using the
rdflib library to manipulate and query RDF data and how to export RDF data in several formats.
Then in a later chapter, we will develop tools to automatically generate RDF data from raw text as
a tool for developing customized Knowledge Graphs.
In one of my previous previous books Loving Common Lisp, or the Savvy Programmer’s Secret
Weapon⁴⁰ I also covered the general purpose graph database Neo4j which I like to use for some use
cases, but for the purposes of this book we stick with RDF.
Sqlite
We will cover two relational databases: Sqlite and PostgreSQL. Sqlite is an embedded database. There
are Sqlite libraries for many programming languages and here we use the Python library.
The following examples are simple but sufficient to show you how to open a single file Sqlite
database, add data, modify data, query data, and delete data. I assume that you have some familiarity
with relational databases, especially concepts like data columns and rows, and SQL queries.
Let’s start with putting common code for using Sqlite into a reusable library in the file sqlite_lib.hy:
³⁹https://schema.org/
⁴⁰https://leanpub.com/lovinglisp
Datastores 74
The function create-db in lines 3-6 creates a database from a file path if it does not already exist.
The function connection (lines 8-9) creates a persistent connection to a database defined by a file
path to the single file used for a Sqlite database. This connection can be reused. The function query
(lines 11-16) requires a connection object and a SQL query represented as a string, makes a database
query, and returns all matching data in nested lists.
The following listing of file sqlite_example.hyshows how to use this simple library:
1 #!/usr/bin/env hy
2
3 (import [sqlite-lib [create-db connection query]])
4
5 (defn test_sqlite-lib []
6 (setv dbpath ":memory:")
7 (create-db dbpath)
8 (setv conn (connection ":memory:"))
9 (query conn "CREATE TABLE people (name TEXT, email TEXT);")
10 (print
11 (query conn "INSERT INTO people VALUES ('Mark', '[email protected]')"))
12 (print
13 (query conn "INSERT INTO people VALUES ('Kiddo', '[email protected]')"))
14 (print
15 (query conn "SELECT * FROM people"))
16 (print
17 (query conn "UPDATE people SET name = ? WHERE email = ?"
18 ["Mark Watson" "[email protected]"]))
Datastores 75
19 (print
20 (query conn "SELECT * FROM people"))
21 (print
22 (query conn "DELETE FROM people WHERE name=?" ["Kiddo"]))
23 (print
24 (query conn "SELECT * FROM people"))
25 (conn.close))
26
27 (test_sqlite-lib)
We opened an in-memory database in lines 7 and 8 but we could have also created a persistent
database on disk using, for example, “test_database.db” instead of :memory. In line 9 we create a
database table with just two columns, each column holding string values.
In lines 15, 20, and 24 we are using a wild card query using the asterisk character to return all column
values for each matched row in the database.
Running the example program produces the following output:
1 $ ./sqlite_example.hy
2 2.6.0
3 []
4 []
5 [('Mark', '[email protected]'), ('Kiddo', '[email protected]')]
6 []
7 [('Mark Watson', '[email protected]'), ('Kiddo', '[email protected]')]
8 []
9 [('Mark Watson', '[email protected]')]
Line 2 shows the version of SQlist we are using. The lists in lines 1-2, 4, and 6 are empty because the
functions to create a table, insert data into a table, update a row in a table, and delete rows do not
return values.
In the next section we will see how PostgreSQL treats JSON data as a native data type. For sqlite,
you can store JSON data as a “dumped” string value but you can’t query by key/value pairs in the
data. You can encode JSON as a string and then decode it back to JSON (or as a dictionary) using:
PostgreSQL
We just saw use cases for the Sqlite embedded database. Now we look at my favorite general purpose
database, PostgreSQL. The PostgreSQL database server is available as a managed service on most
cloud providers and it is easy to also run a PostgreSQL server on your laptop or on a VPS or server.
We will use the psycopg⁴¹ PostgreSQL adapter that is compatible with CPython and can be installed
using:
The following material is self-contained but before using PostgreSQL and psycopg in your own
applications I recommend that you reference the psycopg documentation.
macOS
For macOS we use the PostgreSQL application and we will start by using the postgres command
line utility to create a new database and table in this database. Using postgres account, create a new
database hybook:
⁴¹http://initd.org/psycopg/
Datastores 77
Linux
For Ubuntu Linux first install PostgreSQL and then use sudo to use the account postgres:
To start a local server:
1 sudo su - postgres
2 /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start
1 sudo su - postgres
2 /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile stop
When the PostgreSQL server is running we can use the psql command line program:
1 sudo su - postgres
2 psql
3
4 postgres@pop-os:~$ psql -d "hybook"
5 psql (10.7 (Ubuntu 10.7-0ubuntu0.18.10.1))
6 Type "help" for help.
7
8 hybook=# CREATE TABLE news (uri VARCHAR(50) not null, title VARCHAR(50), articletext\
9 VARCHAR(500), nlpdata VARCHAR(50));
10 CREATE TABLE
11 hybook=# \d
12 List of relations
13 Schema | Name | Type | Owner
14 --------+------+-------+----------
15 public | news | table | postgres
16 (1 row)
Datastores 78
1 Marks-MacBook:datastores $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import json psycopg2)
4 => (setv conn (psycopg2.connect :dbname "hybook" :user "markw"))
5 => (setv cur (conn.cursor))
6 => (cur.execute "INSERT INTO news VALUES (%s, %s, %s, %s)"
7 ["http://knowledgebooks.com/schema" "test schema"
8 "text in article" (json.dumps {"type" "news"})])
9 => (conn.commit)
10 => (cur.execute "SELECT * FROM news")
11 => (for [record cur]
12 ... (print record))
13 ('http://knowledgebooks.com/schema', 'test schema', 'text in article',
14 '{"type": "news"}')
15 => (cur.execute "SELECT nlpdata FROM news")
16 => (for [record cur]
17 ... (print record))
18 ('{"type": "news"}',)
19 => (cur.execute "SELECT nlpdata FROM news")
20 => (for [record cur]
21 ... (print (json.loads (first record))))
22 {'type': 'news'}
23 =>
In lines 6-8 and 13-14 you notice that I am using PostgreSQL’s native JSON support.
As with most of the material in this book, I hope that you have a Hy REPL open and are
experimenting with the APIs and code in the book’s interactive REPL examples.
The file postgres_lib.hy wraps commonly used functionality for accessing a database, adding,
modifying, and querying data in a short reusable library:
Datastores 79
The function query in lines 8-11 executes any SQL comands so in addition to querying a database,
it can also be used with appropriate SQL commands to delete rows, update rows, and create and
destroy tables.
The following file postgres_example.hy contains examples for using the library we just defined:
1 #!/usr/bin/env hy
2
3 (import [postgres-lib [connection-and-cursor query]])
4
5 (defn test-postgres-lib []
6 (setv [conn cursor] (connection-and-cursor "hybook" "markw"))
7 (query cursor "CREATE TABLE people (name TEXT, email TEXT);")
8 (conn.commit)
9 (query cursor "INSERT INTO people VALUES ('Mark', '[email protected]')")
10 (query cursor "INSERT INTO people VALUES ('Kiddo', '[email protected]')")
11 (conn.commit)
12 (query cursor "SELECT * FROM people")
13 (print (cursor.fetchall))
14 (query cursor "UPDATE people SET name = %s WHERE email = %s"
15 ["Mark Watson" "[email protected]"])
16 (query cursor "SELECT * FROM people")
17 (print (cursor.fetchall))
18 (query cursor "DELETE FROM people WHERE name = %s" ["Kiddo"])
19 (query cursor "SELECT * FROM people")
20 (print (cursor.fetchall))
21 (query cursor "DROP TABLE people;")
22 (conn.commit)
23 (conn.close))
24
25 (test-postgres-lib)
Datastores 80
1 Marks-MacBook:datastores $ ./postgres_example.hy
2 [('Mark', '[email protected]'), ('Kiddo', '[email protected]')]
3 [('Kiddo', '[email protected]'), ('Mark Watson', '[email protected]')]
4 [('Mark Watson', '[email protected]')]
I use PostgreSQL more than any other datastore and taking the time to learn how to manage
PostgreSQL servers and write application software will save you time and effort when you are
prototyping new ideas or developing data oriented product at work. I love using PostgreSQL and
personally, I only use Sqlite for very small database tasks or applications.
1 Marks-MacBook:datastores $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import [rdflib [Graph]])
4 => (setv graph (Graph))
5 => (graph.load "http://markwatson.com/index.rdf")
6 => (for [[subject predicate object] graph]
7 ... (print subject "\n " predicate "\n " object " ."))
8 http://markwatson.com/index.rdf#mark_watson_consulting_services
9 http://www.w3.org/1999/02/22-rdf-syntax-ns#label
10 Mark Watson Consulting Services .
11 http://markwatson.com/index.rdf#mark_watson
12 http://www.w3.org/2000/10/swap/pim/contact#firstName
13 Mark .
14 http://markwatson.com/index.rdf#mark_watson
15 http://www.ontoweb.org/ontology/1#name
16 Mark Watson .
17 http://www.markwatson.com/
18 http://purl.org/dc/elements/1.1/language
19 en-us .
20 http://markwatson.com/index.rdf#mark_watson
21 http://www.ontoweb.org/ontology/1#researchTopic
22 Semantic Web .
23 http://www.markwatson.com/
24 http://purl.org/dc/elements/1.1/date
25 2005-7-10 .
26 http://markwatson.com/index.rdf#mark_watson
27 http://www.ontoweb.org/ontology/1#researchTopic
28 RDF and RDF Schema .
29 http://markwatson.com/index.rdf#mark_watson
30 http://www.ontoweb.org/ontology/1#researchTopic
31 ontologies .
32 http://www.markwatson.com/
33 http://purl.org/dc/elements/1.1/title
34 Mark Watson's Home Page .
35 http://markwatson.com/index.rdf#mark_watson
36 http://www.w3.org/2000/10/swap/pim/contact#mailbox
37 mailto:[email protected] .
38 http://markwatson.com/index.rdf#mark_watson
39 http://www.w3.org/2000/10/swap/pim/contact#homepage
40 http://www.markwatson.com/ .
41 http://markwatson.com/index.rdf#mark_watson
42 http://www.w3.org/2000/10/swap/pim/contact#fullName
43 Mark Watson .
Datastores 82
44 http://markwatson.com/index.rdf#mark_watson_consulting_services
45 http://www.ontoweb.org/ontology/1#name
46 Mark Watson Consulting Services .
47 http://markwatson.com/index.rdf#mark_watson
48 http://www.w3.org/2000/10/swap/pim/contact#company
49 Mark Watson Consulting Services .
50 http://markwatson.com/index.rdf#mark_watson
51 http://www.w3.org/1999/02/22-rdf-syntax-ns#type
52 http://www.w3.org/2000/10/swap/pim/contact#Person .
53 http://markwatson.com/index.rdf#mark_watson
54 http://www.w3.org/1999/02/22-rdf-syntax-ns#value
55 Mark Watson .
56 http://www.markwatson.com/
57 http://purl.org/dc/elements/1.1/creator
58 http://markwatson.com/index.rdf#mark_watson .
59 http://www.markwatson.com/
60 http://purl.org/dc/elements/1.1/description
61
62 Mark Watson is the author of 16 published books and a consultant specializing \
63 in artificial intelligence and Java technologies.
64 .
65 http://markwatson.com/index.rdf#mark_watson
66 http://www.w3.org/1999/02/22-rdf-syntax-ns#type
67 http://www.ontoweb.org/ontology/1#Person .
68 http://markwatson.com/index.rdf#mark_watson
69 http://www.w3.org/2000/10/swap/pim/contact#motherTongue
70 en .
71 http://markwatson.com/index.rdf#mark_watson
72 http://www.w3.org/1999/02/22-rdf-syntax-ns#type
73 http://markwatson.com/index.rdf#Consultant .
74 http://markwatson.com/index.rdf#mark_watson_consulting_services
75 http://www.w3.org/1999/02/22-rdf-syntax-ns#type
76 http://www.ontoweb.org/ontology/1#Organization .
77 http://markwatson.com/index.rdf#mark_watson
78 http://www.w3.org/1999/02/22-rdf-syntax-ns#label
79 Mark Watson .
80 http://markwatson.com/index.rdf#Sun_ONE
81 http://www.w3.org/1999/02/22-rdf-syntax-ns#type
82 http://www.ontoweb.org/ontology/1#Book .
83 =>
We will cover the SPARQL query language in more detail in the next chapter but for now, notice
that SPARQL is similar to SQL queries. SPARQL queries can find triples in a graph matching simple
Datastores 83
patterns, match complex patterns, and update and delete triples in a graph. The following simple
SPARQL query finds all triples with the predicate equal to http://www.w3.org/2000/10/swap/pim/
contact#company and prints out the subject and object of any matching triples:
We will see more examples of the SPARQL query language in the next chapter. For now, notice that
the general form of a select query statement is a list of query variables (names beginning with a
question mark) and a where clause in curly brackets that contains matching patterns. This SPARQL
query is simple, but like SQL queries, SPARQL queries can get very complex. I only lightly cover
SPARQL in this book. You can get PDF copies of my two older semantic web books for free: Practical
Semantic Web and Linked Data Applications, Java, Scala, Clojure, and JRuby Edition⁴² and Practical
Semantic Web and Linked Data Applications, Common Lisp Edition⁴³. There are links to relevant
git repos and other information on my book web page⁴⁴.
As I mentioned, the RDF data on my web site has been essentially unchanged since 2005. What if
I wanted to update it noting that more recently I worked as a contractor at Google and as a Deep
Learning engineering manager at Capital One? In the following listing, continuing the same REPL
session, I will add two RDF statements indicating additional jobs and show how to serialize the new
updated graph in three formats: XML, turtle (my favorite RDF notation) and NT:
103 g Services
104 http://markwatson.com/index.rdf#mark_watson contact company: Google
105 http://markwatson.com/index.rdf#mark_watson contact company: Capital One
106 =>
107 => (graph.serialize :format "pretty-xml")
108 <?xml version="1.0" encoding="utf-8"?>
109 <rdf:RDF\n xmlns:dc="http://purl.org/dc/elements/1.1/"
110 xmlns:contact="http://www.w3.org/2000/10/swap/pim/contact#"
111 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
112 xmlns:ow="http://www.ontoweb.org/ontology/1#"
113 xmlns:ns1="http://markwatson.com/index.rdf#">
114
115 LOTS OF STUFF NOT SHOWN
116
117 </rdf:Description>\n</rdf:RDF>
I like Turtle RDF notation better than the XML notation because Turtle is easier to read and
understand. Here, on line 118 we serialize the graph (with new nodes added above in lines 90 to
96) to Turtle:
In addition to the Turtle format I also use the simpler NT format that puts URI prefixes inline and
unlike Turtle does not use prefix abrieviations. Here in line 159 we serialize to NT format:
Wrap-up
We will go into much more detail on practical uses of RDF and SPARQL in the next chapter. I hope
that you worked through the REPL examples in this section because if you understand the basics of
using the rdflib then you will have an easier time understanding the more abstract material in the
next chapter.
⁴⁵https://github.com/RDFLib/rdflib-sqlalchemy
⁴⁶https://en.wikipedia.org/wiki/Virtuoso_Universal_Server
⁴⁷https://www.ontotext.com/products/graphdb/graphdb-free/
Linked Data and the Semantic Web
Tim Berners Lee, James Hendler, and Ora Lassila wrote in 2001 an article for Scientific American
where they introduced the term Semantic Web. Here I do not capitalize semantic web and use the
similar term linked data somewhat interchangeably with semantic web.
I assume that you read the section RDF Data Using the “rdflib” Library in the last chapter.
In the same way that the web allows links between related web pages, linked data supports linking
associated data on the web together. I view linked data as a relatively simple way to specify
relationships between data sources on the web while the semantic web has a much larger vision:
the semantic web has the potential to be the entirety of human knowledge represented as data on
the web in a form that software agents can work with to answer questions, perform research, and
to infer new data from existing data.
While the “web” describes information for human readers, the semantic web is meant to provide
structured data for ingestion by software agents. This distinction will be clear as we compare
WikiPedia, made for human readers, with DBPedia which uses the info boxes on WikiPedia topics to
automatically extract RDF data describing WikiPedia topics. Let’s look at the WikiPedia topic for the
town I live in, Sedona Arizona, and show how the info box on the English version of the WikiPedia
topic page for Sedona https://en.wikipedia.org/wiki/Sedona,_Arizona⁴⁸ maps to the DBPedia page
http://dbpedia.org/page/Sedona,_Arizona⁴⁹. Please open both of these WikiPedia and DBPedia URIs
in two browser tabs and keep them open for reference.
I assume that the format of the WikiPedia page is familiar so let’s look at the DBPedia page for
Sedona that in human readable form shows the RDF statements with Sedona Arizona as the subject.
RDF is used to model and represent data. RDF is defined by three values so an instance of an RDF
statement is called a triple with three parts:
The subject for each Sedona related triple is the above URI for the DBPedia human readable page.
The subject and property references in an RDF triple will almost always be a URI that can ground
an entity to information on the web. The human readable page for Sedona lists several properties
and the values of these properties. One of the properties is “dbo:areaCode” where “dbo” is a name
space reference (in this case for a DatatypeProperty⁵⁰).
⁴⁸https://en.wikipedia.org/wiki/Sedona,_Arizona
⁴⁹http://dbpedia.org/page/Sedona,_Arizona
⁵⁰http://www.w3.org/2002/07/owl#DatatypeProperty
Linked Data and the Semantic Web 88
The following two figures show an abstract representation of linked data and then a sample of linked
data with actual web URIs for resources and properties:
Concrete example using RDF seen in last chapter showing the RDF representation with 2 Resources, 2 literal values,
and 3 Properties
We saw a SPARQL Query (SPARQL for RDF data is similar to SQL for relational database queries)
in the last chapter. Let’s look at another example using the RDF in the last figure:
Linked Data and the Semantic Web 89
This query should return the result “Sun ONE Services - J2EE”. If you wanted to query for all URI
resources that are books with the literal value of their titles, then you can use:
1 select ?s ?v where { ?s
2 <http://www.ontoweb.org/ontology/1#booktitle>
3 ?v }
Note that ?s and ?v are arbitrary query variable names, here standing for “subject” and “value”. You
can use more descriptive variable names like:
We will be diving a little deeper into RDF examples in the next chapter when we write a tool
for generating RDF data from raw text input. For now I want you to understand the idea of RDF
statements represented as triples, that web URIs represent things, properties, and sometimes values,
and that URIs can be followed manually (often called “dereferencing”) to see what they reference in
human readable form.
• RDF https://www.w3.org/TR/rdf-syntax-grammar/⁵¹
⁵¹https://www.w3.org/TR/rdf-syntax-grammar/
Linked Data and the Semantic Web 90
• RDFS https://www.w3.org/TR/rdf-schema/⁵²
• OWL http://www.w3.org/2002/07/owl#⁵³
• XSD http://www.w3.org/2001/XMLSchema#⁵⁴
• FOAF http://xmlns.com/foaf/0.1/⁵⁵
• SKOS http://www.w3.org/2004/02/skos/core#⁵⁶
• DOAP http://usefulinc.com/ns/doap#⁵⁷
• DC http://purl.org/dc/elements/1.1/⁵⁸
• DCTERMS http://purl.org/dc/terms/⁵⁹
• VOID http://rdfs.org/ns/void#⁶⁰
Let’s look into the Friend of a Friend (FOAF) namespace. Click on the above link for FOAF
http://xmlns.com/foaf/0.1/⁶¹ and find the definitions for the FOAF Core:
1 Agent
2 Person
3 name
4 title
5 img
6 depiction (depicts)
7 familyName
8 givenName
9 knows
10 based_near
11 age
12 made (maker)
13 primaryTopic (primaryTopicOf)
14 Project
15 Organization
16 Group
17 member
18 Document
19 Image
⁵²https://www.w3.org/TR/rdf-schema/
⁵³http://www.w3.org/2002/07/owl#
⁵⁴http://www.w3.org/2001/XMLSchema#
⁵⁵http://xmlns.com/foaf/0.1/
⁵⁶http://www.w3.org/2004/02/skos/core#
⁵⁷http://usefulinc.com/ns/doap#
⁵⁸http://purl.org/dc/elements/1.1/
⁵⁹http://purl.org/dc/terms/
⁶⁰http://rdfs.org/ns/void#
⁶¹http://xmlns.com/foaf/0.1/
Linked Data and the Semantic Web 91
1 nick
2 mbox
3 homepage
4 weblog
5 openid
6 jabberID
7 mbox_sha1sum
8 interest
9 topic_interest
10 topic (page)
11 workplaceHomepage
12 workInfoHomepage
13 schoolHomepage
14 publications
15 currentProject
16 pastProject
17 account
18 OnlineAccount
19 accountName
20 accountServiceHomepage
21 PersonalProfileDocument
22 tipjar
23 sha1
24 thumbnail
25 logo
You now have seen a few common Schemas for RDF data. Another Schema that is widely used for
annotating web sites, that we won’t need for our examples here, is schema.org⁶². Let’s now use a Hy
REPL session to explore namespaces and programatically create RDF using rdflib:
1 Marks-MacBook:database $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import [rdflib.namespace [FOAF]])
4 => FOAF
5 Namespace('http://xmlns.com/foaf/0.1/')
6 => FOAF.name
7 rdflib.term.URIRef('http://xmlns.com/foaf/0.1/name')
8 => FOAF.title
9 rdflib.term.URIRef('http://xmlns.com/foaf/0.1/title')
10 => (import rdflib)
11 => (setv graph (rdflib.Graph))
⁶²https://schema.org
Linked Data and the Semantic Web 92
55 =>
If I depend on a library, regardless of the programming language, I like to keep an up-to-date copy
of the source code ready at hand. There is sometimes no substitute for having library code available
to read.
In the next chapter we will use natural language processing to extract structured information from
raw text and automatically generate RDF data.
⁶³https://jena.apache.org/tutorials/sparql.html
⁶⁴https://github.com/RDFLib/rdflib
Knowledge Graph Creator
A Knowledge Graph, that I often abbreviate as KG, is a graph database using a schema to define
types (both objects and relationships between objects) and properties that link property values to
objects. The term “Knowledge Graph” is both a general term and also sometimes refers to the specific
Knowledge Graph used at Google which I worked with while working there in 2013. Here, we use
KG to reference the general technology of storing knowledge in graph databases.
The application we develop here, the Knowledge Graph Creator (which I often refer to as KGCreator)
is a utility that I use to generate small Knowledge Graphs from input text.
Knowledge engineering and knowledge representation are disciplines that started in the 1980s and
are still both current research topics and used in industry. I view linked data, the semantic web, and
KGs as extensions of this earlier work.
We base our work here on RDF. There is a general type of KGs that are also widely used in industry
and that we will not cover here: property graphs, as used in Neo4J. Property graphs are general
graphs that place no restrictions on the number of links a graph node may have and allow general
data structures to be stored as node data and for the property links between nodes. Property links
can have attributes, like nodes in the graph.
Semantic web data as represented by subject/property/value RDF triples are more constrained than
property graphs but support powerful logic inferencing to better use data that is implicit in a graph
but not explicitly stated (i.e., data is more easily inferred).
We covered RDF data in some detail in the last chapter. Here we will implement a toolset for
converting unstructured text into RDF data using a few schema definitions from schema.org⁶⁵. I
believe in both the RDF and the general graph database approaches but here we will just use RDF.
Historically Knowledge Graphs used semantic web technology like Resource Description Frame-
work (RDF)⁶⁶ and Web Ontology Language (OWL)⁶⁷. I wrote two books in 2010 on semantic web
technologies and you can get free PDFs for the Common Lisp version⁶⁸ (code is here⁶⁹) and the
Java/Clojure/Scala version⁷⁰ (code is here⁷¹). These free books might interest you after working
through the material in this chapter.
I have an ongoing personal research project for creating knowledge graphs from various data sources.
You can read more at my KGCreator web site⁷². I have simplified versions of my KGCreator software
⁶⁵https://schema.org/
⁶⁶https://en.wikipedia.org/wiki/Resource_Description_Framework
⁶⁷https://en.wikipedia.org/wiki/Web_Ontology_Language
⁶⁸http://markwatson.com/opencontentdata/book_lisp.pdf
⁶⁹https://github.com/mark-watson/lisp_practical_semantic_web
⁷⁰http://markwatson.com/opencontentdata/book_java.pdf
⁷¹https://github.com/mark-watson/java_practical_semantic_web
⁷²http://www.kgcreator.com/
Knowledge Graph Creator 95
implemented in both my Haskell Book⁷³ and in my most recent Common Lisp book⁷⁴. The example
here is similar to my Common Lisp implementation, except that it is implemented in the Hy language
and I only support generating RDF. The examples in my Haskell and Common Lisp books also
generate data for the Neo4J graph database.
What is a KG? It is a modern way to organize and access structured data and integrate data and
metadata with other automated systems.
A Knowledge Graph is different from just a graph database containing graph data. The difference
is that a KG will in general use Schemas, Taxonomy’s and Ontology’s that define the allowed types
and structure of data and allowed relationships.
There is also an executable aspect of KGs since their primary use may be to support other systems
in an organization.
• At Google we used their Knowledge Graph for researching new internal systems that were
built on their standard Knowledge Graph, with new schemas and data added.
• Digital transformations: start by using a KG to hold metadata for current data in already
existing databases. A KG of metadata can provide you with a virtual data lake. It is common to
build a large data lake and then have staff not be able to find data. Don’t try to do everything
at once.
• Capture and preserve senior human expertise. The act of building an Ontology for in-house
knowledge helps to understand how to organize data and provides people with a common
vocabulary to discuss and model business processes.
• KYC (Know Your Customer) applications using data from many diverse data sources.
• Take advantage of expertise in a domain (e.g., healthcare or financial services) to build a
Taxonomy and Ontology to use to organize available data. For most domains, there are
standard existing Schemas, Taxonomy’s and Ontology’s that can be identified and used as-
is or extended for your organization.
To get started:
⁷³https://leanpub.com/haskell-cookbook
⁷⁴https://leanpub.com/lovinglisp
Knowledge Graph Creator 96
A good way to start is to identify a single problem, determine the best data sources to use, define
an Ontology that is just sufficient to solve the current problem and build a prototype “vertical slice”
application. Lessons learned with a quick prototype will inform you on what was valuable and what
to put effort into when expanding your KG. Start small and don’t try to build a huge system without
taking many small development and evaluation steps.
What about KGs for small organizations? Small companies have less development resources
but starting small and implementing a system that models the key data relationships, customer
relationships, etc., does not require excessive resources. Just capturing where data comes from and
who is responsible for maintaining important data sources can be valuable.
What about KGs for individuals? Given the effort involved in building custom KGs, one possible
individual use case is developing KGs for commercial sale.
The application that we develop next is one way to quickly bootstrap a new KG by populating it with
automatically generated RDF than can be manually curated by removing statements and adding new
statements as appropriate.
$ ls test_data
test1.meta test1.txt test2.meta test2.txt test3.meta test3.txt
The *.txt files contain plain text for analysis and the *.meta files contain the original web source URI
for the corresponding *.txt files. Using the spaCy library and Python/Hy’s standard libraries for file
access, the KGCreator is simple to implement. Here is the overall design of this example:
Knowledge Graph Creator 97
We will develop two versions of the Knowledge Graph Creator. The first generates RDF that uses
string values for the object part of generated RDF statements. The second implementation attempts
to resolve these string values to DBPedia URIs.
Using only the spaCy NLP library that we used earlier and the built in Hy/Python libraries, this first
example (uses strings a object values) is implemented in just 58 lines of Hy code that is seen in the
following three code listings:
1 #!/usr/bin/env hy
2
3 (import [os [scandir]])
4 (import [os.path [splitext exists]])
5 (import spacy)
6
7 (setv nlp-model (spacy.load "en"))
8
9 (defn find-entities-in-text [some-text]
10 (defn clean [s]
11 (.strip (.replace s "\n" " ")))
12 (setv doc (nlp-model some-text))
13 (map list (lfor entity doc.ents [(clean entity.text) entity.label_])))
In lines 3 and 4 we import three standard Python utilities we need for finding all files in a directory,
checking to see if a file exists, and splitting text into tokens. In line 7 we load the English language
spaCy model and save the value of the model in the variable nlp-model. The function find-entities-
in-text uses the spaCy English language model to find entities like organizations, people, etc., in
text and cleans entity names by removing new line characters and other unnecessary white space
(nested function clean in lines 10 and 11). We can run a test in a REPL:
Knowledge Graph Creator 98
=> (list (find-entities-in-text "John Smith went to Los Angeles to work at IBM"))
[['John Smith', 'PERSON'], ['Los Angeles', 'GPE'], ['IBM', 'ORG']]
The function find-entities-in-text returns a map object so I wrapped the results in a list to print out
the entities in the test sentence. The entity types used by spaCy were defined in an earlier chapter,
here we just use the entity types defined in lines 21-26 in the following listing:
In lines 28-39 we open an output file for writing generated RDF data and loop through all text files
in the input directory and call the function process-file for each text + meta file pair in the input
directory:
We will look at generated output, problems with it, and how to fix these problems in the next section.
<https://newsshop.com/may/a1023.html>
<https://schema.org/nationality> "Portuguese" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "Banco Espirito Santo SA" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Person> "John Evans" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "Banco Espirito" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "The Wall Street Journal" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "IBM" .
<https://newsshop.com/may/a1023.html>
Knowledge Graph Creator 100
<https://schema.org/location> "Canada" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "Australian Broadcasting Corporation" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Person> "Frank Smith" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "Australian Writers Guild" .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization> "American University" .
<https://localnews.com/june/z902.html>
<https://schema.org/Organization> "The Wall Street Journal" .
<https://localnews.com/june/z902.html>
<https://schema.org/location> "Mexico" .
<https://localnews.com/june/z902.html>
<https://schema.org/location> "Canada" .
<https://localnews.com/june/z902.html>
<https://schema.org/Person> "Bill Clinton" .
<https://localnews.com/june/z902.html>
<https://schema.org/Organization> "IBM" .
<https://localnews.com/june/z902.html>
<https://schema.org/Organization> "Microsoft" .
<https://abcnews.go.com/US/violent-long-lasting-tornadoes-threaten-oklahoma-texas/st\
ory?id=63146361>
<https://schema.org/Person> "Jane Deerborn" .
<https://abcnews.go.com/US/violent-long-lasting-tornadoes-threaten-oklahoma-texas/st\
ory?id=63146361>
<https://schema.org/location> "Texas" .
Because we used literal values, notice how for example the node for the entity IBM is not shared
and thus a software agent using this RDF data cannot, for example, infer relationships between two
news sources that both have articles about IBM. We will work on a solution to this problem in the
next section.
⁷⁵http://dbpedia.org
⁷⁶https://www.wikidata.org/wiki/Wikidata:Database_download
Knowledge Graph Creator 102
We also make a change in the function data2Rdf to use the map v2umap:
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization>
<http://dbpedia.org/page/IBM> .
<https://newsshop.com/may/a1023.html>
<https://schema.org/Organization>
<http://dbpedia.org/page/Banco_Esp%C3%ADrito_Santo> .
Now when we visualize generated RDF, we share nodes for The Wallstreet Journal and IBM:
Part of the RDF graph that shows shared nodes when URIs are used for RDF values instead of literal strings
While literal values sometimes are useful in generated RDF, using literals for the values in RDF
triples prevents types of queries and inference that can be performed on the data.
Wrap-up
In the field of Artificial Intelligence there are two topics that get me the most excited and I have
been fortunate to be paid to work on both: Deep Learning and Knowledge Graphs. Here we have just
Knowledge Graph Creator 103
touched the surface for creating data for Knowledge Graphs but I hope that between this chapter and
the material on RDF in the chapter Datastores that you have enough information and experience
playing with the examples to get started prototyping a Knowledge Graph in your organizartion. My
advice is to “start small” by picking a problem that your organization has that can be solved by not
moving data around, but rather, by creating a custom Knowledge Graph for metadata for existing
information in your organization.
Knowledge Graph Navigator
The Knowledge Graph Navigator (which I will often refer to as KGN) is a tool for processing a set of
entity names and automatically exploring the public Knowledge Graph DBPedia⁷⁷ using SPARQL
queries. I wrote KGN in Common Lisp for my own use to automate some things I used to do manually
when exploring Knowledge Graphs, and later thought that KGN might be useful also for educational
purposes. KGN uses NLP code developed in earlier chapters and we will reuse that code with a short
review of using the APIs.
Please note that the example is a simplified version that I first wrote in Common Lisp and is also an
example in my book Loving Common Lisp, or the Savvy Programmer’s Secret Weapon⁷⁸ that you
can read for free online. If you are interested you can see screen shots of the Common Lisp version
here⁷⁹.
The following two screen shots show the text based user interface for this example. This example
application asks the user for a list of entity names and uses SPARQL queries to discover potential
matches in DBPedia. We use the python library PyInquirer⁸⁰ for requesting entity names and then
to show the user a list of matches from DBPedia. The following screen shot shows these steps:
To select the entities of interest, the user uses a space character to select or deselect an entity and
the return (or enter) key to accept the list selections.
After the user selects entities from the list, the list disappears. The next screen shot shows the output
from this example after the user finishes selecting entities of interest:
⁷⁷http://dbpedia.org
⁷⁸https://leanpub.com/lovinglisp
⁷⁹http://www.knowledgegraphnavigator.com/screen/
⁸⁰https://github.com/CITGuru/PyInquirer
Knowledge Graph Navigator 105
The code for this application is in the directory kgn. You will need to install the following Python
library that supports console/text user interfaces:
You will also need the spacy library and language model that we used in the earlier chapter on
natural language processing. If you have not already done so, install these requirements:
After listing the generated SPARQL for finding information for the entities in the query, KGN
searches for relationships between these entities. These discovered relationships can be seen at the
end of the last screen shot. Please note that this step makes SPARQL queries on O(n^2) where n is
the number of entities. Local caching of SPARQL queries to DBPedia helps make processing many
entities possible.
Knowledge Graph Navigator 106
Every time KGN makes a SPARQL query web service call to DBPedia the query and response are
cached in a SQLite database in ∼/.kgn_hy_cache.db which can greatly speed up the program,
especially in development mode when testing a set of queries. This caching also takes some load
off of the public DBPedia endpoint, which is a polite thing to do.
(import spacy)
The entity type “GPE” indicates that the entity is some type of location.
Here we store structured data from SPARQL queries as JSON data serialized as string values.
SPARQL Utilities
We will use the caching code from the last section and also the standard Python library requests
to access the DBPedia servers. The following code is found in the file sparql.hy and also provides
Knowledge Graph Navigator 108
support for using both DBPedia and WikiData. We only use DBPedia in this chapter but when you
start incorporating SPARQL queries into applications that you write, you will also probably want to
use WikiData.
The function do-query-helper contains generic code for SPARQL queries and is used in functions
wikidata-sparql and dbpedia-sparql:
(import json)
(import requests)
(require [hy.contrib.walk [let]])
qr)
[]))))
$ hy
hy 0.18.0 using CPython(default) 3.7.4 on Darwin
=> (import sparql)
table dbpedia already exists
=> (sparql.dbpedia-sparql
"select ?s ?p ?o { ?s ?p ?o } limit 1")
[[['s', 'http://www.openlinksw.com/virtrdf-data-formats#default-iid'],
['p', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type'],
['o', 'http://www.openlinksw.com/schemas/virtrdf#QuadMapFormat']]]
=>
This is a wild-card SPARQL query that will match any of the 9.5 billion RDF triples in DBPedia and
return just one result.
This caching layer greatly speeds up my own personal use of KGN. Without caching, queries that
contain many entity references simply take too long to run.
You have seen colorized SPARQL in the two screen shots at the beginning of this chapter.
Knowledge Graph Navigator 111
We embed a SPARQL query that has placeholders for the entity name and type. The filter expression
specifies that we only want triple results with comment values in the English language by using
(lang(?comment) = ‘en’):
1 #!/usr/bin/env hy
2
3 (import [sparql [dbpedia-sparql]])
4 (import [colorize [colorize-sparql]])
5
6 (import [pprint [pprint]])
7 (require [hy.contrib.walk [let]])
8
9 (defn dbpedia-get-entities-by-name [name dbpedia-type]
10 (let [sparql
11 (.format "select distinct ?s ?comment {{ ?s ?p \"{}\"@en . ?s <http://www.w3\
12 .org/2000/01/rdf-schema#comment> ?comment . FILTER (lang(?comment) = 'en') . ?s <\
13 http://www.w3.org/1999/02/22-rdf-syntax-ns#type> {} . }} limit 15" name dbpedia-type\
14 )]
15 (print "Generated SPARQL to get DBPedia entity URIs from a name:")
16 (print (colorize-sparql sparql))
17 (dbpedia-sparql sparql)))
Here is an example:
Knowledge Graph Navigator 112
1 (setv entity-type-to-type-uri
2 {"PERSON" "<http://dbpedia.org/ontology/Person>"
3 "GPE" "<http://dbpedia.org/ontology/Place>"
4 "ORG" "<http://dbpedia.org/ontology/Organisation>"
5 })
When we get entity results from DBPedia, the comments describing entities can be a few paragraphs
of text. We want to shorten the comments so they fit in a single line of the entity selection list that
we have seen earlier. The following code defines a comment shortening function and also a global
variable that we will use to store the entity URIs for each shortened comment:
In line 5, we use the function assoc to add a key and value pair to an existing dictionary short-
comment-to-uri.
Knowledge Graph Navigator 113
Finally, let’s look at the main application loop. In line 4 we are using the function get-query (defined
in file textui.hy) to get a list of entity names from the user. In line 7 we use the function entities-
in-text that we saw earlier to map text to entity types and names. In the nested loops in lines 13-26
we build one line descriptions of people, place, and organizations that we will use to show the user
a menu for selecting entities found in DBPedia from the original query. We are giving the use a
chance to select only the discovered entities that they are interested in.
In lines 33-35 we are converting the shortened comment strings the user selected back to DBPedia
entity URIs. Finally in line 36 we use the function entity-results->relationship-links to find
relationships between the user selected entities.
1 (defn kgn []
2 (while
3 True
4 (let [query (get-query)
5 emap {}]
6 (if (or (= query "quit") (= query "q"))
7 (break))
8 (setv elist (entities-in-text query))
9 (setv people-found-on-dbpedia [])
10 (setv places-found-on-dbpedia [])
11 (setv organizations-found-on-dbpedia [])
12 (global short-comment-to-uri)
13 (setv short-comment-to-uri {})
14 (for [key elist]
15 (setv type-uri (get entity-type-to-type-uri key))
16 (for [name (get elist key)]
17 (setv dbp (dbpedia-get-entities-by-name name type-uri))
18 (for [d dbp]
19 (setv short-comment (shorten-comment (second (second d))
20 (second (first d))))
21 (if (= key "PERSON")
22 (.extend people-found-on-dbpedia [(+ name " || " short-comment)]))
23 (if (= key "GPE")
24 (.extend places-found-on-dbpedia [(+ name " || " short-comment)]))
25 (if (= key "ORG")
26 (.extend organizations-found-on-dbpedia
27 [(+ name " || " short-comment)])))))
28 (setv user-selected-entities
29 (select-entities
30 people-found-on-dbpedia
31 places-found-on-dbpedia
32 organizations-found-on-dbpedia))
Knowledge Graph Navigator 114
If you have not already done so, I hope you experiment running this example application. The first
time you specify an entity name expect some delay while DBPedia is accessed. Thereafter the cache
will make the application more responsive when you use the same name again in a different query.
Wrap-up
If you enjoyed running and experimenting with this example and want to modify it for your own
projects then I hope that I provided a sufficient road map for you to do so.
I got the idea for the KGN application because I was spending quite a bit of time manually setting up
SPARQL queries for DBPedia and other public sources like WikiData, and I wanted to experiment
with partially automating this exploration process.
Book Wrap-up
I love programming in Lisp languages but I often need to use Python libraries for Deep Learning
and NLP. The Hy language is a good fit for me, it is simple to install along with the Python libraries
that I use for my work and it is a fun language to write code in. Most importantly, Hy fits well with
the type of iterative bottom-up REPL-based development that I prefer.
I hope that you enjoyed this short book and that at least a few things that you have learned here
will both help you in your work and give you ideas for new personal projects.
Best regards,
Mark Watson
February 15, 2020