Virantha Namal Ekanayakehttps://virantha.com/2021-04-01T10:33:00-04:00Client-specific DNS and DHCP settings with Dnsmasq and Edgeos / Edgerouter2021-04-01T10:33:00-04:00viranthatag:https://virantha.com,2021-04-01:2021/04/01/edgeos-dnsmasq-client-specific-dns-dhcp/<html><body><p>I have my Edgerouter configured to hand out static leases for all my network clients using
Dnsmasq, which also automatically puts an entry into the DNS with the client name. The
Edgeos configuration for this is relatively straightforward:</p>
<div class="highlight"><pre>dhcp-server {
...
shared-network-name LAN1 {
authoritative enable
subnet 192.168.9.0/24 {
...
static-mapping my-device {
ip-address 192.168.9.33
mac-address XX:XX:XX:XX:XX:XX
}
use-dnsmasq enable
}
}
}
</pre></div>
<p>Now, recently, I needed to get some of my devices (the children's tablets, for example) to
use Pi-Hole for their DNS lookups, while leaving the rest of my clients untouched. In effect,
I needed Dnsmasq to hand out a different DNS when the DHCP request comes in from a specific
client. There's
relatively scant information out there on how to get his to work with the static-mapping
commands, so this may end up helping a few you in similar situations.</p>
<p>As far as I can tell, Edgeos doesn't currenlty support this directly with its own commands in the
<tt class="docutils literal"><span class="pre">dhcp-server</span></tt> section.
What you need to do is put some manual options in the <tt class="docutils literal">dns</tt> section that will be passed on
directly to the Dnsmasq conf file, while not conflicting with the Edgeos generated configuration
options.</p>
<p>So, let's assume that I currently have the following static mapping, and I want to assign
these clients below a separate DNS IP.</p>
<div class="highlight"><pre>dhcp-server {
...
shared-network-name LAN1 {
authoritative enable
subnet 192.168.9.0/24 {
...
static-mapping child-device-1 {
ip-address 192.168.9.10
mac-address XX:XX:XX:XX:XX:XX
}
static-mapping child-device-2 {
ip-address 192.168.9.11
mac-address YY:YY:YY:YY:YY:YY
}
use-dnsmasq enable
}
}
}
</pre></div>
<p>What you need to do is <strong>remove</strong> the static-mapping for these devices from the <tt class="docutils literal">services <span class="pre">dhcp-server</span></tt>
section, and instead put it in manually in <tt class="docutils literal">services dns</tt> as follows:</p>
<div class="highlight"><pre>dns {
forwarding {
...
options dhcp-host=XX:XX:XX:XX:XX:XX,set:LAN1,set:Child,192.168.9.10,child-device-1
options dhcp-host=YY:YY:YY:YY:YY:YY,set:LAN1,set:Child,192.168.9.11,child-device-2
options dhcp-option=tag:Child,option:dns-server,192.168.9.222
...
}
}
</pre></div>
<p>The <tt class="docutils literal"><span class="pre">dhcp-host</span></tt> option assigns the static IP address mapping and host name to the device,
and also adds it to two <strong>tags</strong> via the <em>set</em> directive. The first tag is <em>LAN1</em>, which Edgeos usually sets
for you based on your <tt class="docutils literal"><span class="pre">shared-network-name</span></tt>, but in this case, because we removed the
<tt class="docutils literal"><span class="pre">static-mapping</span></tt> entry for these clients, we need to add it back in manually. The second tag
is the one you want to assign these specific devices to; in this example, I'm tagging these
as my <em>Child</em> devices.</p>
<p>Then, the <tt class="docutils literal"><span class="pre">dhcp-option</span></tt>, for whatever devices match a <em>Child</em> tag, passes
on the <tt class="docutils literal"><span class="pre">dns-server</span></tt> option via DHCP (in this case, child-device-1 and child-device-2 will get
a DNS of 192.168.9.222 sent via DHCP when they get their IP address lease). Obviously for this
to work on the client end, the client devices need to automatically set the DNS via DHCP (and
not be set manually).</p>
<p>You can use this type of tagging to set any other option you want as well. For example,
you could hand out PXE boot information to only certain clients, for example, or have
different clients get different boot images.</p>
</body></html>LEGO®'s new SPIKE Prime BluetoothLE Controlled Robotics Kit2019-04-04T11:09:00-04:00viranthatag:https://virantha.com,2019-04-04:2019/04/04/lego-spike-prime-compared/<html><body><div class="note">
<p class="first admonition-title">Note</p>
<p class="last">Updated 4/9/19 with some new information regarding the sensors from newly available datasheets.</p>
</div>
<p>Lego® just announced their brand-new (as of April 2019) robotics education kit,
the <a class="reference external" href="https://education.lego.com/en-us/meetspikeprime">SPIKE Prime</a>.
It's available for pre-order for US $329.95 from the Lego® Education
Store as part number <a class="reference external" href="https://education.lego.com/en-us/products/lego-education-spike-prime-set/45678">45678</a>, and is expected to ship in August.</p>
<div class="figure align-right">
<img alt=" spike prime" src="/images/2019/spike/spike_overview.png" style="width: 400px;"/>
</div>
<div class="section" id="overview">
<h2><a class="toc-backref" href="#id2">1 Overview</a></h2>
<p>The SPIKE Prime system appears to be using the same LPF2 (Lego® Power Functions 2.0) BluetoothLE
communication system that they introduced with their PoweredUp and Boost robotics gear in the last
few years. This new set introduces a new hub that now has 6 general purpose ports (up from
2 in their previous LPF2 systems), and an expanded set of peripherals such as sensors and motors. I'm hoping
to update my Python library <a class="reference external" href="https://virantha.com/2019/03/30/bricknil/">BrickNil</a> to support this once
it comes out.</p>
<p>The hub now appears to have a 5x5 programable segment LED display, a built-in rechargeable battery, as well as a built-in 6-axis accelerometer and
speaker, and can be <a class="reference external" href="https://education.lego.com/en-us/products/lego-technic-large-hub/45601">ordered separately</a> for relatively pricey $248.</p>
<div class="figure align-left">
<img alt=" spike hub" src="/images/2019/spike/spike_hub.png" style="width: 150px;"/>
</div>
<p>Even more exciting is that it says it's running an embedded MicroPython OS, so hopefully they will release the Python API at some point for
downloadable programs. The processor inside is, however, only specc'ed at 100MHz.</p>
<div class="figure align-center">
<img alt=" spike prime" src="/images/2019/spike/spike_spec_hub_python.png" style="width: 600px;"/>
</div>
<p>The peripherals appear to include two small motors, one large motor, light
sensor, distance sensor, and a touch sensor. Only
the larger motor, which LEGO is calling <a class="reference external" href="https://education.lego.com/en-us/products/lego-technic-large-angular-motor/45602">Technic Large Angular Motor</a>
is currently available separately for pre-order for $35. According to some details from a LEGO developer,
all the SPIKE motors support angular positioning capability with an absolute zero reference. Previous motors with angular
movement would reset their zero reference to whatever position they were in at power-up, so this new feature
should be quite interesting for resetting your system to a known state.</p>
<p>According to the datasheet on the distance sensor, there is easy access to a break-out pin header that should let
anyone design and plug in their own hardware to expand this system with new
sensors/actuators. I wonder if any third-party commercial devices will start
becoming available (assuming Lego® publishes the UART and physical
specification on this connector). Regardless, this should be quite easy for
hobbyists to take advantage of.</p>
<div class="figure align-center">
<img alt=" spike hub" src="/images/2019/spike/spike_breakout.png" style="width: 600px;"/>
</div>
</div>
<div class="section" id="comparison-with-boost-and-poweredup-sets">
<h2><a class="toc-backref" href="#id3">2 Comparison with Boost and PoweredUp Sets</a></h2>
<p>Let's take a look and compare this with the two older LPF2 gear. The included peripherals between
PoweredUp and Boost are interchangeable (also to a lesser extent with the Wedo educational set),
so the hope is that SPIKE Prime will be compatible with the older gear as well, and possibly the new
peripherals will plug-and-play with the older hubs.</p>
<table border="1" class="docutils pretty compact cell-border stripe" id="comparison">
<caption>Lego Power Functions 2.0 Sets Compared</caption>
<colgroup>
<col width="25%"></col>
<col width="25%"></col>
<col width="25%"></col>
<col width="25%"></col>
</colgroup>
<thead valign="bottom">
<tr><th class="head"> </th>
<th class="head">PoweredUp <a class="amazon" href="http://www.amazon.com/gp/product/B07CC37F63?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">Train</a></th>
<th class="head"><a class="amazon" href="http://www.amazon.com/gp/product/B06Y6JCTKH?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">Boost</a></th>
<th class="head">SPIKE <a class="reference external" href="https://education.lego.com/en-us/products/lego-education-spike-prime-set/45678">45678</a></th>
</tr>
</thead>
<tbody valign="top">
<tr><td>Price (USD)</td>
<td>$160+</td>
<td>$160</td>
<td>$330</td>
</tr>
<tr><td>BluetoothLE</td>
<td>✓</td>
<td>✓</td>
<td>✓</td>
</tr>
<tr><td>Remote</td>
<td>✓</td>
<td> </td>
<td> </td>
</tr>
<tr><td>Ports</td>
<td>2</td>
<td>2</td>
<td>6</td>
</tr>
<tr><td>Internal tilt/accelerometer</td>
<td> </td>
<td>✓</td>
<td>✓</td>
</tr>
<tr><td>Train motor</td>
<td>1</td>
<td> </td>
<td> </td>
</tr>
<tr><td>Internal motor</td>
<td> </td>
<td>2</td>
<td> </td>
</tr>
<tr><td>External motor (medium)</td>
<td> </td>
<td>1</td>
<td>2</td>
</tr>
<tr><td>External motor (large)</td>
<td> </td>
<td> </td>
<td>1</td>
</tr>
<tr><td>Color LED</td>
<td>✓</td>
<td>✓</td>
<td>✓</td>
</tr>
<tr><td>Matrix display</td>
<td> </td>
<td> </td>
<td>✓ 5x5</td>
</tr>
<tr><td>Speaker</td>
<td> </td>
<td> </td>
<td>✓</td>
</tr>
<tr><td>Light/Color sensor</td>
<td> </td>
<td>✓</td>
<td>✓</td>
</tr>
<tr><td>Ultrasonic distance sensor</td>
<td> </td>
<td> </td>
<td>✓</td>
</tr>
<tr><td>Touch sensor</td>
<td> </td>
<td> </td>
<td>✓</td>
</tr>
<tr><td>Power</td>
<td>6 AAA</td>
<td>6 AAA</td>
<td>Rechargeable</td>
</tr>
</tbody>
</table>
<p>It does seem that the new SPIKE Prime hub opens up room for significantly increasing interaction with the
environment with its support for a larger number of plug-in peripherals. I'm looking forward to getting
my hands on one when they start shipping in the fall.</p>
</div>
<div class="section" id="external-community-links">
<h2><a class="toc-backref" href="#id4">3 External community links</a></h2>
<p>I'm going to close with a few helpful community links:</p>
<ul>
<li><p class="first">There is an open <a class="reference external" href="https://www.facebook.com/groups/SPIKEcommunity/">Facebook page</a> with some of the developers of the SPIKE prime system that has some great
information (and all credit to the developer there that posted the technical specification datasheets posted above).</p>
</li>
<li><p class="first">And here is a nice video overview that someone did on the internals of this set:</p>
<div style="position: relative; height: 0; overflow: hidden; max-width: 100%; height: auto;">
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/NTg5juB9xhA" width="560"></iframe>
</div></li>
</ul>
</div>
<script charset="utf8" src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script><script charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script><script>
$(document).ready(function() {
$('#comparison').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script></body></html>Setting up a Raspberry Pi with Python 3.7+ using Ansible2019-03-30T21:28:00-04:00viranthatag:https://virantha.com,2019-03-30:2019/03/30/raspberry-pi-setup-with-ansible-python-37/<html><body><p>Here's a set of <a class="reference external" href="https://www.ansible.com/resources/get-started">Ansible</a> playbook files to do an unattended compile of Python 3.7+ with SSL on a
headless wifi-connected Raspberry Pi. A recent project of mine (<a class="reference external" href="https://virantha.github.io/bricknil">bricknil</a>) uses 3.7, and
it was a little frustrating to get the default 3.5 upgraded on my Raspbian image --
I had to compile Python from source with the newer SSL libraries (so that pip works), and get around some uuid
compilation issues that took a fair bit of web research to figure out.</p>
<p>Everything is run from the local machine
talking to the Pi over ssh, and you just need Ansible installed on your local machine (which can be
as easy as <cite>pip install ansible</cite> depending on your OS). Hope this helps save some time for anyone else needing
get modern Python installed on their RPi!</p>
<div class="section" id="steps">
<h2><a class="toc-backref" href="#id1">1 Steps</a></h2>
<div class="section" id="download-a-raspbian-image-and-burn-it-to-a-sd-card">
<h3><a class="toc-backref" href="#id2">1.1 Download a Raspbian image and burn it to a SD card</a></h3>
<ol class="arabic">
<li><p class="first">Download the <a class="reference external" href="https://www.raspberrypi.org/downloads/raspbian/">Raspbian Stretch Lite</a> image</p>
</li>
<li><p class="first">Burn it to a suitable SD card. On OS X, <a class="reference external" href="https://www.balena.io/etcher/">Balena Etcher</a> is a good tool to do this.</p>
</li>
<li><p class="first">On the newly imaged SD card's <cite>/boot</cite> folder:</p>
<ol class="arabic">
<li><p class="first">Make an empty file called <cite>ssh</cite> to enable SSH access.</p>
</li>
<li><p class="first">Make <cite>wpa_supplicant.conf</cite> file to supply your wifi credentials:</p>
<pre class="literal-block">
country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="NETWORK_NAME
psk="WIFI PASSWORD"
}
</pre>
</li>
</ol>
</li>
<li><p class="first">Now, you should be able to use this SD card to boot your raspberry pi. After giving it a few minutes to boot, you can login in by doing the following on your local machine:</p>
<pre class="literal-block">
ssh pi@raspberrypi.local
<enter 'raspberry' as the default password>
</pre>
</li>
</ol>
</div>
<div class="section" id="install-ansbile-on-your-local-machine">
<h3><a class="toc-backref" href="#id3">1.2 Install Ansbile on your local machine</a></h3>
<p>You can just type <cite>pip install ansible</cite></p>
</div>
<div class="section" id="change-the-default-password-on-the-pi">
<h3><a class="toc-backref" href="#id4">1.3 Change the default password on the Pi</a></h3>
<ol class="arabic">
<li><p class="first">You may need to modify the <cite>hosts</cite> file in <cite>pi_setup</cite> directory to put in your own IP address for the Pi.</p>
<div class="gist">
<script src='https://gist.github.com/1b1352c0290b39718596d2bb6891b8bb.js?file=hosts'></script>
<noscript>
<pre><code>[pi]
raspberrypi.local ansible_user=pi
</code></pre>
</noscript>
</div>
</li>
<li><p class="first">You will need to install the python library <cite>passlib</cite> by running <cite>pip install passlib</cite> on your local machine (not the Pi). This is needed to hash the new password.</p>
</li>
<li><p class="first">Use the <cite>change_password.yml</cite> playbook to test your ansible install and change the default password on the pi (use <cite>raspberry</cite> for the ssh password when prompted, and then enter your new password):</p>
<pre class="literal-block">
ansible-playbook -i hosts change_password.yml --ask-pass
</pre>
<p>The playbook being run is below, and just prompts the user for a new password before making a copy of the
shadow file and updating the password:</p>
</li>
</ol>
<div class="gist">
<script src='https://gist.github.com/1b1352c0290b39718596d2bb6891b8bb.js?file=change_password.yml'></script>
<noscript>
<pre><code>- hosts: all
remote_user: pi
become_method: sudo
vars_prompt:
- name: new_password
prompt: "Enter the password you would like to use for the user pi"
confirm: yes
tasks:
- name: backup shadow file
copy:
remote_src: yes
src: /etc/shadow
dest: /tmp/shadow
become: yes
- name: generate hash pass
delegate_to: localhost
command: python -c "from passlib.hash import md5_crypt; import getpass; print (md5_crypt.hash('{{new_password}}'))"
register: hash
- debug:
var: hash.stdout
- name: update password
user:
name: pi
password: '{{hash.stdout}}'
become: yes
</code></pre>
</noscript>
</div>
</div>
<div class="section" id="run-the-setup-playbook">
<h3><a class="toc-backref" href="#id5">1.4 Run the setup playbook</a></h3>
<ol class="arabic">
<li><p class="first">The final step is to just run the the install playbook. This will install all the packages, download and compile OpenSSL, followed by Python 3.7, and set up
a virtualenv. The actual playbook is in <cite>tasks.yml</cite>, shown below and included in the source, and the command to
execute it on your local machine is:</p>
<pre class="literal-block">
ansible-playbook -i hosts tasks.yml --ask-pass
</pre>
<p>The install will take around two hours, so please be patient. Most of the time is spent compiling and installing Python from source. Here's the playbook being executed:</p>
<div class="gist">
<script src='https://gist.github.com/1b1352c0290b39718596d2bb6891b8bb.js?file=tasks.yml'></script>
<noscript>
<pre><code>- hosts: all
remote_user: pi
become_method: sudo
vars:
PYTHON: "3.7.2"
OPEN_SSL: openssl-1.0.2g
build_path: "{{ansible_env.HOME}}/build"
HOME: "{{ansible_env.HOME}}"
tasks:
- name: Update APT package cache
apt: update_cache=yes
become: yes
- name: Update packages
apt: upgrade=dist
become: yes
- name: Install build-essential
package: name="build-essential" state=present
become: yes
- name: Install build packages
apt:
state: present
name:
- tk-dev
- libncurses5-dev
- libncursesw5-dev
- libreadline6-dev
- libdb5.3-dev
- libgdbm-dev
- libsqlite3-dev
- libssl-dev
- libbz2-dev
- libexpat1-dev
- liblzma-dev
- zlib1g-dev
- libffi-dev
- uuid-dev
- vim
- git
- python-pip
- screen
become: yes
- name: Ensure build directory exists
file:
path: "{{ build_path }}"
state: directory
recurse: yes
- name: Download openssl
get_url:
url: "https://www.openssl.org/source/{{ OPEN_SSL }}.tar.gz"
dest: "{{ build_path }}/{{ OPEN_SSL }}.tar.gz"
mode: "755"
register: openssl_download
- name: Untar openssl
unarchive:
remote_src: yes
src: "{{ build_path }}/{{ OPEN_SSL }}.tar.gz"
dest: "{{ build_path }}"
when: openssl_download.changed
- name: configure openssl
command: ./config shared --prefix=/usr/local
args:
chdir: "{{ build_path }}/{{ OPEN_SSL }}"
when: openssl_download.changed
- name: make openssl
command: make -j 4
args:
chdir: "{{ build_path }}/{{ OPEN_SSL }}"
register: make_openssl
when: openssl_download.changed
- name: install openssl
command: make install
args:
chdir: "{{ build_path }}/{{ OPEN_SSL }}"
become: yes
when: make_openssl.changed
- name: set /etc/ld.so.conf.d to add /usr/local/
lineinfile:
path: /etc/ld.so.conf.d/openssl.conf
create: yes
line: "/usr/local/lib"
become: yes
- name: run ldconfig to update dynamic link lib path to use /usr/local/lib
command: ldconfig
become: yes
- name: Download python 3.7
get_url:
url: "https://www.python.org/ftp/python/{{PYTHON}}/Python-{{PYTHON}}.tgz"
dest: "{{ build_path }}/Python-{{ PYTHON }}.tgz"
mode: "755"
register: python_download
- name: Untar python
unarchive:
remote_src: yes
src: "{{ build_path }}/Python-{{ PYTHON }}.tgz"
dest: "{{ build_path }}"
when: python_download.changed
- name: Uncomment SSL
blockinfile:
path: "{{ build_path}}/Python-{{ PYTHON }}/Modules/Setup.dist"
insertbefore: "#SSL=/usr/local/ssl"
block: |
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
- name: configure python
command: ./configure --enable-optimizations --with-openssl=/usr/local --prefix=/usr/local
args:
chdir: "{{ build_path }}/Python-{{ PYTHON }}"
when: python_download.changed
register: python_configure
- name: make python
shell: make -j 4'
args:
chdir: "{{ build_path }}/Python-{{ PYTHON }}"
register: make_python
when: python_configure.changed
- name: install python
shell: make altinstall'
args:
chdir: "{{ build_path }}/Python-{{ PYTHON }}"
become: yes
when: python_configure.changed
- name: Setup .bashrc
blockinfile:
path: "{{ HOME }}/.bashrc"
block: |
export VIRTUALENVWRAPPER_PYTHON="/usr/local/bin/python3.7"
source /usr/local/bin/virtualenvwrapper.sh
- name: Install pip virtualenv
pip:
executable: /usr/local/bin/pip3.7
name:
- virtualenv
- virtualenvwrapper
become: yes
- name: pip install bricknil
pip:
name:
- bricknil
virtualenv: "{{ HOME }}/.virtualenvs/bricknil"
virtualenv_python: python3.7
- name: git clone bricknil
git:
repo: 'https://github.com/virantha/bricknil.git'
dest: '{{ HOME }}/bricknil'
clone: yes</code></pre>
</noscript>
</div>
<p>At the end, there are a couple of steps where I install a sample package and clone a github repo (which in
this case is my <a class="reference external" href="https://virantha.github.io/bricknil">bricknil</a> project for controlling Lego trains and robots. Python 3.7 will be available as
<cite>/usr/local/bin/python3.7</cite> and virtualenvwrapper will use it as the default python.</p>
</li>
<li><p class="first">And that's it! Your raspberry pi should be ready to go as a networked appliance with an up-to-date Python.</p>
</li>
</ol>
</div>
</div>
</body></html>Part 3 - Building a Fast Event-Driven Simulator for Concurrent Systems of Hardware Processes Using Curio and Python 3.62017-09-28T11:20:00-04:00viranthatag:https://virantha.com,2017-09-28:2017/09/28/hardware-simulation-using-curio-3/<html><body><p>This is the third in my series on using <a class="reference external" href="http://curio.readthedocs.io/en/latest/">Curio</a> to build an event-driven
simulator for hardware processes. The first two parts can be found here:
<a class="reference external" href="https://virantha.com/2017/09/22/hardware-simulation-using-curio/">Part 1</a>, <a class="reference external" href="https://virantha.com/2017/09/27/hardware-simulation-using-curio-2/">Part 2</a>. We're going to use the simulator framework to implement
a <em>distributed</em> and <em>concurrent</em> hardware palindrome checker in less than 50
lines of code.</p>
<div class="section" id="palindrome-checker">
<h2><a class="toc-backref" href="#id1">1 Palindrome checker</a></h2>
<p>Let's define what the checker must do as follows:</p>
<ul class="simple">
<li>The input string is a sequence of characters that arrives one by one.</li>
<li>With each character's arrival, the palindrome checker must output a boolean representing whether the sequence
seen so far is a palindrome or not.</li>
<li>As an example, if the sequence "abbab" is sent in (left-most character
first), then the checker should output True, False, False, True, False.</li>
<li>For generality and scalability, the checker must be composed of N processes
(where N is the length of the input sequence) that each use O(1)
storage(memory).</li>
</ul>
</div>
<div class="section" id="a-solution">
<h2><a class="toc-backref" href="#id2">2 A solution</a></h2>
<p>One way to approach such a problem is to think about it recursively. Let's say
that we're about to receive the <em>k</em> th character, and we know something about
the <em>k-1</em> sequence containing a palindrome or not. Can we immediately
determine whether the new sequence with the <em>k</em> th character is a palindrome?</p>
<p>It turns out that we can, <em>if</em> we keep track of what the first character
we saw is. The new sequence formed by adding the <em>k</em> th character is a
palindrome if and only if this new character is identical to the first
character, and the remaining sequence book-ended by these is also a palindrome.</p>
<div class="figure align-center" style="width: 854px; height: auto; max-width: 100%;">
<img alt="Palindrome recursive formulation" height="320" src="/images/2017/chp/palindrome.png" style="width: 854px; height: auto; max-width: 100%;" width="854"/>
</div>
<p>Here's some pythonic pseudo-code that implements such a recursive solution:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">is_palindrome</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o"><=</span><span class="mi">1</span><span class="p">:</span> <span class="c"># A zero or one character string is a palindrome</span>
<span class="k">return</span> <span class="bp">True</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">first</span> <span class="o">=</span> <span class="n">s</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">last</span> <span class="o">=</span> <span class="n">s</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">return</span> <span class="p">(</span><span class="n">first</span><span class="o">==</span><span class="n">last</span><span class="p">)</span> <span class="ow">and</span> <span class="n">is_palindrome</span><span class="p">(</span><span class="n">s</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
</pre></div>
</div>
<div class="section" id="a-concurrent-process">
<h2><a class="toc-backref" href="#id3">3 A concurrent process</a></h2>
<p>Now, how can we implement such a solution using a set of processes that communicate
with each other via channels? In essence, all we do is unroll all the calls
in the recursive construction above into a set of sequentially connected processes!
(you can think of it as expanding out the call stack).</p>
<p>Let's start with a simple process I will call <em>Pal</em>, shown below. It has, on
the left side, an input port <strong>C</strong> on which it receives a sequence of
characters, and an output port <strong>Ans</strong> which sends out whether the last
received character caused a palindrome to be formed or not. It then passes the
character received out on <strong>Cr</strong> and receives an answer on <strong>Ansr</strong> which is True if the
sequence to the right is a palindrome or not.</p>
<div class="figure align-center" style="width: 438px; height: auto; max-width: 100%;">
<img alt="Palindrome recursive formulation" height="159" src="/images/2017/chp/PAL1.png" style="width: 438px; height: auto; max-width: 100%;" width="438"/>
</div>
<p>This process will implement the pseudo-code above, like so:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">curio</span> <span class="kn">import</span> <span class="n">run</span>
<span class="kn">from</span> <span class="nn">chp</span> <span class="kn">import</span> <span class="o">*</span>
<span class="k">class</span> <span class="nc">Pal</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="n">C</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="n">Ans</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="n">Cr</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="n">Ansr</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c"># Do the first char (so we store it)</span>
<span class="n">first</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">C</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Ans</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">is_palindrome</span><span class="p">)</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">C</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span><span class="o">==</span><span class="n">first</span> <span class="ow">and</span> <span class="n">is_palindrome</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">advance_local_time</span><span class="p">()</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Ans</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">is_palindrome</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Cr</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Ansr</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
</pre></div>
<p>First, I import the framework defined in the previous article (I'm calling it <tt class="docutils literal">chp.py</tt>,
and you can download it in the gist at the end of this article)</p>
<p>Then, I define the <em>Pal</em> process as a subclass of a <em>Process</em> and specify its input
and output ports to match our figure above. Then, I implement the pseudo-code in
the <tt class="docutils literal">exec</tt> coroutine method. The code outside the <tt class="docutils literal">while True</tt> statement implements
the base case where the length is 1; each time a character is received, the answer
is True. Then, for each new character received, I can compute whether the
new sequence is a palindrome by comparing the first value seen to the current value,
and <em>AND</em> ing with whether the previous sequence was a palindrome. The result is
sent out on <em>Ans</em>.</p>
<p>Then, the code implements the recursive call by propagating the new character to the
remainder of the sequence on the right, and receiving its result on <em>Ansr</em>.</p>
</div>
<div class="section" id="the-environment">
<h2><a class="toc-backref" href="#id4">4 The environment</a></h2>
<p>At this point, we need to implement a Process to supply the characters in the
sequence to be checked. Here's the <em>Env</em> (environment) process:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Env</span><span class="p">(</span><span class="n">Producer</span><span class="p">):</span>
<span class="n">R</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="n">A</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">string</span> <span class="o">=</span> <span class="n">string</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">string</span><span class="p">)):</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">string</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">A</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">'{self.string[:i+1]} - {is_palindrome}'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">advance_local_time</span><span class="p">()</span>
</pre></div>
<p>Remember to subclass from <em>Producer</em> if you have a process like this that
generates values unconditionally. You can initialize this process with the
string to be checked. In the <tt class="docutils literal">exec</tt> coroutine, the process repeatedly sends
out each character of the string, leftmost first. It receives a result (True
if a palindrome) and then prints a message.</p>
</div>
<div class="section" id="instantiate-the-system">
<h2><a class="toc-backref" href="#id5">5 Instantiate the system</a></h2>
<p>Now, let's instantiate the environment, and set of processes for a sample string.
Here, we're going to check the string 'A man a plan a cat a ham a yak a yam a hat a canal Panama' (minus the spaces, of course!)</p>
<div class="highlight"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">system</span><span class="p">():</span>
<span class="n">string</span> <span class="o">=</span> <span class="s">'amanaplanacatahamayakayamahatacanalpanama'</span>
<span class="n">env</span> <span class="o">=</span> <span class="n">Env</span><span class="p">(</span><span class="s">'env'</span><span class="p">,</span> <span class="n">string</span><span class="p">)</span>
<span class="n">pals</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">string</span><span class="p">)):</span>
<span class="n">pals</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Pal</span><span class="p">(</span><span class="n">f</span><span class="s">'pal[{i}]'</span><span class="p">))</span>
<span class="n">connect</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">R</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">C</span><span class="p">)</span>
<span class="n">connect</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">A</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">Ans</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">string</span><span class="p">)</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
<span class="n">connect</span><span class="p">(</span><span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">Cr</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">C</span><span class="p">)</span>
<span class="n">connect</span><span class="p">(</span><span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">Ans</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">Ansr</span><span class="p">)</span>
<span class="n">await</span> <span class="n">run_all</span><span class="p">()</span>
<span class="n">run</span><span class="p">(</span><span class="n">system</span><span class="p">(),</span> <span class="n">with_monitor</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
<p>We instantiate one <em>Env</em> process, and then one <em>Pal</em> process for each character in the
input string. Then we connect up the processes, and call <tt class="docutils literal">run_all</tt>.</p>
</div>
<div class="section" id="let-s-run-it">
<h2><a class="toc-backref" href="#id6">6 Let's run it!</a></h2>
<p>Here's the complete code that we've built so far on top of our framework (only 44 lines of code!):</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">curio</span> <span class="kn">import</span> <span class="n">run</span>
<span class="kn">from</span> <span class="nn">chp</span> <span class="kn">import</span> <span class="o">*</span>
<span class="k">class</span> <span class="nc">Pal</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="n">C</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="n">Ans</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="n">Cr</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="n">Ansr</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c"># Do the first char (so we store it)</span>
<span class="n">first</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">C</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Ans</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">is_palindrome</span><span class="p">)</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">C</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span><span class="o">==</span><span class="n">first</span> <span class="ow">and</span> <span class="n">is_palindrome</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">advance_local_time</span><span class="p">()</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Ans</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">is_palindrome</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Cr</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">Ansr</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="k">class</span> <span class="nc">Env</span><span class="p">(</span><span class="n">Producer</span><span class="p">):</span>
<span class="n">R</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="n">A</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">string</span> <span class="o">=</span> <span class="n">string</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">string</span><span class="p">)):</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">string</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="n">is_palindrome</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">A</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">'{self.string[:i+1]} - {is_palindrome}'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">advance_local_time</span><span class="p">()</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">system</span><span class="p">():</span>
<span class="n">string</span> <span class="o">=</span> <span class="s">'amanaplanacatahamayakayamahatacanalpanama'</span>
<span class="n">env</span> <span class="o">=</span> <span class="n">Env</span><span class="p">(</span><span class="s">'env'</span><span class="p">,</span> <span class="n">string</span><span class="p">)</span>
<span class="n">pals</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">string</span><span class="p">)):</span>
<span class="n">pals</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Pal</span><span class="p">(</span><span class="n">f</span><span class="s">'pal[{i}]'</span><span class="p">))</span>
<span class="n">connect</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">R</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">C</span><span class="p">)</span>
<span class="n">connect</span><span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">A</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">Ans</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">string</span><span class="p">)</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
<span class="n">connect</span><span class="p">(</span><span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">Cr</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">C</span><span class="p">)</span>
<span class="n">connect</span><span class="p">(</span><span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">Ans</span><span class="p">,</span> <span class="n">pals</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">Ansr</span><span class="p">)</span>
<span class="n">await</span> <span class="n">run_all</span><span class="p">()</span>
<span class="n">run</span><span class="p">(</span><span class="n">system</span><span class="p">(),</span> <span class="n">with_monitor</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
<p>Just run <tt class="docutils literal">python palindrome.py</tt> to execute it, which will give you the following output:</p>
<div class="highlight"><pre>T:0: env.0 - a - True
T:200: env.0 - am - False
T:400: env.0 - ama - True
T:600: env.0 - aman - False
T:800: env.0 - amana - False
T:1000: env.0 - amanap - False
T:1200: env.0 - amanapl - False
T:1400: env.0 - amanapla - False
T:1600: env.0 - amanaplan - False
T:1800: env.0 - amanaplana - False
T:2000: env.0 - amanaplanac - False
T:2200: env.0 - amanaplanaca - False
T:2400: env.0 - amanaplanacat - False
T:2600: env.0 - amanaplanacata - False
T:2800: env.0 - amanaplanacatah - False
T:3000: env.0 - amanaplanacataha - False
T:3200: env.0 - amanaplanacataham - False
T:3400: env.0 - amanaplanacatahama - False
T:3600: env.0 - amanaplanacatahamay - False
T:3800: env.0 - amanaplanacatahamaya - False
T:4000: env.0 - amanaplanacatahamayak - False
T:4200: env.0 - amanaplanacatahamayaka - False
T:4400: env.0 - amanaplanacatahamayakay - False
T:4600: env.0 - amanaplanacatahamayakaya - False
T:4800: env.0 - amanaplanacatahamayakayam - False
T:5000: env.0 - amanaplanacatahamayakayama - False
T:5200: env.0 - amanaplanacatahamayakayamah - False
T:5400: env.0 - amanaplanacatahamayakayamaha - False
T:5600: env.0 - amanaplanacatahamayakayamahat - False
T:5800: env.0 - amanaplanacatahamayakayamahata - False
T:6000: env.0 - amanaplanacatahamayakayamahatac - False
T:6200: env.0 - amanaplanacatahamayakayamahataca - False
T:6400: env.0 - amanaplanacatahamayakayamahatacan - False
T:6600: env.0 - amanaplanacatahamayakayamahatacana - False
T:6800: env.0 - amanaplanacatahamayakayamahatacanal - False
T:7000: env.0 - amanaplanacatahamayakayamahatacanalp - False
T:7200: env.0 - amanaplanacatahamayakayamahatacanalpa - False
T:7400: env.0 - amanaplanacatahamayakayamahatacanalpan - False
T:7600: env.0 - amanaplanacatahamayakayamahatacanalpana - False
T:7800: env.0 - amanaplanacatahamayakayamahatacanalpanam - False
T:8000: env.0 - amanaplanacatahamayakayamahatacanalpanama - True
</pre></div>
<p>And success! It's remarkable how little code it took to accomplish this kind of simulation.</p>
<p>You can find the complete runnable code including the framework (Python 3.6 and up!) in this <a class="reference external" href="https://gist.github.com/4c9b770244de999117f42b6002f08178">gist</a>.</p>
</div>
<div class="section" id="summary">
<h2><a class="toc-backref" href="#id7">7 Summary</a></h2>
<p>In this article, I've gone over an example of how to simulate a concurrent hardware system
based on message passing between sequential processes using the framework we built
in the first two articles (<a class="reference external" href="https://virantha.com/2017/09/22/hardware-simulation-using-curio/">Part 1</a>, <a class="reference external" href="https://virantha.com/2017/09/27/hardware-simulation-using-curio-2/">Part 2</a>) of this series. As you can see, it's really quite simple
to set up and execute new processes that can implement non-trivial systems very quickly.
Perhaps at some point I'll get around to porting the sensor network processor I designed in
grad school into this framework, but as you can see, there really is no limit to
the amount of complexity you could try to model with this type of simulator.</p>
</div>
</body></html>Part 2 - Building a Fast Event-Driven Simulator for Concurrent Systems of Hardware Processes Using Curio and Python 3.62017-09-27T11:20:00-04:00viranthatag:https://virantha.com,2017-09-27:2017/09/27/hardware-simulation-using-curio-2/<html><body><p>This is the second in my series on using <a class="reference external" href="http://curio.readthedocs.io/en/latest/">Curio</a> to build an event-driven simulator for
hardware processes. (see <a class="reference external" href="https://virantha.com/2017/09/22/hardware-simulation-using-curio/">Part 1</a> for the first article)</p>
<div class="section" id="enhancing-the-simulator-framework">
<h2><a class="toc-backref" href="#id1">1 Enhancing the Simulator Framework</a></h2>
<p>Now, we're going to make this framework a little bit more robust and less verbose
when defining the system. In order to do this, I'm going to first introduce the
<em>Port</em> class that will be used to define how <em>Processes</em> are connected. Then
we'll write a utility function to start and end the system automatically, and
leverage type-annotations to make the code cleaner. Finally, we'll also
introduce a distributed timestamp system to keep track of simulator time.</p>
</div>
<div class="section" id="introducing-ports">
<h2><a class="toc-backref" href="#id2">2 Introducing Ports</a></h2>
<p>Let's add the following classes to our framework:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Port</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">chan</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">class</span> <span class="nc">InputPort</span><span class="p">(</span><span class="n">Port</span><span class="p">):</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">recv</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">chan</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="k">return</span> <span class="n">tok</span>
<span class="k">class</span> <span class="nc">OutputPort</span><span class="p">(</span><span class="n">Port</span><span class="p">):</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">send</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">val</span><span class="p">):</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">chan</span><span class="o">.</span><span class="n">send</span><span class="p">((</span><span class="n">val</span><span class="p">))</span>
</pre></div>
<p>An <em>OutputPort</em> is meant to connect to the sender side of a <em>Channel</em>, so it has a
<tt class="docutils literal">send</tt> method, while the <em>InputPort</em> connects to the receiver side with a <tt class="docutils literal">recv</tt>
method.</p>
<p>Both kinds of <em>Ports</em> contain a link to the
<em>Channel</em> to which it is connected to.</p>
<p>Now, let's introduce a utility function that given an <em>OutputPort</em> and an <em>InputPort</em>,
will instantiate a channel to be used to connect the two ports. This lets us
hide the concept of a <em>Channel</em> inside this utility <tt class="docutils literal">connect</tt> function and will
eliminate the need to manually instantiate it in our simulation like we did
previously.</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">''</span><span class="p">):</span>
<span class="c"># Connect ports together by instantiating a channel</span>
<span class="n">chan</span> <span class="o">=</span> <span class="n">Channel</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="c"># Check to make sure the ports have not been connected previously to other channels!</span>
<span class="k">assert</span> <span class="ow">not</span> <span class="n">a</span><span class="o">.</span><span class="n">chan</span><span class="p">,</span> <span class="n">f</span><span class="s">"Channel {a} has already been connected!"</span>
<span class="k">assert</span> <span class="ow">not</span> <span class="n">b</span><span class="o">.</span><span class="n">chan</span><span class="p">,</span> <span class="n">f</span><span class="s">"Channel {b} has already been connected!"</span>
<span class="c"># Check to make sure the two ports are of opposite type (input/output)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">InputPort</span><span class="p">):</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">OutputPort</span><span class="p">),</span> <span class="n">f</span><span class="s">"Channel {a} and {b} are both input ports!"</span>
<span class="c"># Store the ports this channel is connected to</span>
<span class="c"># b ---chan---> a</span>
<span class="n">chan</span><span class="o">.</span><span class="n">l</span> <span class="o">=</span> <span class="n">b</span>
<span class="n">chan</span><span class="o">.</span><span class="n">r</span> <span class="o">=</span> <span class="n">a</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">InputPort</span><span class="p">),</span> <span class="n">f</span><span class="s">"Channel {a} and {b} are both output ports!"</span>
<span class="c"># Store the ports this channel is connected to</span>
<span class="c"># a ---chan---> b</span>
<span class="n">chan</span><span class="o">.</span><span class="n">l</span> <span class="o">=</span> <span class="n">a</span>
<span class="n">chan</span><span class="o">.</span><span class="n">r</span> <span class="o">=</span> <span class="n">b</span>
<span class="c"># Now assign the channel to the two ports</span>
<span class="n">a</span><span class="o">.</span><span class="n">chan</span> <span class="o">=</span> <span class="n">chan</span>
<span class="n">b</span><span class="o">.</span><span class="n">chan</span> <span class="o">=</span> <span class="n">chan</span>
</pre></div>
<p>This allows us to add a variety of checks like making sure a <em>Port</em> is only connected once
to a <em>Channel</em> (recall that channels are point-to-point connections), and that
one is an input and the other is an output.</p>
<p>Next, let's modify our <em>Processes</em> to perform <tt class="docutils literal">send</tt> and <tt class="docutils literal">recv</tt> actions on
their internal ports instead of channels.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Source</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">srcval</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">val</span> <span class="o">=</span> <span class="n">srcval</span>
<span class="bp">self</span><span class="o">.</span><span class="n">length</span> <span class="o">=</span> <span class="n">length</span>
<span class="bp">self</span><span class="o">.</span><span class="n">R</span> <span class="o">=</span> <span class="n">OutputPort</span><span class="p">()</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">length</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {self.val}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">val</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sent {self.val}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="s">"terminated"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Sink</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">L</span> <span class="o">=</span> <span class="n">InputPort</span><span class="p">()</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">tok_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">L</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">tok_count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"received {tok}"</span><span class="p">)</span>
<span class="k">except</span> <span class="n">CancelledError</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"{tok_count} tokens received"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Buffer</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">L</span> <span class="o">=</span> <span class="n">InputPort</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">R</span> <span class="o">=</span> <span class="n">OutputPort</span><span class="p">()</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">L</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"received {tok}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {tok}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">tok</span><span class="p">)</span>
</pre></div>
<p>In each constructor now, we've defined member variables for the <em>Ports</em> that each
<em>Process</em> needs, and we've gotten rid of the <tt class="docutils literal">connect</tt> method and delegated that
to the new <tt class="docutils literal">connect</tt> function.</p>
<p>At this point, we can rewrite our <tt class="docutils literal">system</tt> and remove any instantiations of <em>Channels</em>,
which is pretty neat. And there's much more error checking going on to make sure we
don't try to connect mismatched <em>Port</em> types, for example.</p>
<div class="highlight"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">system</span><span class="p">():</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">10</span> <span class="c"># How many buffers in our linear pipeline</span>
<span class="c"># Instantiate the processes</span>
<span class="n">src</span> <span class="o">=</span> <span class="n">Source</span><span class="p">(</span><span class="s">'src1'</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="n">Buffer</span><span class="p">(</span><span class="n">f</span><span class="s">'buf[{i}]'</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">)]</span>
<span class="n">snk</span> <span class="o">=</span> <span class="n">Sink</span><span class="p">(</span><span class="s">'snk'</span><span class="p">)</span>
<span class="c"># Connect the processes with the channels</span>
<span class="n">connect</span><span class="p">(</span><span class="n">src</span><span class="o">.</span><span class="n">R</span><span class="p">,</span> <span class="n">buf</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">L</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">):</span>
<span class="n">connect</span><span class="p">(</span><span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">R</span><span class="p">,</span> <span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">L</span><span class="p">)</span>
<span class="n">connect</span><span class="p">(</span><span class="n">snk</span><span class="o">.</span><span class="n">L</span><span class="p">,</span> <span class="n">buf</span><span class="p">[</span><span class="n">N</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">R</span><span class="p">)</span>
<span class="c"># Start the processes</span>
<span class="n">p_src</span> <span class="o">=</span> <span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">src</span><span class="o">.</span><span class="k">exec</span><span class="p">())</span>
<span class="n">p_snk</span> <span class="o">=</span> <span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">snk</span><span class="o">.</span><span class="k">exec</span><span class="p">())</span>
<span class="n">p_buf</span> <span class="o">=</span> <span class="p">[</span><span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="k">exec</span><span class="p">())</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">)]</span>
<span class="c"># Wait for the source to finish sending all its values</span>
<span class="n">await</span> <span class="n">p_src</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="c"># Cancel the remaining processes</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">):</span>
<span class="n">await</span> <span class="n">p_buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
<span class="n">await</span> <span class="n">p_snk</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
</pre></div>
<p>Running this will yield the same output as our first simulator. You can find the
complete code file in this <a class="reference external" href="https://gist.github.com/93f052ea6f1eef5ddf483487edcf9289">port_gist</a>.</p>
</div>
<div class="section" id="simplifying-the-execution">
<h2><a class="toc-backref" href="#id3">3 Simplifying the execution</a></h2>
<p>One further step we can take now is to eliminate the need to manually <tt class="docutils literal">spawn</tt> and <tt class="docutils literal">join</tt>/<tt class="docutils literal">cancel</tt> the processes to run our system. Wouldn't it be nice if we could just
say something like <tt class="docutils literal">run_all</tt> which would automatically start the defined processes?</p>
<p>Well, let's do just that! The basic idea is to keep track of all the instantiated process
in the <em>Process</em> base class. However, in order to <tt class="docutils literal">join</tt> on the correct type of <em>Process</em> (i.e. only Processes that generate values), we'll need to distinguish two kinds of
processes, <em>Producers</em> and others (non-producers). We'll make any of the former inherit
from a new (abstract) class called <em>Producer</em> so we can keep track of those instances.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Process</span><span class="p">:</span>
<span class="n">next_id</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">non_producer_processes</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">producer_processes</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">Process</span><span class="o">.</span><span class="n">next_id</span>
<span class="n">Process</span><span class="o">.</span><span class="n">next_id</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c"># Keep track of all source processes (join on these at the end), and non-source processes (cancel on these at the end)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">Producer</span><span class="p">):</span>
<span class="n">Process</span><span class="o">.</span><span class="n">producer_processes</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">Process</span><span class="o">.</span><span class="n">non_producer_processes</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">f</span><span class="s">"{self.name}.{self.id}"</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">f</span><span class="s">"{type(self).__name__}('{self.name}')"</span>
<span class="k">def</span> <span class="nf">message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">m</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">"{self}: {m}"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Producer</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="c"># All processes that drive the system (by injecting values in on channels unconditionally)</span>
<span class="c"># must subclass this process</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">Source</span><span class="p">(</span><span class="n">Producer</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">srcval</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">val</span> <span class="o">=</span> <span class="n">srcval</span>
<span class="bp">self</span><span class="o">.</span><span class="n">length</span> <span class="o">=</span> <span class="n">length</span>
<span class="bp">self</span><span class="o">.</span><span class="n">R</span> <span class="o">=</span> <span class="n">OutputPort</span><span class="p">()</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">length</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {self.val}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">val</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sent {self.val}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="s">"terminated"</span><span class="p">)</span>
</pre></div>
<p>Now, the <tt class="docutils literal">producer_processes</tt> dict keeps track of all the <em>Producer</em> type processes,
and <tt class="docutils literal">non_producer_processes</tt> stores the rest. We also change <em>Source</em> to inherit from
the new <em>Producer</em> subclass.</p>
<p>And that's pretty much it, we can now create a generic <tt class="docutils literal">run_all</tt> function like so:</p>
<div class="highlight"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">run_all</span><span class="p">():</span>
<span class="n">source_tasks</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">other_tasks</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">Process</span><span class="o">.</span><span class="n">producer_processes</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
<span class="n">source_tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="k">exec</span><span class="p">()))</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">Process</span><span class="o">.</span><span class="n">non_producer_processes</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
<span class="n">other_tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="k">exec</span><span class="p">()))</span>
<span class="c"># Now wait for all sources to end</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">source_tasks</span><span class="p">:</span>
<span class="n">await</span> <span class="n">task</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">other_tasks</span><span class="p">:</span>
<span class="n">await</span> <span class="n">task</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
</pre></div>
<p>And our <tt class="docutils literal">system</tt> reduces to just the following:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">curio</span> <span class="kn">import</span> <span class="n">run</span><span class="p">,</span> <span class="n">spawn</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">system</span><span class="p">():</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">10</span> <span class="c"># How many buffers in our linear pipeline</span>
<span class="c"># Instantiate the processes</span>
<span class="n">src</span> <span class="o">=</span> <span class="n">Source</span><span class="p">(</span><span class="s">'src1'</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="n">Buffer</span><span class="p">(</span><span class="n">f</span><span class="s">'buf[{i}]'</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">)]</span>
<span class="n">snk</span> <span class="o">=</span> <span class="n">Sink</span><span class="p">(</span><span class="s">'snk'</span><span class="p">)</span>
<span class="c"># Connect the processes with the channels</span>
<span class="n">connect</span><span class="p">(</span><span class="n">src</span><span class="o">.</span><span class="n">R</span><span class="p">,</span> <span class="n">buf</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">L</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">):</span>
<span class="n">connect</span><span class="p">(</span><span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">R</span><span class="p">,</span> <span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">L</span><span class="p">)</span>
<span class="n">connect</span><span class="p">(</span><span class="n">snk</span><span class="o">.</span><span class="n">L</span><span class="p">,</span> <span class="n">buf</span><span class="p">[</span><span class="n">N</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">R</span><span class="p">)</span>
<span class="n">await</span> <span class="n">run_all</span><span class="p">()</span>
</pre></div>
<p>Much simpler and requires just the <em>Process</em> and <em>Connection</em> definitions! The runnable
code is at <a class="reference external" href="https://gist.github.com/f674356a82fadc95841ffae437e38838">run_all_gist</a>.</p>
</div>
<div class="section" id="type-annotations-for-readability">
<h2><a class="toc-backref" href="#id4">4 Type-annotations for readability</a></h2>
<p>Python 3 introduced type annotations as part of the syntax, and I'm going to leverage
that to specify the ports on a Process in a cleaner way. For example, instead of
instantiating an <em>InputPort</em> and an <em>OutputPort</em> manually in the constructor of the
<em>Buffer</em> process, I'm going to add support to do the following:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Buffer</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="n">L</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="n">R</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">L</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"received {tok}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {tok}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">tok</span><span class="p">)</span>
</pre></div>
<p>The syntax of <tt class="docutils literal">variable: type</tt> is an annotation, and although Python does not
yet assert any type checks based on this (although there are static type
checking tools out there that do), we can still get access to an object's annotations
by looking at the <tt class="docutils literal">__annotations__</tt> member dict, which is a mapping of the name to
the type like so:</p>
<div class="highlight"><pre><span class="o">>>></span> <span class="n">b</span> <span class="o">=</span> <span class="n">Buffer</span><span class="p">(</span><span class="s">"buf"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">b</span><span class="o">.</span><span class="n">__annotations__</span>
<span class="p">{</span><span class="s">'L'</span><span class="p">:</span> <span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">__main__</span><span class="o">.</span><span class="n">InputPort</span><span class="s">'>, 'R': <class '</span><span class="n">__main__</span><span class="o">.</span><span class="n">OutputPort</span><span class="s">'>}</span>
</pre></div>
<p>I'm going to modify the <em>Process</em> base class's constructor to take any annotations
that are ports, and inject <strong>instances</strong> of those types into the object like so:</p>
<div class="highlight"><pre> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">Process</span><span class="o">.</span><span class="n">next_id</span>
<span class="n">Process</span><span class="o">.</span><span class="n">next_id</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c"># Keep track of all source processes (join on these at the end), and non-source processes (cancel on these at the end)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">Producer</span><span class="p">):</span>
<span class="n">Process</span><span class="o">.</span><span class="n">producer_processes</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">Process</span><span class="o">.</span><span class="n">non_producer_processes</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span>
<span class="c"># Inject the ports from the annotations on this instance</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">val</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">__annotations__</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">Port</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">'injecting port({val}) {name} onto {self.__class__}:{self.name}'</span><span class="p">)</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">val</span><span class="p">()</span>
<span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span>
</pre></div>
<p>Ignoring the <tt class="docutils literal">print</tt> that's only three extra lines needed to move to this cleaner
syntax. Isn't Python awesome?</p>
<p>You can find the complete code at <a class="reference external" href="https://gist.github.com/54a7d02d4d50a9fcbce8082c34ff774d">annotate_gist</a>.</p>
</div>
<div class="section" id="adding-simulation-timestamps-to-the-framework">
<h2><a class="toc-backref" href="#id5">5 Adding simulation timestamps to the framework</a></h2>
<p>Finally, let's make our simulator able to model the performance of our hardware system.
We can make each <em>Process</em> have a timestep that corresponds to how long it takes to
execute its code. Each time through, each <em>Process</em> will update its local time with
this timestep. Then, when communication occurs, we simply dispatch the local time
with each sent value. At the receiving end, the local time is either smaller or larger
than the message timestamp. Since messages can't have come from the future, if the
local time is smaller, then we need to update the receiver's local time to the
received messages timestamp, in order not to violate causality.</p>
<p>So what will it take to do this? I'm going to introduce a default <em>timestep</em> variable
to the <em>Process</em> class, and set it to 100 units. I'm also going to introduce an
<tt class="docutils literal">advance_time</tt> method that will increment the local <tt class="docutils literal">_time</tt> by the timestep
when called like so:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Process</span><span class="p">:</span>
<span class="n">next_id</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">non_producer_processes</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">producer_processes</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">timestep</span> <span class="o">=</span> <span class="mi">100</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">Process</span><span class="o">.</span><span class="n">next_id</span>
<span class="n">Process</span><span class="o">.</span><span class="n">next_id</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c"># Keep track of all source processes (join on these at the end), and non-source processes (cancel on these at the end)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">Producer</span><span class="p">):</span>
<span class="n">Process</span><span class="o">.</span><span class="n">producer_processes</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">Process</span><span class="o">.</span><span class="n">non_producer_processes</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span>
<span class="c"># Inject the ports from the annotations on this instance</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">val</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">__annotations__</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">Port</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">'injecting port({val}) {name} onto {self.__class__}:{self.name}'</span><span class="p">)</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">val</span><span class="p">()</span>
<span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span>
<span class="n">port</span><span class="o">.</span><span class="n">proc</span> <span class="o">=</span> <span class="bp">self</span> <span class="c"># Store the process this port is a part of (used for updating local time in the proc on a receive)</span>
<span class="c"># Local time</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_time</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">advance_local_time</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_time</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">timestep</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">f</span><span class="s">"{self.name}.{self.id}"</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">f</span><span class="s">"{type(self).__name__}('{self.name}')"</span>
<span class="k">def</span> <span class="nf">message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">m</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span> <span class="n">f</span><span class="s">'T:{self._time}: {self} - { m }'</span><span class="p">)</span>
</pre></div>
<p>I've also modified the <tt class="docutils literal">message</tt> method to provide the current time on each print
while the simulation is running. One other thing to note is that I changed the
<em>Port</em> injection code to allow every <em>Port</em> to know its containing <em>Process</em>; this
gives every <em>Port</em> the ability to access the local <tt class="docutils literal">_time</tt> variable. Once each
<em>Port</em> has this information, we just make the <tt class="docutils literal">send</tt> method transmit a tuple of the
value <em>and</em> the local <tt class="docutils literal">_time</tt>, and the <tt class="docutils literal">receive</tt> method can compare each
incoming message's timestamp with the local <tt class="docutils literal">_time</tt>, and update it if necessary:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Port</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">chan</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">proc</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">class</span> <span class="nc">InputPort</span><span class="p">(</span><span class="n">Port</span><span class="p">):</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">recv</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">tok</span><span class="p">,</span> <span class="n">timestamp</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">chan</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">proc</span><span class="o">.</span><span class="n">_time</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">proc</span><span class="o">.</span><span class="n">_time</span><span class="p">,</span> <span class="n">timestamp</span><span class="p">)</span>
<span class="k">return</span> <span class="n">tok</span>
<span class="k">class</span> <span class="nc">OutputPort</span><span class="p">(</span><span class="n">Port</span><span class="p">):</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">send</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">val</span><span class="p">):</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">chan</span><span class="o">.</span><span class="n">send</span><span class="p">((</span><span class="n">val</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">proc</span><span class="o">.</span><span class="n">_time</span><span class="p">))</span>
</pre></div>
<p>The only remaining thing to modify is each <em>Process</em> to make sure we advance
the local time at the proper place. For example, in the buffer we will advance the
time after the receive but before the send, to model the execution time of processing
a value like so:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Buffer</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="n">L</span><span class="p">:</span> <span class="n">InputPort</span>
<span class="n">R</span><span class="p">:</span> <span class="n">OutputPort</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">L</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"received {tok}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">advance_local_time</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {tok}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">R</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">tok</span><span class="p">)</span>
</pre></div>
<p>And that's it. If we run the simulation now, we get the following output, with
the simulation time reported in every message:</p>
<div class="highlight"><pre>T:0: src1.0 - sending 1
T:100: src1.0 - sent 1
T:100: src1.0 - sending 1
T:100: buf<span class="o">[</span>0<span class="o">]</span>.1 - received 1
T:200: buf<span class="o">[</span>0<span class="o">]</span>.1 - sending 1
T:200: buf<span class="o">[</span>1<span class="o">]</span>.2 - received 1
T:300: buf<span class="o">[</span>1<span class="o">]</span>.2 - sending 1
.
.
.
T:1700: buf<span class="o">[</span>7<span class="o">]</span>.8 - received 1
T:1800: buf<span class="o">[</span>7<span class="o">]</span>.8 - sending 1
T:1800: buf<span class="o">[</span>9<span class="o">]</span>.10 - received 1
T:1900: buf<span class="o">[</span>9<span class="o">]</span>.10 - sending 1
T:1800: buf<span class="o">[</span>8<span class="o">]</span>.9 - received 1
T:1900: buf<span class="o">[</span>8<span class="o">]</span>.9 - sending 1
T:2000: snk.11 - received 1
T:1900: buf<span class="o">[</span>9<span class="o">]</span>.10 - received 1
T:2000: buf<span class="o">[</span>9<span class="o">]</span>.10 - sending 1
T:2100: snk.11 - received 1
T:2100: snk.11 - 10 tokens received
</pre></div>
<p>The complete runnable code is available as a gist - <a class="reference external" href="https://gist.github.com/6c7602a08ce5bb4abfc30c7f66f63f67">timestamp_gist</a>.</p>
<p>And that brings us to the close of this article. In <a class="reference external" href="https://virantha.com/2017/09/28/hardware-simulation-using-curio-3/">Part 3</a>, I'll demonstrate
how we can build and model a hardware palindrome checker using this framework. Thanks for reading!</p>
</div>
</body></html>Part 1 - Building a Fast Event-Driven Simulator for Concurrent Systems of Hardware Processes Using Curio and Python 3.62017-09-22T11:20:00-04:00viranthatag:https://virantha.com,2017-09-22:2017/09/22/hardware-simulation-using-curio/<p>In this post, the first in a series of three, I'm going to show you how to
build an elegant simulator of a concurrent system using the amazing <a class="reference external" href="http://curio.readthedocs.io/en/latest/">Curio</a>
library from David Beazley that's built on Python 3.6's async coroutine
features. I hope it'll provide a nice intro to how to take advantage of
event-driven programming in Python using the <a class="reference external" href="http://curio.readthedocs.io/en/latest/">Curio</a> library and the new
async/await semantics in Python 3.4+ (although I will take advantage of some
cool new 3.6 features to simplify my code).</p>
<p>In Part 1, we'll quickly build a simulation framework, which
will then be enhanced and optimized into a "proper" framework in <a class="reference external" href="https://virantha.com/2017/09/27/hardware-simulation-using-curio-2/">Part 2</a>. I intend
to finish up in <a class="reference external" href="https://virantha.com/2017/09/28/hardware-simulation-using-curio-3/">Part 3</a> with a set of concrete examples on how to use it to simulate
your own message-passing hardware systems that can solve interesting problems.</p>
<div id="background" class="section">
<h2><a class="toc-backref" href="#id1">1 Background</a></h2>
<p>First, some background on what I'll be describing:
In graduate school and in my startup, we built hardware using asynchronous
digital logic, and used a high-level abstraction called Communicating Hardware
Processes (CHP) to model and reason about designs. CHP, based on Tony Hoare's
<a class="reference external" href="http://www.usingcsp.com">CSP</a>, is a way to represent circuits as a concurrent collection of sequential
processes communicating and synchronizing via messaging channels. I'm not
going to dive into the nuts and bolts of CHP, but I will show you how to build
a simulator that models these types of event-driven systems using an incredibly
small amount of code.</p>
<p>Second, why not <tt class="docutils literal">asyncio</tt>, the new standard library that provides
event-driven programming hooks? I did start there (and I highly recommend
watching this <a class="reference external" href="https://www.youtube.com/watch?v=M-UcUs7IMIM">talk</a> by Robert Smallshire that builds up to the async constructs
in Python 3.4 from the ground up), but in the end, I found <tt class="docutils literal">asyncio</tt> a little
too complicated and convoluted to get into quickly. It provides a strong set
of different event-driven abstractions based on generators, tasks, channels,
callbacks, and coroutines, and concrete implementations of different uses, but
I was really just looking for something that provided a coroutine based
event-loop, something that Curio provides quite naturally.</p>
</div>
<div id="describing-hardware-using-communicating-processes" class="section">
<h2><a class="toc-backref" href="#id2">2 Describing hardware using communicating processes</a></h2>
<p>Let's try to nail down some terminology.</p>
<ul class="simple">
<li><strong>Process</strong> - This represents a piece of processing or code that's being executed sequentially.</li>
<li><strong>Channel</strong> - A channel is used to connect two <em>processes</em> together. A
channel is strictly unidirectional, and one side is connected to the <em>sender</em>
process and the other side to a <em>receiver</em> process. Channels are used to
transfer data (like ints or strings) from the sender to the receiver. A
channel, as a by-product of the way it works, also functions to synchronize
the sender and receiver processes -- if the receiver isn't ready to receive
and the channel is full, then the sender will block until the message is
transferred into the channel.</li>
<li><strong>Port</strong> - The sender process has an <strong>output port</strong> that is connected to the channel, and the receiver has an <strong>input port</strong> on its end of the channel.</li>
</ul>
<p>As an example, let's consider the simple one-place buffer that's shown below.
<sup id="sf-hardware-simulation-using-curio-1-back"><a title="from preprint of Asynchronous VLSI Systems by Rajit Manohar" class="simple-footnote" href="#sf-hardware-simulation-using-curio-1">1</a></sup> The
single buffer has two ports, with L being the input port, and R being the
output port. All the buffer does is receive a value on L, store it in <em>x</em>, and then send
it out on R. We can then construct a <strong>system</strong> of these buffers that implement a
linear pipeline by connecting the adjacent R and L ports with <em>channels</em>.</p>
<div class="figure align-center" style="width: 836px; height: auto; max-width: 100%;">
<img height="175" alt="Linear buffers" src="/images/2017/chp/buf.png" width="836" style="width: 836px; height: auto; max-width: 100%;">
</div>
</div>
<div id="building-the-simulator" class="section">
<h2><a class="toc-backref" href="#id3">3 Building the Simulator</a></h2>
<div id="the-process-class" class="section">
<h3><a class="toc-backref" href="#id4">3.1 The Process class</a></h3>
<p>First thing, let's start with a simple class to represent a <em>Process</em> (reminder
that all code I'm writing probably <strong>requires Python 3.6</strong>)</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Process</span><span class="p">:</span>
<span class="n">next_id</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">Process</span><span class="o">.</span><span class="n">next_id</span>
<span class="n">Process</span><span class="o">.</span><span class="n">next_id</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"{}.{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"{}('{}')"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">m</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s">"{}: {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="p">),</span> <span class="n">m</span><span class="p">))</span>
</pre></div>
<p>Nothing too crazy here; just a constructor that keeps track of a unique process id for
every single Process that's created, plus some utility functions to create nice
string representations and also a <em>message</em> method to hand output from the process in
a standard way.</p>
<p>I want to segue here and introduce a new feature in 3.6 called <a class="reference external" href="https://www.python.org/dev/peps/pep-0498/">fstrings</a>, which
are faster, cleaner to read, and just plain awesome once you start using them.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Process</span><span class="p">:</span>
<span class="n">next_id</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">Process</span><span class="o">.</span><span class="n">next_id</span>
<span class="n">Process</span><span class="o">.</span><span class="n">next_id</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">f</span><span class="s">"{self.name}.{self.id}"</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">f</span><span class="s">"{type(self).__name__}('{self.name}')"</span>
<span class="k">def</span> <span class="nf">message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">m</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">"{self}: {m}"</span><span class="p">)</span>
</pre></div>
<p>Look at that, interpolation of code inside strings!</p>
<p>Alright, now let's create a Process subclass that actually does something
useful: a <em>Source</em> process that sends a specified value on a channel a fixed
number of times.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Source</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">srcval</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">val</span> <span class="o">=</span> <span class="n">srcval</span>
<span class="bp">self</span><span class="o">.</span><span class="n">length</span> <span class="o">=</span> <span class="n">length</span>
<span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">chan</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">out_chan</span> <span class="o">=</span> <span class="n">chan</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">length</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {self.val}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">out_chan</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">val</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sent {self.val}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="s">"terminated"</span><span class="p">)</span>
</pre></div>
<p>The constructor lets us set the value and sequence length to output, and the
<em>connect</em> rather naively just connects the passed in channel to an instance variable.
We'll define channels in the next section, but for now, let's focus on the
<strong>exec</strong> method which does the actual work. The <tt class="docutils literal">async</tt> keyword marks this as
an asynchronously executing coroutine (technically, it marks the method as an <em>awaitable</em>). This is the code that will get executed in <a class="reference external" href="http://curio.readthedocs.io/en/latest/">Curio</a>'s event loop.</p>
<p>The code itself iterates the length of the sequence and calls a send method on the output
channel. The method call is preceded by an <tt class="docutils literal">await</tt> keyword, which is how you call
<tt class="docutils literal">async</tt> methods (the <tt class="docutils literal">send</tt> method will be an async method). You can read up
on the semantics of this (things like <tt class="docutils literal">yield from</tt>), but if you just remember to use
<tt class="docutils literal">await</tt> before calling an async method as a rule, you'll be all set.</p>
<p>Now, let's further introduce a <em>Sink</em> process that 'eats' values (say from a source), and
the <em>Buffer</em> process described in the previous section that just receives and passes
on a value.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">curio</span> <span class="kn">import</span> <span class="n">CancelledError</span>
<span class="k">class</span> <span class="nc">Sink</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">chan</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">in_chan</span> <span class="o">=</span> <span class="n">chan</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">tok_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_chan</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">tok_count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"received {tok}"</span><span class="p">)</span>
<span class="k">except</span> <span class="n">CancelledError</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"{tok_count} tokens received"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Buffer</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">chan_l</span><span class="p">,</span> <span class="n">chan_r</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">l_chan</span> <span class="o">=</span> <span class="n">chan_l</span>
<span class="bp">self</span><span class="o">.</span><span class="n">r_chan</span> <span class="o">=</span> <span class="n">chan_r</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">exec</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">l_chan</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"received {tok}"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">(</span><span class="n">f</span><span class="s">"sending {tok}"</span><span class="p">)</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">r_chan</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">tok</span><span class="p">)</span>
</pre></div>
<p>One new thing here is how we receive data from an <tt class="docutils literal">async</tt> method. The
incoming data (or token) is assigned to the result of the <tt class="docutils literal">await
self.in_chan.recv()</tt> call. (Recall that the shortcut is, wherever we'd do a
function call in synchronous code, we do an <tt class="docutils literal">await</tt> on the function call
instead in async code). The other point to note is that we can catch a
<em>CancelledError</em> like in the <em>Sink</em> process to print something out if this
process is terminated by the environment (which it will be, since otherwise
it will just hang waiting for more input).</p>
</div>
<div id="the-channel-class" class="section">
<h3><a class="toc-backref" href="#id5">3.2 The Channel class</a></h3>
<p>One more thing before we can make our first attempt at simulating a system of
processes: We need a way to implement the communication channels. Luckily, Curio has
a built-in Queue type that's explicitly designed for this kind of communication.</p>
<p>Here's our first stab at the Channel class, assuming we've done a <tt class="docutils literal">from curio import Queue</tt>:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">curio</span> <span class="kn">import</span> <span class="n">Queue</span>
<span class="k">class</span> <span class="nc">Channel</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">(</span><span class="n">maxsize</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="c"># Max buffering of 1</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">send</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">val</span><span class="p">):</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">val</span><span class="p">)</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">recv</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">tok</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">q</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
<span class="k">return</span> <span class="n">tok</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">q</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
<p>Here, we've defined a Channel that has the actions <em>send</em> and <em>recv</em> (and a
utility function <em>close</em> to terminate it). The <em>send</em> inserts a value into the
internal Queue, and the <em>recv</em> will <tt class="docutils literal">await</tt> on the value (and block if none
is available). Once the <em>recv</em> is done, it's important to call the
<tt class="docutils literal">task_done</tt> method on the Queue to signal that the receive is complete.</p>
</div>
</div>
<div id="running-the-system" class="section">
<h2><a class="toc-backref" href="#id6">4 Running the system</a></h2>
<p>Now we have all the initial pieces available to put together a complete simulation
of a linear pipeline of buffers with its environment. Here's our first version of the code:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">curio</span> <span class="kn">import</span> <span class="n">run</span><span class="p">,</span> <span class="n">spawn</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">system</span><span class="p">():</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">10</span> <span class="c"># How many buffers in our linear pipeline</span>
<span class="c"># Define the channels</span>
<span class="n">chan_l</span> <span class="o">=</span> <span class="n">Channel</span><span class="p">(</span><span class="s">'l'</span><span class="p">)</span>
<span class="n">chan_r</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">):</span>
<span class="n">chan_r</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Channel</span><span class="p">(</span><span class="n">f</span><span class="s">'R[{i}]'</span><span class="p">))</span>
<span class="c"># Instantiate the processes</span>
<span class="n">src</span> <span class="o">=</span> <span class="n">Source</span><span class="p">(</span><span class="s">'src1'</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="n">Buffer</span><span class="p">(</span><span class="n">f</span><span class="s">'buf[{i}]'</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">)]</span>
<span class="n">snk</span> <span class="o">=</span> <span class="n">Sink</span><span class="p">(</span><span class="s">'snk'</span><span class="p">)</span>
<span class="c"># Connect the processes with the channels</span>
<span class="n">src</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">chan_l</span><span class="p">)</span>
<span class="n">buf</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">chan_l</span><span class="p">,</span> <span class="n">chan_r</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">):</span>
<span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">chan_r</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">chan_r</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="n">snk</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">chan_r</span><span class="p">[</span><span class="n">N</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c"># Start the processes</span>
<span class="n">p_src</span> <span class="o">=</span> <span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">src</span><span class="o">.</span><span class="k">exec</span><span class="p">())</span>
<span class="n">p_snk</span> <span class="o">=</span> <span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">snk</span><span class="o">.</span><span class="k">exec</span><span class="p">())</span>
<span class="n">p_buf</span> <span class="o">=</span> <span class="p">[</span><span class="n">await</span> <span class="n">spawn</span><span class="p">(</span><span class="n">buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="k">exec</span><span class="p">())</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">)]</span>
<span class="c"># Wait for the source to finish sending all its values</span>
<span class="n">await</span> <span class="n">p_src</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="c"># Cancel the remaining processes</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span><span class="p">):</span>
<span class="n">await</span> <span class="n">p_buf</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
<span class="n">await</span> <span class="n">p_snk</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span><span class="o">==</span><span class="s">'__main__'</span><span class="p">:</span>
<span class="n">run</span><span class="p">(</span><span class="n">system</span><span class="p">(),</span> <span class="n">with_monitor</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
<p>It's still a little more verbose than I'd like, but we'll fix that in the next part.
Notice how all the Processes are started: We do an <tt class="docutils literal">await spawn</tt> on the coroutines
(in this case the <tt class="docutils literal">exec</tt> coroutine method of our Processes) to start all the
Processes running as tasks in Curio's event loop. Then, we just wait for the
<em>Source</em> process to be complete by doing a <tt class="docutils literal">join()</tt> on its task. Once we know the
<em>Source</em> is done, we can go ahead and call <tt class="docutils literal">cancel()</tt> on all the remaining tasks,
because we know they are just waiting on their input channels.</p>
<div class="highlight"><pre>src1.0: sending 1
src1.0: sent 1
src1.0: sending 1
buf[0].1: received 1
buf[0].1: sending 1
buf[1].2: received 1
buf[1].2: sending 1
buf[2].3: received 1
buf[2].3: sending 1
.
.
.
src1.0: sent 1
src1.0: terminated
buf[1].2: received 1
buf[1].2: sending 1
.
.
.
buf[8].9: received 1
buf[8].9: sending 1
snk.11: received 1
buf[9].10: received 1
buf[9].10: sending 1
snk.11: received 1
snk.11: 10 tokens received
</pre></div>
<p>And there you go, a complete distributed system, simulated using Curio's
concurrent event loop! You can find the complete runnable code (Python 3.6!)
in this <a class="reference external" href="https://gist.github.com/ce2bca039ce3522e1b4efccf0fe49c01">gist</a>.</p>
<p><a class="reference external" href="https://virantha.com/2017/09/27/hardware-simulation-using-curio-2/">In the next article</a> in this series, we'll talk about enhancing the simulator
and adding some error-checking.</p>
</div>
<ol class="simple-footnotes">Notes:<li id="sf-hardware-simulation-using-curio-1">from preprint of Asynchronous VLSI Systems by Rajit Manohar <a class="simple-footnote-back" href="#sf-hardware-simulation-using-curio-1-back">↩</a></li></ol>How to use Python to generate extensions with a GUI for Moneydance 20172016-11-28T12:27:00-05:00viranthatag:https://virantha.com,2016-11-28:2016/11/28/moneydance-python-extension-tips/<html><body><p>This post describes some of my experiences writing a python extension for Moneydance, a cross-platform
personal finance manager (an alternative to Intuit Quicken). I was originally attracted to this product
because it runs on a Mac (albeit using a Java look and feel), uses Direct Connect to download transactions from financial instituitions,
has a reasonable price, and supports scripting to get at the entire transaction and account internals.</p>
<p>Most of this post is built-around using Python (via the built-in Jython 2.7
interface) to build an extension based on the information in the links below.
Hopefully, this post will fill in some of the gaps. I was completely
unfamiliar with Jython when I started, but I've been amazed at how easy it is to
seamlessly integrate with any Java library, including Swing for building GUI's.
You just import the Java library you want and you have complete access to all
the APIs.</p>
<ul class="simple">
<li><a class="reference external" href="https://infinitekind.com/developer">Moneydance Developer Resource page</a></li>
<li><a class="reference external" href="http://help.infinitekind.com/discussions/moneydance-development">Moneydance developer forum</a></li>
<li><a class="reference external" href="https://infinitekind.com/dev/python_template.py">Sample template from Moneydance</a></li>
<li><a class="reference external" href="http://infinitekind.com/dev/RM-NetWorth/wiki_jython.html">Moneydance Jython notes</a></li>
</ul>
<div class="section" id="pop-up-a-simple-frame-with-a-button">
<h2><a class="toc-backref" href="#id1">1 Pop up a simple frame with a button</a></h2>
<p>Here's a simple script based off the Moneydance template that will pop-up a frame with a scrollable list of the names of the current accounts.</p>
<div class="figure" style="width: 1368px; height: auto; max-width: 100%;">
<img alt="Popup" height="850" src="https://virantha.com/images/moneydance/popup.png" style="width: 1368px; height: auto; max-width: 100%;" width="1368"/>
</div>
<p>Keep in mind that in Moneydance developer terminology an account can be either
a top-level user account like 'Checking' or a <em>Category</em>.</p>
<p>You can see the power
and conciseness of Jython married with access to Moneydance internals in the few lines of code it took to accomplish the above:</p>
<div class="gist">
<script src='https://gist.github.com/5ace1fa44a2e1156e8542523596124f1.js?file=1-simple-moneydance-popup.py'></script>
<noscript>
<pre><code>from com.infinitekind.moneydance.model import *
import os
from javax.swing import JButton, JFrame, JScrollPane, JTextArea, BoxLayout, BorderFactory
class SimplePopupExtension(object):
def initialize(self, extension_context, extension_object):
self.moneydanceContext = extension_context
self.moneydanceExtensionObject = extension_object
# here we register ourselves with a menu item to invoke a feature
# (ignore the button and icon mentions in the docs)
self.moneydanceContext.registerFeature(extension_object, "popup", None, "Test popup")
# invoke(eventstring) is called when we receive a callback for the feature that
# we registered in the initialize method
def invoke(self, eventString=""):
self.moneydanceContext.setStatus("Python extension received command: %s" % (eventString))
if eventString=='popup':
self._show_popup()
def __str__(self):
return "SimplePopup"
# Our actual user code
def _show_popup(self):
self._initUI()
book = moneydance.getCurrentAccountBook()
account_names = []
for acct in AccountUtil.getAccountIterator(book):
account_names.append(acct.getFullAccountName())
self.text_area.text = '\n'.join(account_names)
def _initUI(self):
frame = JFrame('Simple popup', defaultCloseOperation = JFrame.DISPOSE_ON_CLOSE, size=(300,300))
frame.setLayout(BoxLayout(frame.contentPane, BoxLayout.PAGE_AXIS))
# Call-back to close popup
def closeDialog(event):
frame.visible = False
frame.dispose()
# Instantiate components
self.text_area = JTextArea()
msgScroller = JScrollPane()
msgScroller.setBorder(BorderFactory.createTitledBorder("Accounts"))
msgScroller.setViewportView(self.text_area)
self.close_button = JButton('Close', actionPerformed = closeDialog)
# Add components to frame
frame.add(msgScroller)
frame.add(self.close_button)
frame.visible = True
# Tell moneydance this is an extension
moneydance_extension = SimplePopupExtension()
</code></pre>
</noscript>
</div>
<p>To get this installed, simply open Moneydance 2017 (previous versions will not work) and select <tt class="docutils literal">Python Scripting</tt> from the <tt class="docutils literal">Windows</tt> menu.
This will pop-up the Python interface where you'll do the following:</p>
<div class="figure" style="width: 1368px; height: auto; max-width: 100%;">
<img alt="Installing the extension" height="928" src="https://virantha.com/images/moneydance/install.png" style="width: 1368px; height: auto; max-width: 100%;" width="1368"/>
</div>
<p>If all goes well, in the <tt class="docutils literal">Extensions</tt> menu you will see a new item called <tt class="docutils literal">Test popup</tt>. If you select this, then the sample
code will run and generate the popup.</p>
</div>
<div class="section" id="importing-python-files">
<h2><a class="toc-backref" href="#id2">2 Importing Python files</a></h2>
<p>If you want to import code from another local file, you need to append the location of your script to sys.path before doing the import in your top-level script:</p>
<div class="highlight"><pre><span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span> <span class="ow">and</span> <span class="n">__package__</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">search_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">__file__</span><span class="p">))</span>
<span class="k">if</span> <span class="n">search_path</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">search_path</span><span class="p">)</span>
</pre></div>
<p><strong>Caveat</strong>: It seems like Moneydance does not re-import any imports if you reload a script,
making any updates to files other than your top-level script not visible except
for the first time after a clean Moneydance startup. While not an issue when
deploying a finished extension, this is a pain during development when you're
changing code all the time, and the work-around I've found is to <tt class="docutils literal">reload</tt>
your module right after the import:</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">my_module</span>
<span class="nb">reload</span><span class="p">(</span><span class="n">my_module</span><span class="p">)</span>
</pre></div>
<p>This forces moneydance to re-import your local dependencies when you want to break your
code into multiple files.</p>
</div>
<div class="section" id="import-jar-files">
<h2><a class="toc-backref" href="#id3">3 Import JAR files</a></h2>
<p>This took me a long time to figure out. If
you need to use a third-party Java library supplied as a JAR, I've so far found
two ways to do this:</p>
<p>1. Copy the JAR into Moneydance's Java library path. On my Mac, this was
inside the application bundle at
<tt class="docutils literal">/Applications/Moneydance.app/Contents/Java/Library</tt>. I suspect if you modify
your
system classpath and put it in there, that will work too, but I have not tried this. Regardless, if you use this method,
then all you need is to add the import like so.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">net.miginfocom.swing</span> <span class="kn">import</span> <span class="n">MigLayout</span>
</pre></div>
<p>2. If you need a more portable way to distribute this without having the user
install random jar files into their system libraries, I found I had to use a
custom classloader to make sure all the introspection was supported (For
example, I was trying to use a third-party Swing layout library called
<a class="reference external" href="http://www.miglayout.com">MigLayout</a>). You can read more about this <a class="reference external" href="http://stackoverflow.com/questions/3015059/jython-classpath-sys-path-and-jdbc-drivers">here</a>,
but here's the code you need to insert at the top of your script.</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">loadJar</span><span class="p">(</span><span class="n">jarFile</span><span class="p">):</span>
<span class="sd">'''load a jar at runtime using the system Classloader (needed for JDBC)</span>
<span class="sd"> adapted from http://forum.java.sun.com/thread.jspa?threadID=300557</span>
<span class="sd"> Author: Steve (SG) Langer Jan 2007 translated the above Java to Jython</span>
<span class="sd"> Reference: https://wiki.python.org/jython/JythonMonthly/Articles/January2007/3</span>
<span class="sd"> Author: seansummers@gmail.com simplified and updated for jython-2.5.3b3+</span>
<span class="sd"> '''</span>
<span class="kn">from</span> <span class="nn">java</span> <span class="kn">import</span> <span class="n">io</span><span class="p">,</span> <span class="n">net</span><span class="p">,</span> <span class="n">lang</span>
<span class="n">u</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">File</span><span class="p">(</span><span class="n">jarFile</span><span class="p">)</span><span class="o">.</span><span class="n">toURL</span><span class="p">()</span> <span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">jarFile</span><span class="p">)</span> <span class="o"><></span> <span class="n">net</span><span class="o">.</span><span class="n">URL</span> <span class="k">else</span> <span class="n">jarFile</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">net</span><span class="o">.</span><span class="n">URLClassLoader</span><span class="o">.</span><span class="n">getDeclaredMethod</span><span class="p">(</span><span class="s">'addURL'</span><span class="p">,</span> <span class="p">[</span><span class="n">net</span><span class="o">.</span><span class="n">URL</span><span class="p">])</span>
<span class="n">m</span><span class="o">.</span><span class="n">accessible</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">m</span><span class="o">.</span><span class="n">invoke</span><span class="p">(</span><span class="n">lang</span><span class="o">.</span><span class="n">ClassLoader</span><span class="o">.</span><span class="n">getSystemClassLoader</span><span class="p">(),</span> <span class="p">[</span><span class="n">u</span><span class="p">])</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span> <span class="ow">and</span> <span class="n">__package__</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">search_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">__file__</span><span class="p">))</span>
<span class="n">loadJar</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">search_path</span><span class="p">,</span> <span class="s">'miglayout.jar'</span><span class="p">))</span>
<span class="kn">from</span> <span class="nn">java</span> <span class="kn">import</span> <span class="n">lang</span>
<span class="n">MigLayout</span> <span class="o">=</span> <span class="n">lang</span><span class="o">.</span><span class="n">Class</span><span class="o">.</span><span class="n">forName</span><span class="p">(</span><span class="s">'net.miginfocom.swing.MigLayout'</span><span class="p">)</span>
</pre></div>
<p>Now, the MigLayout class is available for use just like an import. Note that this assumes the
.jar file is in the same location as your top-level script (<tt class="docutils literal">search_path</tt>)</p>
</div>
<div class="section" id="persisting-data-in-your-secure-moneydance-file">
<h2><a class="toc-backref" href="#id4">4 Persisting data in your secure moneydance file</a></h2>
<p>If your extension needs to persist or save data to a file between sessions, there's an easy way
to do it through Moneydance's <tt class="docutils literal">LocalStorage</tt> API. This basically gives you programmatic access
to the currently open and encrypted .moneydance file that stores your user information, providing
a secure place to store things like passwords and pins.</p>
<p>Here are the steps to access it and persist a .pickle file to it:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">com.infinitekind.moneydance.model</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">from</span> <span class="nn">org.python.core.util</span> <span class="kn">import</span> <span class="n">FileUtil</span>
<span class="kn">from</span> <span class="nn">java.io</span> <span class="kn">import</span> <span class="n">FileNotFoundException</span>
<span class="kn">import</span> <span class="nn">pickle</span>
<span class="n">my_data_object</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'anything'</span><span class="p">:</span> <span class="s">'whatever'</span><span class="p">}</span>
<span class="n">root_account</span> <span class="o">=</span> <span class="n">moneydance</span><span class="o">.</span><span class="n">getRootAccount</span><span class="p">()</span>
<span class="n">local_storage</span> <span class="o">=</span> <span class="n">root_account</span><span class="o">.</span><span class="n">getBook</span><span class="p">()</span><span class="o">.</span><span class="n">getLocalStorage</span><span class="p">()</span>
<span class="c"># Save a file</span>
<span class="n">ostr</span> <span class="o">=</span> <span class="n">local_storage</span><span class="o">.</span><span class="n">openFileForWriting</span><span class="p">(</span><span class="s">'test.pickle'</span><span class="p">)</span>
<span class="n">save_file</span> <span class="o">=</span> <span class="n">FileUtil</span><span class="o">.</span><span class="n">wrap</span><span class="p">(</span><span class="n">ostr</span><span class="p">)</span>
<span class="n">pickle</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">my_data_object</span><span class="p">,</span> <span class="n">save_file</span><span class="p">)</span>
<span class="n">save_file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="c"># Open the file</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">istr</span> <span class="o">=</span> <span class="n">local_storage</span><span class="o">.</span><span class="n">openFileForReading</span><span class="p">(</span><span class="s">'test.pickle'</span><span class="p">)</span>
<span class="n">load_file</span> <span class="o">=</span> <span class="n">FileUtil</span><span class="o">.</span><span class="n">wrap</span><span class="p">(</span><span class="n">istr</span><span class="p">)</span>
<span class="n">my_data_object</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">load_file</span><span class="p">)</span>
<span class="k">except</span> <span class="n">FileNotFoundException</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">my_data_object</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">except</span> <span class="ne">EOFError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">my_data_object</span> <span class="o">=</span> <span class="bp">None</span>
</pre></div>
</div>
<div class="section" id="bundling-python-into-a-moneydance-java-extension">
<h2><a class="toc-backref" href="#id5">5 Bundling Python into a Moneydance Java Extension</a></h2>
<p>One thing you may notice is that installing Python extensions in this way only
keeps your extension around while Moneydance is running. If you quit and
restart, then you have to reinstall the extension, which is probably a current
limitation of the way Python extensions are supported in Moneydance.</p>
<p>However, after quite a bit of reading on the way Jython is integrated into
Java (including <a class="reference external" href="http://blog.smartbear.com/programming/embedding-jython-in-java-applications/">this page</a>), I've come up with the flow below to package up your Python extension as a
standard Moneydance .mxt extension and deploy it through the regular (and
persistent) extension manager.</p>
<div class="section" id="install-the-developer-kit">
<h3><a class="toc-backref" href="#id6">5.1 Install the Developer Kit</a></h3>
<p>These instructions assume you're able to get the Java extension development kit
downloaded and running. The example found <a class="reference external" href="http://infinitekind.com/developer">on the Developer page</a> under 'Download Developer's Kit' is quite
easy to get running. All you need is the JDK and the Ant build system (if
you're on a Mac, just install Sun's JDK and use Homebrew to install ant).</p>
<p>You can test the sample extension supplied in this kit by going into the <tt class="docutils literal">src</tt> sub-directory, which
contains the build instructions in <tt class="docutils literal">build.xml</tt>,
and first generating your signing keys by running the following:</p>
<pre class="literal-block">
ant genkeys
</pre>
<p>Then, everytime you want to build the extension, you type</p>
<pre class="literal-block">
ant myextension
</pre>
<p>It will prompt you to enter the passphrase you used for the keys in the
<tt class="docutils literal">genkeys</tt> step, and build the file <tt class="docutils literal">dist/myextension.mxt</tt>. Then, just go
to the Moneydance menu <tt class="docutils literal">Extensions <span class="pre">-></span> Manage <span class="pre">Extensions...</span></tt> and click on the
button at the bottom that says <tt class="docutils literal">Add from <span class="pre">file...</span></tt>. Find the new
myextension.mxt and install it, and then go to your Extensions menu and pick
the new item <tt class="docutils literal">Account list</tt> that will use the extension you just installed to
pop up a list of your accounts.</p>
<p>Now that you have Moneydance's example extension building successfully, we can
modify it slightly as discussed in the following to allow Python extensions.</p>
</div>
<div class="section" id="download-the-jython-standalone-jar">
<h3><a class="toc-backref" href="#id7">5.2 Download the Jython Standalone .jar</a></h3>
<p>In order to build extensions with Python, we need to add the Jython runtime to
each extension (since it's currently not bundled with Moneydance). Go to
<a class="reference external" href="http://www.jython.org/downloads.html">the Jython downloads page</a> and get the
<strong>Standalone.jar</strong> for Jython 2.7.0. Move this file into your devkit's <tt class="docutils literal">lib</tt>
directory.</p>
</div>
<div class="section" id="modify-build-xml">
<h3><a class="toc-backref" href="#id8">5.3 Modify build.xml</a></h3>
<p>We're going to slightly modify the supplied build.xml as shown below:</p>
<div class="gist">
<script src='https://gist.github.com/4464c7be4dc40cfe66dc3658059cdc31.js?file=build.xml'></script>
<noscript>
<pre><code><!--
build file for ant
http://jakarta.apache.org/ant/index.html
-->
<project name="myextension" default="all" basedir=".">
<property name="version" value="2.2"/>
<property name="src" value="."/>
<property name="build" value="./build"/>
<property name="privkeyfile" value="${src}/priv_key"/>
<property name="pubkeyfile" value="${src}/pub_key"/>
<property name="privkeyid" value="99"/>
<property name="build.compiler" value="classic"/>
<property name="build.compiler.fulldepend" value="true"/>
<property name="build.sysclasspath" value="ignore" /> <!-- suppress ridiculous "includeantruntime not set" messages from ant -->
<property name="build.includeantruntime" value="false"/>
<property name="dist" value="../dist"/>
<property name="tmp" value="../tmp"/>
<property name="debug" value="on"/>
<property name="optimize" value="off"/>
<path id="classpath">
<pathelement path="../lib/extadmin.jar"/>
<pathelement path="../lib/moneydance-dev.jar"/>
<pathelement path="../lib/jython-standalone-2.7.0.jar"/>
</path>
<target name="init">
<mkdir dir="${dist}"/>
<mkdir dir="${build}"/>
</target>
<target name="myextension" depends="init">
<javac target="1.6" source="1.6" srcdir="${src}" debug="${debug}" optimize="${optimize}"
classpathref="classpath" destdir="${build}"
includes="com/moneydance/modules/features/myextension/**"/>
<jar destfile="${dist}/myextension.mxt">
<fileset dir="${src}" includes="
com/moneydance/modules/features/myextension/meta_info.dict
com/moneydance/modules/features/myextension/*.gif
com/moneydance/modules/features/myextension/*.jpg
com/moneydance/modules/features/myextension/*.jpeg"/>
<fileset dir="${build}" includes="com/moneydance/modules/features/myextension/**"/>
<fileset dir="${src}/com/moneydance/modules/features/myextension/python" includes="**/**.py"/>
</jar>
<java newenvironment="true"
classpathref="classpath"
classname="com.moneydance.admin.KeyAdmin">
<arg value="signextjar"/>
<arg value="${privkeyfile}"/>
<arg value="${privkeyid}"/>
<arg value="myextension"/>
<arg line="${dist}/myextension.mxt"/>
</java>
<move file="${src}/s-myextension.mxt" tofile="${dist}/myextension.mxt"/>
</target>
<target name="genkeys">
<java
classpathref="classpath"
classname="com.moneydance.admin.KeyAdmin">
<arg value="genkey"/>
<arg value="${privkeyfile}"/>
<arg value="${pubkeyfile}"/>
</java>
</target>
<target name="all" depends="myextension"/>
</project>
</code></pre>
</noscript>
</div>
<p>First, around line 26 we add our <tt class="docutils literal">jython.jar</tt> include, and then we add a python
subdirectory to the code directory on line 46. The python subdirectory gets copied to
the root of the .mxt file during the build, which is important for being able to import
the extension in the next section.</p>
<p>Go ahead and replace your build.xml with this file.</p>
</div>
<div class="section" id="modify-main-java">
<h3><a class="toc-backref" href="#id9">5.4 Modify Main.java</a></h3>
<p>Now, we'll modify the main Java file for the extension to support arbitrary
Python extensions like so:</p>
<div class="gist">
<script src='https://gist.github.com/4464c7be4dc40cfe66dc3658059cdc31.js?file=Main.java'></script>
<noscript>
<pre><code>package com.moneydance.modules.features.myextension;
import com.moneydance.apps.md.controller.FeatureModule;
import com.moneydance.apps.md.controller.FeatureModuleContext;
import com.moneydance.apps.md.controller.ModuleUtil;
import com.moneydance.apps.md.controller.UserPreferences;
import java.io.*;
import java.util.*;
import java.text.*;
import java.awt.*;
import java.net.*;
import org.python.util.PythonInterpreter;
import org.python.core.*;
/** Pluggable module used to give users access to a Account List
interface to Moneydance.
*/
public class Main
extends FeatureModule
{
private PythonInterpreter pi = null; // Use this for running our python code
public void init() {
// the first thing we will do is register this module to be invoked
// via the application toolbar
FeatureModuleContext context = getContext();
try {
context.registerFeature(this, "popup",
getIcon("accountlist"),
getName());
this.pi = new PythonInterpreter();
String path = getSourceFile().toString(); // Get the location of this mxt
this.pi.set("jarpath", path);
this.pi.exec("import sys");
this.pi.exec("if jarpath not in sys.path: sys.path.append(jarpath)");
this.pi.exec("from com.infinitekind.moneydance.model import *");
this.pi.exec("import pyextension");
this.pi.exec("reload(pyextension)");
this.pi.set("context", context);
this.pi.set("extensionobject", this);
this.pi.exec("ext = pyextension.SimplePopupExtension()");
this.pi.exec("ext.initialize (context, extensionobject, True)");
}
catch (Exception e) {
e.printStackTrace(System.err);
}
}
public void cleanup() {
closeConsole();
}
private Image getIcon(String action) {
try {
ClassLoader cl = getClass().getClassLoader();
java.io.InputStream in =
cl.getResourceAsStream("/com/moneydance/modules/features/myextension/icon.gif");
if (in != null) {
ByteArrayOutputStream bout = new ByteArrayOutputStream(1000);
byte buf[] = new byte[256];
int n = 0;
while((n=in.read(buf, 0, buf.length))>=0)
bout.write(buf, 0, n);
return Toolkit.getDefaultToolkit().createImage(bout.toByteArray());
}
} catch (Throwable e) { }
return null;
}
/** Process an invokation of this module with the given URI */
public void invoke(String uri) {
String command = uri;
String parameters = "";
int theIdx = uri.indexOf('?');
if(theIdx>=0) {
command = uri.substring(0, theIdx);
parameters = uri.substring(theIdx+1);
}
else {
theIdx = uri.indexOf(':');
if(theIdx>=0) {
command = uri.substring(0, theIdx);
}
}
// Pass through the invocation string to the Python handler
try {
this.pi.set("invoke_string", command);
this.pi.exec("ext.invoke(invoke_string)");
}
catch (Exception e) {
e.printStackTrace(System.err);
}
}
public String getName() {
return "Show popup";
}
FeatureModuleContext getUnprotectedContext() {
return getContext();
}
synchronized void closeConsole() {
System.gc();
}
}
</code></pre>
</noscript>
</div>
<p>Here's what's going on:</p>
<ul>
<li><p class="first">First, to clean up, I'm taking out all references to the AccountList popup. We'll use
our own Python extension from the earlier part of this blog post to replace it.</p>
</li>
<li><p class="first">Then, I'm importing the Jython supplied PythonInterpreter, which is used to instatiate an interpreter
from which we'll call our Python code.</p>
</li>
<li><p class="first">In the init method, I register the invocation string as 'popup', which is the same string used in our
Python extension.</p>
</li>
<li><dl class="first docutils">
<dt>Following that, I do the following on lines 35-46:</dt>
<dd><ol class="first last arabic simple">
<li>Instantiate the interpreter.</li>
<li>Get the path to the extension using the builtin <tt class="docutils literal">getSourceFile</tt>. This is a key step, because we need this .mxt to point
our sys.path at.</li>
<li>Now, set the <tt class="docutils literal">jarpath</tt> variable to this .mxt file, and execute some Python statements to import the Python extension (I'm calling it pyextension.py).</li>
<li>The <tt class="docutils literal">reload</tt> is very important to make sure any changes to your Python extension during deveoplment are updated every time the extension is reinstalled.</li>
<li>I then instantiate the extension and manually instantiate it using the Moneydance context.</li>
</ol>
</dd>
</dl>
</li>
<li><p class="first">Then, in the <tt class="docutils literal">invoke</tt> method on line 75, I merely passthrough whatever invocation command Moneydance sends me (in this case, only <tt class="docutils literal">popup</tt> from the registration
command) to our Python extension.</p>
</li>
</ul>
<p>That's it!</p>
</div>
<div class="section" id="python-extension">
<h3><a class="toc-backref" href="#id10">5.5 Python extension</a></h3>
<p>The Python extension is hardly changed from the original that we discussed above:</p>
<div class="gist">
<script src='https://gist.github.com/4464c7be4dc40cfe66dc3658059cdc31.js?file=pyextension.py'></script>
<noscript>
<pre><code>from com.infinitekind.moneydance.model import *
import os
from javax.swing import JButton, JFrame, JScrollPane, JTextArea, BoxLayout, BorderFactory
class SimplePopupExtension(object):
def initialize(self, extension_context, extension_object, from_java=False):
self.moneydanceContext = extension_context
self.moneydanceExtensionObject = extension_object
# here we register ourselves with a menu item to invoke a feature
# (ignore the button and icon mentions in the docs)
if not from_java:
self.moneydanceContext.registerFeature(extension_object, "popup", None, "Test popup")
# invoke(eventstring) is called when we receive a callback for the feature that
# we registered in the initialize method
def invoke(self, eventString=""):
self.moneydanceContext.setStatus("Python extension received command: %s" % (eventString))
if eventString=='popup':
self._show_popup()
def __str__(self):
return "SimplePopup"
# Our actual user code
def _show_popup(self):
self._initUI()
pass
book = self.moneydanceContext.getCurrentAccountBook()
account_names = []
for acct in AccountUtil.getAccountIterator(book):
account_names.append(acct.getFullAccountName())
self.text_area.text = '\n'.join(account_names)
def _initUI(self):
frame = JFrame('Simple popup', defaultCloseOperation = JFrame.DISPOSE_ON_CLOSE, size=(300,300))
frame.setLayout(BoxLayout(frame.contentPane, BoxLayout.PAGE_AXIS))
# Call-back to close popup
def closeDialog(event):
frame.visible = False
frame.dispose()
# Instantiate components
self.text_area = JTextArea()
msgScroller = JScrollPane()
msgScroller.setBorder(BorderFactory.createTitledBorder("Accounts"))
msgScroller.setViewportView(self.text_area)
self.close_button = JButton('Close', actionPerformed = closeDialog)
# Add components to frame
frame.add(msgScroller)
frame.add(self.close_button)
frame.visible = True
# Tell moneydance this is an extension
moneydance_extension = SimplePopupExtension()
</code></pre>
</noscript>
</div>
<p>The only change is I added a parameter to the <tt class="docutils literal">initialize</tt> method called <em>from_java</em>, which if set to True, will prevent
the Python extension from also registering a menu item. I also changed the reference to get the Context to using a member variable (not sure if this is needed or not, but it seemed cleaner).</p>
<p>And that's it! Now, the nice thing is that you can test the Python extension unchanged through the Python interface in Moneydance, and when you're ready, you can
just compile the java extension for deployment.</p>
<p>Just put this file in <tt class="docutils literal">src/com/moneydance/modules/features/myextension/python</tt> (new python subdirectory).</p>
</div>
<div class="section" id="compiling">
<h3><a class="toc-backref" href="#id11">5.6 Compiling</a></h3>
<p>Just run <tt class="docutils literal">ant myextension</tt> again from your <tt class="docutils literal">src</tt> directory and this should rebuild the extension. And there you go, a Python extension bundled into a .mxt file that can
be installed inside Moneydance permanently!</p>
</div>
</div>
</body></html>Elegant command line parsing in Python using docopt, schema, and yaml2016-06-23T14:42:00-04:00viranthatag:https://virantha.com,2016-06-23:2016/06/23/elegant-command-line-parsing-in-python/<html><body><p>Both for my work and personal use, I tend to develop many command-line Python
scripts that are very data-driven and configurable. In Python, it's pretty
trivial to use argparse to support complex argument and option structures, and
then use the integration with configparser to allow one to write files that
capture specific option/configuration values for posterity. Unfortunately, the
syntax of argparse and configparser tend to be pretty verbose, and incremental
changes after weeks or months long development gaps can be error prone. In addition,
I find the <tt class="docutils literal">ini</tt> syntax of configparser rather limiting.</p>
<p>In the past year, I've switched over completely to using <a class="reference external" href="http://docopt.org">docopt</a>, where you
write the example usage and options in a string inside your script, and it
builds the command-line invocation parser for you automatically. This is an
incredibly powerful and concise way to write a self-documenting option parser.
Then, you combine that with a data validation library like <a class="reference external" href="https://pypi.python.org/pypi/schema">schema</a> to check the
incoming options, and use <a class="reference external" href="http://pyyaml.org">yaml</a> to allow for options to be supplied from a text
file, and you're pretty much in option parsing nirvana.</p>
<p>In this post, I'll go through a script I wrote recently that implements a very
common pattern for anyone in an engineering field:</p>
<ul class="simple">
<li>Take a file that defines an engineering simulation</li>
<li>Additionally, take another input file that specifies a set of parameters that define different simulation scenarios</li>
<li>Generate all possible combinations of parameter values</li>
<li>For each combination, emit the simulation file with the specific parameters</li>
<li>Execute an external simulator on the simulation file</li>
<li>Parse the results and log the results in a CSV file</li>
</ul>
<div class="section" id="running-parametrized-spice-simulations">
<h2><a class="toc-backref" href="#id1">1 Running parametrized SPICE simulations</a></h2>
<p>For the purposes of this example, I'm going to walk-through a simple script that I've
developed to do parameter exploration of a digital circuit using SPICE (hspice).
The input is a simple spice deck that's a <a class="reference external" href="http://www.makotemplates.org">Mako</a> template, and the parameters are
specified in a YAML file. The script does simulation and logging of results as
two separate flow steps (so you could, for example, re-write the results file based
on a previously run set of simulations).</p>
<p>Here's the spice file written as a Mako template. Notice that all the parameters are within the <tt class="docutils literal"><span class="pre">${...}</span></tt> blocks. We're really not
using any of the advanced features of the templating language (like
conditionals or loops, or even arbitrary python), but it's available if you
need to make more complex simulation files.</p>
<div class="highlight"><pre>*
.temp ${temp}
.option brief=1
.lib 'LIBBFILE.l' ${corner}
vvdd vdd 0 ${vdd}
.subckt xinv in out
xmp out in vdd vdd pfet w=20 l=2
xmn out in 0 0 nfet w=10 l=2
.ends xinv
xinv_t qf qt xinv
.meas tran StaticCurrentVdd avg i(vdd) from=1n to=3n
.meas tran StaticSupplyPowerVdd PARAM='-StaticCurrentVdd*${vdd}'
.tran 0.005ns 3ns sweep monte=1000
</pre></div>
<p>So, we have the following list of parameters that need to be supplied to this simulation file:</p>
<ul class="simple">
<li><em>temp</em> - Circuit simulation temperature</li>
<li><em>corner</em> - Device/transistor corner (e.g. typical, fast, slow, etc)</li>
<li><em>vdd</em> - Simulation voltage</li>
</ul>
<p>We'll define these in another file using the YAML syntax like so:</p>
<div class="highlight"><pre><span class="l-Scalar-Plain">temp</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">85</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">125</span>
<span class="l-Scalar-Plain">vdd</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">0.6</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">0.8</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">1.0</span>
<span class="l-Scalar-Plain">corner</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">TT</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">FF</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">SS</span>
</pre></div>
<p>Here, we've defined values for the different parameters, and we'd like to
simulate every single combination of these. So, for example, <tt class="docutils literal">temp=125,
vdd=0.6, corner=SS</tt> would be one possible substitution into the simulation
file.</p>
</div>
<div class="section" id="setting-up-the-options">
<h2><a class="toc-backref" href="#id2">2 Setting up the options</a></h2>
<p>First, let's start by setting up the usage of this script using <a class="reference external" href="http://docopt.org">docopt</a>.</p>
<div class="highlight"><pre><span class="sd">"""Simulate</span>
<span class="sd">Usage:</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK all</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK (sim|log)...</span>
<span class="sd"> simulate.py -h</span>
<span class="sd">Arguments:</span>
<span class="sd"> PARAMFILE YAML file with variables to iterate over</span>
<span class="sd"> SPICEDECK Mako templated spice deck</span>
<span class="sd"> all Run all steps in the flow</span>
<span class="sd"> sim Run the simulations</span>
<span class="sd"> log Just collate the available results into one file</span>
<span class="sd">Options:</span>
<span class="sd"> -h --help show this message</span>
<span class="sd"> -v --verbose show more information</span>
<span class="sd"> --rundir=PATH set path for running simulations in [default: runs]</span>
<span class="sd"> --resultsfile=FILE set filename for writing results to [default: index.txt]</span>
<span class="sd">"""</span>
<span class="kn">from</span> <span class="nn">docopt</span> <span class="kn">import</span> <span class="n">docopt</span>
<span class="k">class</span> <span class="nc">Sim</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">docopt</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">prog</span> <span class="o">=</span> <span class="n">Sim</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">get_options</span><span class="p">()</span>
</pre></div>
<p>Here's the beauty of docopt: the entire options parsing is defined in the string at the start of the file.
If you just type <tt class="docutils literal">simulate.py <span class="pre">-h</span></tt>, you'll get the following output:</p>
<div class="highlight"><pre>Usage:
simulate.py [options] PARAMFILE SPICEDECK all
simulate.py [options] PARAMFILE SPICEDECK (sim|log)...
simulate.py -h
Arguments:
PARAMFILE YAML file with variables to iterate over
SPICEDECK Mako templated spice deck
all Run all steps in the flow
sim Run the simulations
log Just collate the available results into one file
Options:
-h --help show this message
-v --verbose show more information
--rundir=PATH set path for running simulations in [default: runs]
--resultsfile=FILE set filename for writing results to [default: index.txt]
</pre></div>
<p>The script is setup to take in the parameter file (with all the values defined), followed by the simulation spice
template file, followed by the flow step we want to run (either all, or one or more of sim/log). You can also
optionally set the run directory and the name of the resultsfile. Here's the output with a valid command line:</p>
<div class="highlight"><pre><span class="o">[</span>1<span class="o">]</span> virantha@virantha-macbook-243> python code_sim.py params.yml sim.sp all
<span class="o">{</span><span class="s1">'--help'</span>: False,
<span class="s1">'--resultsfile'</span>: <span class="s1">'index.txt'</span>,
<span class="s1">'--rundir'</span>: <span class="s1">'runs'</span>,
<span class="s1">'--verbose'</span>: False,
<span class="s1">'PARAMFILE'</span>: <span class="s1">'params.yml'</span>,
<span class="s1">'SPICEDECK'</span>: <span class="s1">'sim.sp'</span>,
<span class="s1">'all'</span>: True,
<span class="s1">'log'</span>: 0,
<span class="s1">'sim'</span>: 0<span class="o">}</span>
</pre></div>
<p>And all the arguments are parsed into a nice dictionary! One idiosyncracy is that the way we've defined sim/log
as one or more optional keywords, we end up with a counter for each keyword. So, for example, if we specified
<tt class="docutils literal">sim</tt> as a flow step, then <tt class="docutils literal">all: False, sim: 1</tt> would appear in our <tt class="docutils literal">args</tt> dictionary.</p>
</div>
<div class="section" id="validating-options-with-schema">
<h2><a class="toc-backref" href="#id3">3 Validating options with schema</a></h2>
<p>Now, let's do some rudimentary checking of the user supplied options. For example, it would be nice
to make sure that the parameter and spice template file actually exist and are readable, so let's add those
checks using <a class="reference external" href="https://pypi.python.org/pypi/schema">schema</a> to validate our dictionary:</p>
<div class="highlight"><pre><span class="sd">"""Simulate</span>
<span class="sd">Usage:</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK all</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK (sim|log)...</span>
<span class="sd"> simulate.py -h</span>
<span class="sd">Arguments:</span>
<span class="sd"> PARAMFILE YAML file with variables to iterate over</span>
<span class="sd"> SPICEDECK Mako templated spice deck</span>
<span class="sd"> all Run all steps in the flow</span>
<span class="sd"> sim Run the simulations</span>
<span class="sd"> log Just collate the available results into one file</span>
<span class="sd">Options:</span>
<span class="sd"> -h --help show this message</span>
<span class="sd"> -v --verbose show more information</span>
<span class="sd"> --rundir=PATH set path for running simulations in [default: runs]</span>
<span class="sd"> --resultsfile=FILE set filename for writing results to [default: index.txt]</span>
<span class="sd">"""</span>
<span class="kn">from</span> <span class="nn">docopt</span> <span class="kn">import</span> <span class="n">docopt</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">schema</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">And</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Or</span><span class="p">,</span> <span class="n">Use</span><span class="p">,</span> <span class="n">SchemaError</span>
<span class="k">class</span> <span class="nc">Sim</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">docopt</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">({</span>
<span class="s">'PARAMFILE'</span><span class="p">:</span> <span class="n">Use</span><span class="p">(</span><span class="nb">open</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="s">'PARAMFILE should be readable'</span><span class="p">),</span>
<span class="s">'SPICEDECK'</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">,</span>
<span class="nb">object</span><span class="p">:</span> <span class="nb">object</span>
<span class="p">})</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">validate</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="k">except</span> <span class="n">SchemaError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">exit</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">prog</span> <span class="o">=</span> <span class="n">Sim</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">get_options</span><span class="p">()</span>
</pre></div>
<p>We've made sure <tt class="docutils literal">PARAMFILE</tt> is a file and we've opened and converted it into a file handle, since we'll
be passing that into the yaml loader in the next step. Next, we've just checked that the spice simulation
file exists; no need to open it since we'll just be passing that off to the Mako template engine in a later
step.</p>
<p>Very simple, and you can read on the schema docs to add more complex checking of your options.</p>
</div>
<div class="section" id="using-a-configuration-file-to-supply-options">
<h2><a class="toc-backref" href="#id4">4 Using a configuration file to supply options</a></h2>
<p>Now, let's add in configparser type functionality, except we'll use the yaml syntax for more flexibility.</p>
<div class="highlight"><pre><span class="sd">"""Simulate</span>
<span class="sd">Usage:</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK all</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK (sim|log)...</span>
<span class="sd"> simulate.py --conf=FILE</span>
<span class="sd"> simulate.py -h</span>
<span class="sd">Arguments:</span>
<span class="sd"> PARAMFILE YAML file with variables to iterate over</span>
<span class="sd"> SPICEDECK Mako templated spice deck</span>
<span class="sd"> all Run all steps in the flow</span>
<span class="sd"> sim Run the simulations</span>
<span class="sd"> log Just collate the available results into one file</span>
<span class="sd">Options:</span>
<span class="sd"> -h --help show this message</span>
<span class="sd"> -v --verbose show more information</span>
<span class="sd"> --rundir=PATH set path for running simulations in [default: runs]</span>
<span class="sd"> --resultsfile=FILE set filename for writing results to [default: index.txt]</span>
<span class="sd"> --conf=FILE load options from file</span>
<span class="sd">"""</span>
<span class="kn">from</span> <span class="nn">docopt</span> <span class="kn">import</span> <span class="n">docopt</span>
<span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">schema</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">And</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Or</span><span class="p">,</span> <span class="n">Use</span><span class="p">,</span> <span class="n">SchemaError</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="k">class</span> <span class="nc">Sim</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">merge_args</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conf_args</span><span class="p">,</span> <span class="n">orig_args</span><span class="p">):</span>
<span class="sd">""" Return new dict with args, and then conf_args merged in.</span>
<span class="sd"> Make sure that any keys in conf_args are also present in args</span>
<span class="sd"> """</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">conf_args</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">orig_args</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"ERROR: Configuration file has unknown option </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">k</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">orig_args</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">conf_args</span><span class="p">)</span>
<span class="k">return</span> <span class="n">args</span>
<span class="k">def</span> <span class="nf">get_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">docopt</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">]:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">])</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">args</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">merge_args</span><span class="p">(</span><span class="n">conf_args</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
<span class="k">print</span> <span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">({</span>
<span class="s">'PARAMFILE'</span><span class="p">:</span> <span class="n">Use</span><span class="p">(</span><span class="nb">open</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="s">'PARAMFILE should be readable'</span><span class="p">),</span>
<span class="s">'SPICEDECK'</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">,</span>
<span class="nb">object</span><span class="p">:</span> <span class="nb">object</span>
<span class="p">})</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">validate</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="k">except</span> <span class="n">SchemaError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">exit</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">prog</span> <span class="o">=</span> <span class="n">Sim</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">get_options</span><span class="p">()</span>
</pre></div>
<p>Now, we can supply options from a configuration file (which takes priority) in YAML format like so:</p>
<div class="highlight"><pre><span class="l-Scalar-Plain">--verbose</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">True</span>
<span class="l-Scalar-Plain">SPICEDECK</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">sim2.sp</span>
</pre></div>
<p>Let's call this file <tt class="docutils literal">conf.yml</tt>, and now we'll get the following:</p>
<div class="highlight"><pre><span class="o">[</span>1<span class="o">]</span> virantha@virantha-macbook-243> python simulate.py params.yml sim.sp all --conf<span class="o">=</span>conf.yml
<span class="o">{</span><span class="s1">'--help'</span>: False,
<span class="s1">'--resultsfile'</span>: <span class="s1">'index.txt'</span>,
<span class="s1">'--rundir'</span>: <span class="s1">'runs'</span>,
<span class="s1">'--verbose'</span>: True,
<span class="s1">'PARAMFILE'</span>: <span class="s1">'params.yml'</span>,
<span class="s1">'SPICEDECK'</span>: <span class="s1">'sim2.sp'</span>,
<span class="s1">'all'</span>: True,
<span class="s1">'log'</span>: 0,
<span class="s1">'sim'</span>: 0<span class="o">}</span>
</pre></div>
<p>Note that the configuration file values take precedence over the command line (you could see this <a class="reference external" href="https://github.com/docopt/docopt/blob/master/examples/config_file_example.py">example</a>
to make the command-line take precedence instead). We've also put in a check to catch any typos in the conf
file by erroring out if an option not specified in the docopt is found.</p>
</div>
<div class="section" id="loading-in-the-parameters">
<h2><a class="toc-backref" href="#id5">5 Loading in the parameters</a></h2>
<p>Next, let's add in some more option processing and load in the parameters for each simulation scenario from
the <tt class="docutils literal">PARAMFILE</tt> argument.</p>
<div class="highlight"><pre><span class="sd">"""Simulate</span>
<span class="sd">Usage:</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK all</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK (sim|log)...</span>
<span class="sd"> simulate.py --conf=FILE</span>
<span class="sd"> simulate.py -h</span>
<span class="sd">Arguments:</span>
<span class="sd"> PARAMFILE YAML file with variables to iterate over</span>
<span class="sd"> SPICEDECK Mako templated spice deck</span>
<span class="sd"> all Run all steps in the flow</span>
<span class="sd"> sim Run the simulations</span>
<span class="sd"> log Just collate the available results into one file</span>
<span class="sd">Options:</span>
<span class="sd"> -h --help show this message</span>
<span class="sd"> -v --verbose show more information</span>
<span class="sd"> --rundir=PATH set path for running simulations in [default: runs]</span>
<span class="sd"> --resultsfile=FILE set filename for writing results to [default: index.txt]</span>
<span class="sd"> --conf=FILE load options from file</span>
<span class="sd">"""</span>
<span class="kn">from</span> <span class="nn">docopt</span> <span class="kn">import</span> <span class="n">docopt</span>
<span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">schema</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">And</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Or</span><span class="p">,</span> <span class="n">Use</span><span class="p">,</span> <span class="n">SchemaError</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="k">def</span> <span class="nf">ordered_load</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">Loader</span><span class="o">=</span><span class="n">yaml</span><span class="o">.</span><span class="n">Loader</span><span class="p">,</span> <span class="n">object_pairs_hook</span><span class="o">=</span><span class="n">OrderedDict</span><span class="p">):</span>
<span class="sd">""" Helper function to allow yaml load routine to use an OrderedDict instead of regular dict.</span>
<span class="sd"> This helps keeps things sane when ordering the runs and printing out routines</span>
<span class="sd"> """</span>
<span class="k">class</span> <span class="nc">OrderedLoader</span><span class="p">(</span><span class="n">Loader</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">construct_mapping</span><span class="p">(</span><span class="n">loader</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="n">loader</span><span class="o">.</span><span class="n">flatten_mapping</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="k">return</span> <span class="n">object_pairs_hook</span><span class="p">(</span><span class="n">loader</span><span class="o">.</span><span class="n">construct_pairs</span><span class="p">(</span><span class="n">node</span><span class="p">))</span>
<span class="n">OrderedLoader</span><span class="o">.</span><span class="n">add_constructor</span><span class="p">(</span>
<span class="n">yaml</span><span class="o">.</span><span class="n">resolver</span><span class="o">.</span><span class="n">BaseResolver</span><span class="o">.</span><span class="n">DEFAULT_MAPPING_TAG</span><span class="p">,</span>
<span class="n">construct_mapping</span><span class="p">)</span>
<span class="k">return</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">OrderedLoader</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Sim</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">merge_args</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conf_args</span><span class="p">,</span> <span class="n">orig_args</span><span class="p">):</span>
<span class="sd">""" Return new dict with args, and then conf_args merged in.</span>
<span class="sd"> Make sure that any keys in conf_args are also present in args</span>
<span class="sd"> """</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">conf_args</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">orig_args</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"ERROR: Configuration file has unknown option </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">k</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">orig_args</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">conf_args</span><span class="p">)</span>
<span class="k">return</span> <span class="n">args</span>
<span class="k">def</span> <span class="nf">get_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">docopt</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">]:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">])</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">args</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">merge_args</span><span class="p">(</span><span class="n">conf_args</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
<span class="k">print</span> <span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">({</span>
<span class="s">'PARAMFILE'</span><span class="p">:</span> <span class="n">Use</span><span class="p">(</span><span class="nb">open</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="s">'PARAMFILE should be readable'</span><span class="p">),</span>
<span class="s">'SPICEDECK'</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">,</span>
<span class="nb">object</span><span class="p">:</span> <span class="nb">object</span>
<span class="p">})</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">validate</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="k">except</span> <span class="n">SchemaError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">exit</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">flow</span> <span class="o">=</span> <span class="p">[</span><span class="s">'sim'</span><span class="p">,</span> <span class="s">'log'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'all'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'sim'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s">'sim'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'log'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s">'log'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">ordered_load</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'PARAMFILE'</span><span class="p">])</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_dir</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'--rundir'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">results_file</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'--resultsfile'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spice_filename</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'SPICEDECK'</span><span class="p">]</span>
<span class="k">print</span> <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">prog</span> <span class="o">=</span> <span class="n">Sim</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">get_options</span><span class="p">()</span>
</pre></div>
<p>Notice that we've defined a <a class="reference external" href="http://stackoverflow.com/questions/5121931/in-python-how-can-you-load-yaml-mappings-as-ordereddicts">custom loader</a> for reading in the parameter yaml, so that we can keep everything
in an OrderedDict that keeps the same order as present in the YAML file. While not strictly necessary,
this makes the simulation order predictable for the user.</p>
</div>
<div class="section" id="generating-and-running-the-scenarios">
<h2><a class="toc-backref" href="#id6">6 Generating and running the scenarios</a></h2>
<p>Now, we'll introduce a generator function that will yield scenarios based on the parameter values. This
funtion, <tt class="docutils literal">_get_spice_run</tt> uses Python's built-in itertools product function to generate the cross-product
of all the parameter values. Each combination of parameter values is then provided as a <em>context</em> to
Mako to substitute into the template simulation file (spice deck). The <tt class="docutils literal">run</tt> method just iterates over
this generator function, and calls the <tt class="docutils literal">run_sim</tt> method to run the simulator on each scenario.</p>
<div class="highlight"><pre><span class="sd">"""Simulate</span>
<span class="sd">Usage:</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK all</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK (sim|log)...</span>
<span class="sd"> simulate.py --conf=FILE</span>
<span class="sd"> simulate.py -h</span>
<span class="sd">Arguments:</span>
<span class="sd"> PARAMFILE YAML file with variables to iterate over</span>
<span class="sd"> SPICEDECK Mako templated spice deck</span>
<span class="sd"> all Run all steps in the flow</span>
<span class="sd"> sim Run the simulations</span>
<span class="sd"> log Just collate the available results into one file</span>
<span class="sd">Options:</span>
<span class="sd"> -h --help show this message</span>
<span class="sd"> -v --verbose show more information</span>
<span class="sd"> --rundir=PATH set path for running simulations in [default: runs]</span>
<span class="sd"> --resultsfile=FILE set filename for writing results to [default: index.txt]</span>
<span class="sd"> --conf=FILE load options from file</span>
<span class="sd">"""</span>
<span class="kn">from</span> <span class="nn">docopt</span> <span class="kn">import</span> <span class="n">docopt</span>
<span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">subprocess</span>
<span class="kn">from</span> <span class="nn">schema</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">And</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Or</span><span class="p">,</span> <span class="n">Use</span><span class="p">,</span> <span class="n">SchemaError</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="kn">from</span> <span class="nn">mako.template</span> <span class="kn">import</span> <span class="n">Template</span>
<span class="kn">from</span> <span class="nn">mako.lookup</span> <span class="kn">import</span> <span class="n">TemplateLookup</span>
<span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">product</span>
<span class="k">def</span> <span class="nf">ordered_load</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">Loader</span><span class="o">=</span><span class="n">yaml</span><span class="o">.</span><span class="n">Loader</span><span class="p">,</span> <span class="n">object_pairs_hook</span><span class="o">=</span><span class="n">OrderedDict</span><span class="p">):</span>
<span class="sd">""" Helper function to allow yaml load routine to use an OrderedDict instead of regular dict.</span>
<span class="sd"> This helps keeps things sane when ordering the runs and printing out routines</span>
<span class="sd"> """</span>
<span class="k">class</span> <span class="nc">OrderedLoader</span><span class="p">(</span><span class="n">Loader</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">construct_mapping</span><span class="p">(</span><span class="n">loader</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="n">loader</span><span class="o">.</span><span class="n">flatten_mapping</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="k">return</span> <span class="n">object_pairs_hook</span><span class="p">(</span><span class="n">loader</span><span class="o">.</span><span class="n">construct_pairs</span><span class="p">(</span><span class="n">node</span><span class="p">))</span>
<span class="n">OrderedLoader</span><span class="o">.</span><span class="n">add_constructor</span><span class="p">(</span>
<span class="n">yaml</span><span class="o">.</span><span class="n">resolver</span><span class="o">.</span><span class="n">BaseResolver</span><span class="o">.</span><span class="n">DEFAULT_MAPPING_TAG</span><span class="p">,</span>
<span class="n">construct_mapping</span><span class="p">)</span>
<span class="k">return</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">OrderedLoader</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Sim</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">merge_args</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conf_args</span><span class="p">,</span> <span class="n">orig_args</span><span class="p">):</span>
<span class="sd">""" Return new dict with args, and then conf_args merged in.</span>
<span class="sd"> Make sure that any keys in conf_args are also present in args</span>
<span class="sd"> """</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">conf_args</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">orig_args</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"ERROR: Configuration file has unknown option </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">k</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">orig_args</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">conf_args</span><span class="p">)</span>
<span class="k">return</span> <span class="n">args</span>
<span class="k">def</span> <span class="nf">_get_spice_run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">mylookup</span> <span class="o">=</span> <span class="n">TemplateLookup</span><span class="p">(</span><span class="n">directories</span><span class="o">=</span><span class="p">[</span><span class="s">'.'</span><span class="p">])</span>
<span class="n">my_template</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="n">filename</span><span class="p">,</span> <span class="n">lookup</span><span class="o">=</span><span class="n">mylookup</span><span class="p">)</span>
<span class="n">k</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
<span class="c"># For each cross-product of all the parameter values</span>
<span class="k">for</span> <span class="n">params</span> <span class="ow">in</span> <span class="n">product</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
<span class="c"># Generate a dict of param name to value for the context of the template</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="n">k</span><span class="p">,</span><span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span><span class="n">v</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">keys</span><span class="p">(),</span> <span class="n">params</span><span class="p">)])</span>
<span class="k">yield</span> <span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">my_template</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="o">**</span><span class="n">context</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">get_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">docopt</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">]:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">])</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">args</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">merge_args</span><span class="p">(</span><span class="n">conf_args</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
<span class="k">print</span> <span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">({</span>
<span class="s">'PARAMFILE'</span><span class="p">:</span> <span class="n">Use</span><span class="p">(</span><span class="nb">open</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="s">'PARAMFILE should be readable'</span><span class="p">),</span>
<span class="s">'SPICEDECK'</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">,</span>
<span class="nb">object</span><span class="p">:</span> <span class="nb">object</span>
<span class="p">})</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">validate</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="k">except</span> <span class="n">SchemaError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">exit</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">flow</span> <span class="o">=</span> <span class="p">[</span><span class="s">'sim'</span><span class="p">,</span> <span class="s">'log'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'all'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'sim'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s">'sim'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'log'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s">'log'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">ordered_load</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'PARAMFILE'</span><span class="p">])</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_dir</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'--rundir'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">results_file</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'--resultsfile'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spice_filename</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'SPICEDECK'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">_make_dirs</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">d</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">d</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run_sim</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">parent_dir</span><span class="p">,</span> <span class="n">spice_deck</span><span class="p">):</span>
<span class="n">cwd</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span>
<span class="n">run_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">parent_dir</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">index</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_make_dirs</span><span class="p">(</span><span class="n">run_dir</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">run_dir</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'run.sp'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">spice_deck</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">([</span><span class="s">'hspice'</span><span class="p">,</span> <span class="s">'run.sp'</span><span class="p">])</span>
<span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"WARNING: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">e</span><span class="p">)</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">output</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">cwd</span><span class="p">)</span>
<span class="k">return</span> <span class="n">output</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">run_dir</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_dir</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_make_dirs</span><span class="p">(</span><span class="n">run_dir</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="p">(</span><span class="n">context</span><span class="p">,</span><span class="n">spice_deck</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_get_spice_run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">spice_filename</span><span class="p">)):</span>
<span class="k">print</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">values</span><span class="p">())</span>
<span class="k">if</span> <span class="s">'sim'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="p">:</span>
<span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_sim</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">run_dir</span><span class="p">,</span> <span class="n">spice_deck</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">prog</span> <span class="o">=</span> <span class="n">Sim</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">get_options</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
</div>
<div class="section" id="logging-the-results-and-complete-script">
<h2><a class="toc-backref" href="#id7">7 Logging the results and complete script</a></h2>
<p>Now, we just introduce some results parsing and outputting to a CSV file for the <tt class="docutils literal">log</tt> flow step, which
gives us our complete script. Notice that the results CSV output is completely data-driven and based on the
context dictionary provided for each scenario.</p>
<div class="highlight"><pre><span class="sd">"""Simulate</span>
<span class="sd">Usage:</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK all</span>
<span class="sd"> simulate.py [options] PARAMFILE SPICEDECK (sim|log)...</span>
<span class="sd"> simulate.py --conf=FILE</span>
<span class="sd"> simulate.py -h</span>
<span class="sd">Arguments:</span>
<span class="sd"> PARAMFILE YAML file with variables to iterate over</span>
<span class="sd"> SPICEDECK Mako templated spice deck</span>
<span class="sd"> all Run all steps in the flow</span>
<span class="sd"> sim Run the simulations</span>
<span class="sd"> log Just collate the available results into one file</span>
<span class="sd">Options:</span>
<span class="sd"> -h --help show this message</span>
<span class="sd"> -v --verbose show more information</span>
<span class="sd"> --rundir=PATH set path for running simulations in [default: runs]</span>
<span class="sd"> --resultsfile=FILE set filename for writing results to [default: index.txt]</span>
<span class="sd"> --conf=FILE load options from file</span>
<span class="sd">"""</span>
<span class="kn">from</span> <span class="nn">docopt</span> <span class="kn">import</span> <span class="n">docopt</span>
<span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">subprocess</span>
<span class="kn">from</span> <span class="nn">schema</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">And</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Or</span><span class="p">,</span> <span class="n">Use</span><span class="p">,</span> <span class="n">SchemaError</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="kn">from</span> <span class="nn">mako.template</span> <span class="kn">import</span> <span class="n">Template</span>
<span class="kn">from</span> <span class="nn">mako.lookup</span> <span class="kn">import</span> <span class="n">TemplateLookup</span>
<span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">product</span>
<span class="k">def</span> <span class="nf">ordered_load</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">Loader</span><span class="o">=</span><span class="n">yaml</span><span class="o">.</span><span class="n">Loader</span><span class="p">,</span> <span class="n">object_pairs_hook</span><span class="o">=</span><span class="n">OrderedDict</span><span class="p">):</span>
<span class="sd">""" Helper function to allow yaml load routine to use an OrderedDict instead of regular dict.</span>
<span class="sd"> This helps keeps things sane when ordering the runs and printing out routines</span>
<span class="sd"> """</span>
<span class="k">class</span> <span class="nc">OrderedLoader</span><span class="p">(</span><span class="n">Loader</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">construct_mapping</span><span class="p">(</span><span class="n">loader</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="n">loader</span><span class="o">.</span><span class="n">flatten_mapping</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="k">return</span> <span class="n">object_pairs_hook</span><span class="p">(</span><span class="n">loader</span><span class="o">.</span><span class="n">construct_pairs</span><span class="p">(</span><span class="n">node</span><span class="p">))</span>
<span class="n">OrderedLoader</span><span class="o">.</span><span class="n">add_constructor</span><span class="p">(</span>
<span class="n">yaml</span><span class="o">.</span><span class="n">resolver</span><span class="o">.</span><span class="n">BaseResolver</span><span class="o">.</span><span class="n">DEFAULT_MAPPING_TAG</span><span class="p">,</span>
<span class="n">construct_mapping</span><span class="p">)</span>
<span class="k">return</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">OrderedLoader</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Sim</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">merge_args</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conf_args</span><span class="p">,</span> <span class="n">orig_args</span><span class="p">):</span>
<span class="sd">""" Return new dict with args, and then conf_args merged in.</span>
<span class="sd"> Make sure that any keys in conf_args are also present in args</span>
<span class="sd"> """</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">conf_args</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">orig_args</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"ERROR: Configuration file has unknown option </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">k</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">orig_args</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">conf_args</span><span class="p">)</span>
<span class="k">return</span> <span class="n">args</span>
<span class="k">def</span> <span class="nf">_get_spice_run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">mylookup</span> <span class="o">=</span> <span class="n">TemplateLookup</span><span class="p">(</span><span class="n">directories</span><span class="o">=</span><span class="p">[</span><span class="s">'.'</span><span class="p">])</span>
<span class="n">my_template</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="n">filename</span><span class="p">,</span> <span class="n">lookup</span><span class="o">=</span><span class="n">mylookup</span><span class="p">)</span>
<span class="n">k</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
<span class="c"># For each cross-product of all the parameter values</span>
<span class="k">for</span> <span class="n">params</span> <span class="ow">in</span> <span class="n">product</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
<span class="c"># Generate a dict of param name to value for the context of the template</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="n">k</span><span class="p">,</span><span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span><span class="n">v</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">keys</span><span class="p">(),</span> <span class="n">params</span><span class="p">)])</span>
<span class="k">yield</span> <span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">my_template</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="o">**</span><span class="n">context</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">get_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">docopt</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">]:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'--conf'</span><span class="p">])</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conf_args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">args</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">merge_args</span><span class="p">(</span><span class="n">conf_args</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
<span class="k">print</span> <span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">({</span>
<span class="s">'PARAMFILE'</span><span class="p">:</span> <span class="n">Use</span><span class="p">(</span><span class="nb">open</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="s">'PARAMFILE should be readable'</span><span class="p">),</span>
<span class="s">'SPICEDECK'</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">,</span>
<span class="nb">object</span><span class="p">:</span> <span class="nb">object</span>
<span class="p">})</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">validate</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="k">except</span> <span class="n">SchemaError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">exit</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">flow</span> <span class="o">=</span> <span class="p">[</span><span class="s">'sim'</span><span class="p">,</span> <span class="s">'log'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'all'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'sim'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s">'sim'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="s">'log'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s">'log'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">ordered_load</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="s">'PARAMFILE'</span><span class="p">])</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_dir</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'--rundir'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">results_file</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'--resultsfile'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spice_filename</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="s">'SPICEDECK'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">_make_dirs</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">d</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">d</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run_sim</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">parent_dir</span><span class="p">,</span> <span class="n">spice_deck</span><span class="p">):</span>
<span class="n">cwd</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span>
<span class="n">run_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">parent_dir</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">index</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_make_dirs</span><span class="p">(</span><span class="n">run_dir</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">run_dir</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'run.sp'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">spice_deck</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">([</span><span class="s">'hspice'</span><span class="p">,</span> <span class="s">'run.sp'</span><span class="p">])</span>
<span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"WARNING: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">e</span><span class="p">)</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">output</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">cwd</span><span class="p">)</span>
<span class="k">return</span> <span class="n">output</span>
<span class="k">def</span> <span class="nf">get_log</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">parent_dir</span><span class="p">):</span>
<span class="n">run_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">parent_dir</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">index</span><span class="p">))</span>
<span class="n">output_file</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">run_dir</span><span class="p">,</span> <span class="s">'run.mpp0'</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">output_file</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">results_text</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">readlines</span><span class="p">()</span>
<span class="k">return</span> <span class="n">results_text</span>
<span class="k">def</span> <span class="nf">get_results</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">results_text</span><span class="p">):</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">results_text</span><span class="p">:</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'staticsupplypowervdd'</span><span class="p">):</span>
<span class="n">splits</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="n">res</span><span class="p">[</span><span class="s">'mean'</span><span class="p">]</span> <span class="o">=</span> <span class="n">splits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">res</span><span class="p">[</span><span class="s">'median'</span><span class="p">]</span> <span class="o">=</span> <span class="n">splits</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">res</span><span class="p">[</span><span class="s">'stdev'</span><span class="p">]</span> <span class="o">=</span> <span class="n">splits</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">run_dir</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_dir</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_make_dirs</span><span class="p">(</span><span class="n">run_dir</span><span class="p">)</span>
<span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">run_dir</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">results_file</span><span class="p">),</span> <span class="s">'w'</span><span class="p">)</span>
<span class="k">print</span> <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
<span class="k">print</span> <span class="o">>></span> <span class="n">f</span><span class="p">,</span> <span class="s">','</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="s">'Run'</span><span class="p">,</span> <span class="s">'Mean'</span><span class="p">,</span> <span class="s">'Median'</span><span class="p">,</span> <span class="s">'Stdev'</span><span class="p">]</span><span class="o">+</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="p">(</span><span class="n">context</span><span class="p">,</span><span class="n">spice_deck</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_get_spice_run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">spice_filename</span><span class="p">)):</span>
<span class="k">print</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">values</span><span class="p">())</span>
<span class="k">if</span> <span class="s">'sim'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="p">:</span>
<span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_sim</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">run_dir</span><span class="p">,</span> <span class="n">spice_deck</span><span class="p">)</span>
<span class="k">if</span> <span class="s">'log'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">flow</span><span class="p">:</span>
<span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_log</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">run_dir</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_results</span><span class="p">(</span><span class="n">output</span><span class="p">)</span>
<span class="k">print</span> <span class="o">>></span> <span class="n">f</span><span class="p">,</span> <span class="s">','</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="nb">str</span><span class="p">(</span><span class="n">i</span><span class="p">)]</span><span class="o">+</span><span class="n">results</span><span class="o">.</span><span class="n">values</span><span class="p">()</span><span class="o">+</span><span class="p">[</span><span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">context</span><span class="o">.</span><span class="n">values</span><span class="p">()])</span>
<span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="k">print</span> <span class="n">results</span>
<span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">prog</span> <span class="o">=</span> <span class="n">Sim</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">get_options</span><span class="p">()</span>
<span class="n">prog</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
</div>
</body></html>[UPDATED] Reverse engineering the remote control protocol for the Nissan LEAF (the new Nissan EV protocol)2016-04-07T11:00:00-04:00viranthatag:https://virantha.com,2016-04-07:2016/04/07/updated-reverse-engineering-nissan-connect-ev-protocol/<html><body><div class="note">
<p class="first admonition-title">Note</p>
<p class="last">This is an update to my <a class="reference external" href="https://virantha.com/2016/01/12/reverse-engineering-nissan-connect-ev-protocol/">earlier post on the Nissan Leaf remote telematics</a>. Nissan
had taken their API offline in February 2016 to address <a class="reference external" href="http://www.troyhunt.com/2016/02/controlling-vehicle-features-of-nissan.html?m=1">the security
vulnerabilities</a>,
and this post details how to connect to the new API that went into place in
early April.</p>
</div>
<div class="section" id="nissan-connect-ev">
<h2>1 Nissan Connect EV</h2>
<p>Most of this updated login information is courtesy of users on the MyNissanLeaf portal and specifically <a class="reference external" href="http://www.mynissanleaf.com/viewtopic.php?f=31&t=2214&start=110#p457263">this post</a>
The new endpoints are now rooted at '<a class="reference external" href="https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/">https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/</a>' instead of '<a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/</a>', and all accesses are now
POST requests.</p>
<div class="section" id="initial-login-to-nissan-servers">
<h3>1.1 Initial login to Nissan servers</h3>
<p>The initial login is now a two step process.</p>
<div class="section" id="request-an-encryption-key">
<h4>1.1.1 Request an encryption key</h4>
<p>First, you need to call InitialApp.php with the parameter <tt class="docutils literal">initial_app_string</tt> set to <tt class="docutils literal">geORNtsZe5I4lRGjG9GZiA</tt> (taken from analyzing the iOS app traffic).</p>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&initial_app_strings=geORNtsZe5I4lRGjG9GZiA"</span> <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/InitialApp.php"</span></tt></blockquote>
<p>This returns an encryption key in the parameter <tt class="docutils literal">baseprm</tt></p>
<div class="highlight"><pre><span class="p">{</span><span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span><span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span><span class="nt">"baseprm"</span><span class="p">:</span><span class="s2">"uyI5Dj9g8VCOFDnBRUbr3g"</span><span class="p">}</span>
</pre></div>
</div>
<div class="section" id="initial-logon-request">
<h4>1.1.2 Initial logon request</h4>
<p>Once you have the encryption key, you need to use it to encrypt the user's NissanConnect password using Blowfish ECB with PKCS5Padding. The encrypted key
is then Base64 encoded, and then also urlencoded before being used in the login API request. Here's some sample Python code to show what needs to be done, assuming
you have the pycrypto library installed.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">Crypto.Cipher</span> <span class="kn">import</span> <span class="n">Blowfish</span>
<span class="kn">import</span> <span class="nn">base64</span>
<span class="k">def</span> <span class="nf">pad_string</span><span class="p">(</span><span class="nb">str</span><span class="p">):</span>
<span class="n">new_str</span> <span class="o">=</span> <span class="nb">str</span>
<span class="n">pad_chars</span> <span class="o">=</span> <span class="mi">8</span><span class="o">-</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span> <span class="o">%</span> <span class="mi">8</span><span class="p">)</span>
<span class="k">if</span> <span class="n">pad_chars</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">pad_chars</span><span class="p">):</span>
<span class="n">new_str</span> <span class="o">+=</span> <span class="nb">chr</span><span class="p">(</span><span class="n">pad_chars</span><span class="p">)</span>
<span class="k">return</span> <span class="n">new_str</span>
<span class="n">plaintext</span> <span class="o">=</span> <span class="s">"YOURPASSWORD"</span>
<span class="n">crypt_obj</span> <span class="o">=</span> <span class="n">Blowfish</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s">'uyI5Dj9g8VCOFDnBRUbr3g'</span><span class="p">,</span> <span class="n">Blowfish</span><span class="o">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="n">crypt_obj</span><span class="o">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">pad_string</span><span class="p">(</span><span class="n">plaintext</span><span class="p">))</span>
<span class="k">print</span> <span class="s">"Plaintext: "</span> <span class="o">+</span> <span class="n">plaintext</span>
<span class="k">print</span> <span class="s">"Blowfish Cyphertext: "</span> <span class="o">+</span> <span class="n">ciphertext</span>
<span class="k">print</span> <span class="s">"Blowfish base64: "</span> <span class="o">+</span> <span class="n">base64</span><span class="o">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">)</span>
</pre></div>
<p>The base64 encoded string can then be used in the following API call:</p>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&initial_app_strings=geORNtsZe5I4lRGjG9GZiA&UserId=YOUREMAIL"</span> <span class="pre">--data-urlencode</span> "Password=YOURBASE64PASSWORD" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/UserLoginRequest.php"</span></tt></blockquote>
<p>If successful, you will get back the following JSON, where <tt class="docutils literal">custom_sessionid</tt> is what you'll be using for subsequent requests.</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="nt">"sessionId"</span><span class="p">:</span> <span class="s2">"XXXXXXXX"</span><span class="p">,</span>
<span class="nt">"VehicleInfoList"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"VehicleInfo"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"charger20066"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
<span class="nt">"nickname"</span><span class="p">:</span> <span class="s2">"VLeaf"</span><span class="p">,</span>
<span class="nt">"telematicsEnabled"</span><span class="p">:</span> <span class="s2">"true"</span><span class="p">,</span>
<span class="nt">"vin"</span><span class="p">:</span> <span class="s2">"XXXX"</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="nt">"vehicleInfo"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"charger20066"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
<span class="nt">"nickname"</span><span class="p">:</span> <span class="s2">"VLeaf"</span><span class="p">,</span>
<span class="nt">"telematicsEnabled"</span><span class="p">:</span> <span class="s2">"true"</span><span class="p">,</span>
<span class="nt">"vin"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"custom_sessionid"</span><span class="p">:</span> <span class="s2">"XXXX"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="nt">"vehicle"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"profile"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"vin"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"gdcUserId"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"gdcPassword"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"encAuthToken"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"dcmId"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"nickname"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="s2">"ACCEPTED"</span><span class="p">,</span>
<span class="nt">"statusDate"</span><span class="p">:</span> <span class="s2">"Sep 24, 2015 12:00 AM"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"EncAuthToken"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"CustomerInfo"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"UserId"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"Language"</span><span class="p">:</span> <span class="s2">"en-US"</span><span class="p">,</span>
<span class="nt">"Timezone"</span><span class="p">:</span> <span class="s2">"America/New_York"</span><span class="p">,</span>
<span class="nt">"RegionCode"</span><span class="p">:</span> <span class="s2">"NNA"</span><span class="p">,</span>
<span class="nt">"OwnerId"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"Nickname"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"Country"</span><span class="p">:</span> <span class="s2">"US"</span><span class="p">,</span>
<span class="nt">"VehicleImage"</span><span class="p">:</span> <span class="s2">"/content/language/default/images/img/ph_car.jpg"</span><span class="p">,</span>
<span class="nt">"UserVehicleBoundDurationSec"</span><span class="p">:</span> <span class="s2">"946771200"</span><span class="p">,</span>
<span class="nt">"VehicleInfo"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"VIN"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"DCMID"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"SIMID"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"NAVIID"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"EncryptedNAVIID"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"MSN"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"LastVehicleLoginTime"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"UserVehicleBoundTime"</span><span class="p">:</span> <span class="s2">"2015-09-24T18:10:06Z"</span><span class="p">,</span>
<span class="nt">"LastDCMUseTime"</span><span class="p">:</span> <span class="s2">"Apr 7, 2016 12:27 PM"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"UserInfoRevisionNo"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="section" id="request-current-status">
<h3>1.2 Request current status</h3>
<p>Request the current battery status (not necessarily the latest, but whatever is cached from the last update on Nissan's server), using the urlencoded <tt class="docutils literal">custom_session_id</tt></p>
<blockquote>
curl --data "RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York" --data-urlencode "custom_sessionid=XXXXXXXXXX" -X POST "<a class="reference external" href="https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/BatteryStatusRecordsRequest.php">https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/BatteryStatusRecordsRequest.php</a>"</blockquote>
<p>Response:</p>
<div class="highlight"><pre><span class="s2">"status"</span><span class="err">:</span> <span class="mi">200</span><span class="err">,</span>
<span class="s2">"BatteryStatusRecords"</span><span class="err">:</span> <span class="p">{</span>
<span class="nt">"OperationResult"</span><span class="p">:</span> <span class="s2">"START"</span><span class="p">,</span>
<span class="nt">"OperationDateAndTime"</span><span class="p">:</span> <span class="s2">"Apr 6, 2016 05:57 PM"</span><span class="p">,</span>
<span class="nt">"BatteryStatus"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"BatteryChargingStatus"</span><span class="p">:</span> <span class="s2">"NORMAL_CHARGING"</span><span class="p">,</span>
<span class="nt">"BatteryCapacity"</span><span class="p">:</span> <span class="s2">"12"</span><span class="p">,</span>
<span class="nt">"BatteryRemainingAmount"</span><span class="p">:</span> <span class="s2">"9"</span><span class="p">,</span>
<span class="nt">"BatteryRemainingAmountWH"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"BatteryRemainingAmountkWH"</span><span class="p">:</span> <span class="s2">""</span>
<span class="p">},</span>
<span class="nt">"PluginState"</span><span class="p">:</span> <span class="s2">"CONNECTED"</span><span class="p">,</span>
<span class="nt">"CruisingRangeAcOn"</span><span class="p">:</span> <span class="s2">"110760.0"</span><span class="p">,</span>
<span class="nt">"CruisingRangeAcOff"</span><span class="p">:</span> <span class="s2">"112320.0"</span><span class="p">,</span>
<span class="nt">"TimeRequiredToFull200_6kW"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"HourRequiredToFull"</span><span class="p">:</span> <span class="s2">"2"</span><span class="p">,</span>
<span class="nt">"MinutesRequiredToFull"</span><span class="p">:</span> <span class="s2">"30"</span>
<span class="p">},</span>
<span class="nt">"NotificationDateAndTime"</span><span class="p">:</span> <span class="s2">"2016/04/06 21:58"</span><span class="p">,</span>
<span class="nt">"TargetDate"</span><span class="p">:</span> <span class="s2">"2016/04/06 21:57"</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="update-status">
<h3>1.3 Update status</h3>
<p>Ask for an update from the car:</p>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York"</span> <span class="pre">--data-urlencode</span> "custom_sessionid=XXXXXXXXXX" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/BatteryStatusCheckRequest.php"</span></tt></blockquote>
<div class="highlight"><pre><span class="p">{</span> <span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span><span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span><span class="nt">"userId"</span><span class="p">:</span><span class="s2">"XXXXXX"</span><span class="p">,</span><span class="nt">"vin"</span><span class="p">:</span><span class="s2">"XXXXXXXXX"</span><span class="p">,</span><span class="nt">"resultKey"</span><span class="p">:</span><span class="s2">"XXXXXXXXXX"</span><span class="p">}</span><span class="err">%</span>
</pre></div>
<p>The resultKey can be used to poll for when the updated value is available:</p>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York&resultKey=XXXXXX"</span> <span class="pre">--data-urlencode</span> "custom_sessionid=XXXXXXXXXX" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/BatteryStatusCheckResultRequest.php"</span></tt></blockquote>
<p>If it's not ready yet:</p>
<div class="highlight"><pre><span class="p">{</span><span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span><span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span><span class="nt">"responseFlag"</span><span class="p">:</span><span class="s2">"0"</span><span class="p">}</span>
</pre></div>
<p>Once status is updated, you get:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"responseFlag"</span><span class="p">:</span><span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"operationResult"</span><span class="p">:</span><span class="s2">"START"</span><span class="p">,</span>
<span class="nt">"timeStamp"</span><span class="p">:</span><span class="s2">"2016-02-20 20:29:33"</span><span class="p">,</span>
<span class="nt">"cruisingRangeAcOn"</span><span class="p">:</span><span class="s2">"129712.0"</span><span class="p">,</span>
<span class="nt">"cruisingRangeAcOff"</span><span class="p">:</span><span class="s2">"133584.0"</span><span class="p">,</span>
<span class="nt">"currentChargeLevel"</span><span class="p">:</span><span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"chargeMode"</span><span class="p">:</span><span class="s2">"NOT_CHARGING"</span><span class="p">,</span>
<span class="nt">"pluginState"</span><span class="p">:</span><span class="s2">"NOT_CONNECTED"</span><span class="p">,</span>
<span class="nt">"charging"</span><span class="p">:</span><span class="s2">"NO"</span><span class="p">,</span>
<span class="nt">"chargeStatus"</span><span class="p">:</span><span class="s2">"CT"</span><span class="p">,</span>
<span class="nt">"batteryDegradation"</span><span class="p">:</span><span class="s2">"11"</span><span class="p">,</span>
<span class="nt">"batteryCapacity"</span><span class="p">:</span><span class="s2">"12"</span><span class="p">,</span>
<span class="nt">"timeRequiredToFull"</span><span class="p">:{</span><span class="nt">"hours"</span><span class="p">:</span><span class="s2">""</span><span class="p">,</span><span class="nt">"minutes"</span><span class="p">:</span><span class="s2">""</span><span class="p">},</span>
<span class="nt">"timeRequiredToFull200"</span><span class="p">:{</span><span class="nt">"hours"</span><span class="p">:</span><span class="s2">""</span><span class="p">,</span><span class="nt">"minutes"</span><span class="p">:</span><span class="s2">""</span><span class="p">},</span>
<span class="nt">"timeRequiredToFull200_6kW"</span><span class="p">:{</span><span class="nt">"hours"</span><span class="p">:</span><span class="s2">""</span><span class="p">,</span><span class="nt">"minutes"</span><span class="p">:</span><span class="s2">""</span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="remote-climate-control">
<h3>1.4 Remote Climate Control</h3>
<div class="section" id="get-current-ac-status">
<h4>1.4.1 Get current AC status</h4>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York"</span> <span class="pre">--data-urlencode</span> "custom_sessionid=XXXXXXXXXX" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/remoteACRecordsRequest.php"</span></tt></blockquote>
<p>Response:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="nt">"RemoteACRecords"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"OperationResult"</span><span class="p">:</span> <span class="s2">"FINISH"</span><span class="p">,</span>
<span class="nt">"OperationDateAndTime"</span><span class="p">:</span> <span class="s2">"Apr 6, 2016 07:19 PM"</span><span class="p">,</span>
<span class="nt">"RemoteACOperation"</span><span class="p">:</span> <span class="s2">"STOP"</span><span class="p">,</span>
<span class="nt">"ACStartStopDateAndTime"</span><span class="p">:</span> <span class="s2">"Apr 6, 2016 07:19 PM"</span><span class="p">,</span>
<span class="nt">"CruisingRangeAcOn"</span><span class="p">:</span> <span class="s2">"152880.0"</span><span class="p">,</span>
<span class="nt">"CruisingRangeAcOff"</span><span class="p">:</span> <span class="s2">"157248.0"</span><span class="p">,</span>
<span class="nt">"ACStartStopURL"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"PluginState"</span><span class="p">:</span> <span class="s2">"NOT_CONNECTED"</span><span class="p">,</span>
<span class="nt">"ACDurationBatterySec"</span><span class="p">:</span> <span class="s2">"900"</span><span class="p">,</span>
<span class="nt">"ACDurationPluggedSec"</span><span class="p">:</span> <span class="s2">"7200"</span>
<span class="p">},</span>
<span class="nt">"OperationDateAndTime"</span><span class="p">:</span> <span class="s2">""</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="turn-on-ac">
<h4>1.4.2 Turn on AC</h4>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York"</span> <span class="pre">--data-urlencode</span> "custom_sessionid=XXXXXXXXXX" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/ACRemoteRequest.php"</span></tt></blockquote>
<p>Response:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="nt">"userId"</span><span class="p">:</span> <span class="s2">"virantha@gmail.com"</span><span class="p">,</span>
<span class="nt">"vin"</span><span class="p">:</span> <span class="s2">"XXXX"</span><span class="p">,</span>
<span class="nt">"resultKey"</span><span class="p">:</span> <span class="s2">"XXXXXXXXXXXXXX"</span>
<span class="p">}</span>
</pre></div>
<p>Use the <tt class="docutils literal">resultKey</tt> to get the status of the AC request:</p>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York&resultKey=XXXXXX"</span> <span class="pre">--data-urlencode</span> "custom_sessionid=XXXXXXXXXX" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/ACRemoteRsult.php"</span></tt></blockquote>
<p>If the status is not ready yet:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="nt">"responseFlag"</span><span class="p">:</span> <span class="s2">"0"</span>
<span class="p">}</span>
</pre></div>
<p>Once the AC is turned on, the following response will be sent:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="nt">"responseFlag"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"operationResult"</span><span class="p">:</span> <span class="s2">"START_BATTERY"</span><span class="p">,</span>
<span class="nt">"acContinueTime"</span><span class="p">:</span> <span class="s2">"15"</span><span class="p">,</span>
<span class="nt">"cruisingRangeAcOn"</span><span class="p">:</span> <span class="s2">"104720.0"</span><span class="p">,</span>
<span class="nt">"cruisingRangeAcOff"</span><span class="p">:</span> <span class="s2">"109208.0"</span><span class="p">,</span>
<span class="nt">"timeStamp"</span><span class="p">:</span> <span class="s2">"2016-04-07 16:47:41"</span><span class="p">,</span>
<span class="nt">"hvacStatus"</span><span class="p">:</span> <span class="s2">"ON"</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="turn-off-ac">
<h4>1.4.3 Turn off AC</h4>
<blockquote>
<tt class="docutils literal">curl <span class="pre">--data</span> <span class="pre">"RegionCode=NNA&lg=en-US&DCMID=XXXX&VIN=XXXX&tz=America/New_York"</span> <span class="pre">--data-urlencode</span> "custom_sessionid=XXXXXXXXXX" <span class="pre">-X</span> POST <span class="pre">"https://gdcportalgw.its-mo.com/gworchest_0323A/gdc/ACRemoteOffRequest.php"</span></tt></blockquote>
<p>The response checking is identical to the AC on request.</p>
</div>
</div>
</div>
<div class="section" id="to-be-continued">
<h2>2 to be continued....</h2>
</div>
</body></html>An Example Provisioning of HACKT on a Centos 7 VM using Vagrant and Ansible2016-03-10T10:14:00-05:00viranthatag:https://virantha.com,2016-03-10:2016/03/10/example-provisioning-of-hackt-using-vagrant-ansible/<html><body><p>I recently needed to get an open-source circuit toolkit (<a class="reference external" href="https://github.com/fangism/hackt/">HACKT</a>)compiled and deployed on my local system. I already knew
the C++ source compiled and installed fine on Linux, and didn't want to jump through the hoops to get it running
on my Mac 10.11 (El Capitan) system, so I decided to bring up a reproducible build on a VM. I'd used Ansible in the
past but had forgotten some of the syntax, so I decided to write a simple playbook that would get this done, as well
as document some of the build details required for <a class="reference external" href="https://github.com/fangism/hackt/">HACKT</a>. Some of the details on using Ansible locally on a Vagrant
VM weren't also immediately obvious to me, so this is also an attempt to collate that information for future reference.</p>
<div class="section" id="install-vagrant-and-ansible">
<h2>1 Install Vagrant and Ansible</h2>
<p>Vagrant is a nice way to bring up a virtual machine (running through
VirtualBox, for example) from predefined 'boxes'
very easily. Ansible is a separate provisioning system that lets you easily,
and in a reproducible and documented fashion, customize these virtual machines,
and only requires SSH running on the target system. Vagrant also ships with
some nice bindings to Ansible, allowing for really easy provisioning of the deployed boxes.
Even more, you don't even need Ansible installed on your local machine as Vagrant will
install it automatically in your virtual machine and execute the provisioning inside it,
which is a really lightweight way to do it.</p>
<p>I won't go into the details of installing these tools, but they were very
straight-forward to get up and running.</p>
</div>
<div class="section" id="get-the-vanilla-virtual-machine-up">
<h2>2 Get the vanilla virtual machine up</h2>
<p>First, just go to a directory where you will be keeping all the files needed to build your
server. Then, within that directory, do the following:</p>
<div class="highlight"><pre>vagrant init
vagrant box add centos/7
</pre></div>
<p>The first writes a template <tt class="docutils literal">Vagrantfile</tt> to your current directory, that will contain
the configuration information for customizing your new VM. The second command downloads
a pre-built CentOS 7 system from the <a class="reference external" href="https://atlas.hashicorp.com/boxes/search?utm_source=vagrantcloud.com&vagrantcloud=1">Vagrant community</a> and stores it in your <tt class="docutils literal"><span class="pre">$HOME/.vagrant.d/boxes</span></tt></p>
<p>Next, we want to configure and provision our VM that will be brought up from this downloaded image. So,
edit your Vagrantfile, remove all the crud, and insert the following lines:</p>
<div class="highlight"><pre><span class="no">Vagrant</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">box</span> <span class="o">=</span> <span class="s2">"centos/7"</span>
<span class="k">end</span>
</pre></div>
<p>Now, run the following command to start your VM:</p>
<div class="highlight"><pre>vagrant up
</pre></div>
<p>This will bring up your new CentOS VM, as well as rsync your current directory to the VM under <tt class="docutils literal">/home/vagrant/sync</tt>.</p>
<p>You can ssh into the VM by doing:</p>
<div class="highlight"><pre>vagrant ssh
</pre></div>
</div>
<div class="section" id="getting-ansible-provisioning-working">
<h2>3 Getting Ansible Provisioning working</h2>
<p>To enable Ansible provisioning inside your , add the following lines to your <tt class="docutils literal">Vagrantfile</tt>:</p>
<div class="highlight"><pre><span class="no">Vagrant</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">box</span> <span class="o">=</span> <span class="s2">"centos/7"</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">provision</span> <span class="s2">"ansible_local"</span> <span class="k">do</span> <span class="o">|</span><span class="n">ansible</span><span class="o">|</span>
<span class="n">ansible</span><span class="o">.</span><span class="n">provisioning_path</span> <span class="o">=</span> <span class="s2">"/home/vagrant/sync"</span>
<span class="n">ansible</span><span class="o">.</span><span class="n">playbook</span> <span class="o">=</span> <span class="s2">"playbook.yml"</span>
<span class="k">end</span>
<span class="k">end</span>
</pre></div>
<p>The <tt class="docutils literal">playbook.yml</tt> is where we'll write the Ansible instructions (in YAML
format) that will provision this VM. This file should stay in your current
directory, but when Vagrant brings up the VM, it will rsync this file to your
<tt class="docutils literal">/home/vagrant/sync</tt> directory inside your VM, so that's why we need the
<tt class="docutils literal">provisioning_path</tt> option specified.</p>
<p>Also note that the configuration to Vagrant is specifying <tt class="docutils literal">ansible_local</tt>, which means Vagrant will
install and run Ansible inside the VM.</p>
<p>Next, we'll build up the <tt class="docutils literal">playbook.yml</tt> file step by step.</p>
<div class="section" id="connection-setting">
<h3>3.1 Connection setting</h3>
<p>At the top of the file is the connection setings:</p>
<div class="highlight"><pre><span class="nn">---</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">hosts</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">all</span>
<span class="l-Scalar-Plain">connection</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">local</span>
</pre></div>
<p>This is all that is needed to run the provisioning locally on the VM.</p>
</div>
<div class="section" id="setup-some-variables">
<h3>3.2 Setup some variables</h3>
<div class="highlight"><pre><span class="l-Scalar-Plain">vars</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">src_dir</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">/home/vagrant/src</span>
<span class="l-Scalar-Plain">hackt_dir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">src_dir</span><span class="nv"> </span><span class="s">}}/hackt"</span>
</pre></div>
</div>
<div class="section" id="start-the-tasks-section-and-update-packages">
<h3>3.3 Start the tasks section and update packages</h3>
<div class="highlight"><pre><span class="l-Scalar-Plain">tasks</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Update yum</span>
<span class="l-Scalar-Plain">yum</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name=* state=latest</span>
<span class="l-Scalar-Plain">sudo</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">true</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Install packages</span>
<span class="l-Scalar-Plain">yum</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name="{{ item }}" state=present</span>
<span class="l-Scalar-Plain">sudo</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">true</span>
<span class="l-Scalar-Plain">with_items</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="s">"@Development</span><span class="nv"> </span><span class="s">tools"</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">guile</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">libtool-ltdl-devel</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">wget</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">texinfo</span>
</pre></div>
<p>The nice thing about using Ansible is that the actions are idempotent, so you
can reprovision any number of times and it will keep a consistent state, as well as only
running commands that will end up modifying the current state.</p>
<p>The other tricky thing here is the quotes when installing packages, especially the syntax
for the "Development tools" group.</p>
</div>
<div class="section" id="downgrading-bison-version">
<h3>3.4 Downgrading Bison version</h3>
<p><a class="reference external" href="https://github.com/fangism/hackt/">HACKT</a>, as of this writing, seems to have issues with Bison 2.7 on CentOS7. The work-around for now
is to downgrade to version 2.3.
First, we <tt class="docutils literal">yum remove bison</tt>, download version 2.3, build, and then install it.</p>
<div class="highlight"><pre><span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Remove newest bison</span>
<span class="l-Scalar-Plain">yum</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name=bison state=removed</span>
<span class="l-Scalar-Plain">sudo</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">true</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Create src directory</span>
<span class="l-Scalar-Plain">file</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">path={{ src_dir }} state=directory</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Download old bison for hackt</span>
<span class="l-Scalar-Plain">get_url</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">url=http://ftp.gnu.org/gnu/bison/bison-2.3.tar.gz dest={{ src_dir }}</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Unzip bison</span>
<span class="l-Scalar-Plain">unarchive</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">src={{ src_dir }}/bison-2.3.tar.gz dest={{ src_dir }}</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Build and install new bison</span>
<span class="l-Scalar-Plain">shell</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">item</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">args</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">chdir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">src_dir</span><span class="nv"> </span><span class="s">}}/bison-2.3"</span>
<span class="l-Scalar-Plain">with_items</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">./configure</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">make</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">sudo make install</span>
</pre></div>
</div>
<div class="section" id="building-hackt">
<h3>3.5 Building HACKT</h3>
<p>Next, we clone the <a class="reference external" href="https://github.com/fangism/hackt/">HACKT</a> repository and checkout a specific snapshot, bootstrap, configure, and make.</p>
<p>The bootstrap step has a <tt class="docutils literal">creates</tt> arg in it that doesn't rerun it if the <tt class="docutils literal">configure</tt> script is already
there.</p>
<p>The compiler <tt class="docutils literal">CXXFLAGS</tt> does require warnings not to turn into errors because of some unused-typedef
warnings in the current gcc.</p>
<div class="highlight"><pre><span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Get HACKT</span>
<span class="l-Scalar-Plain">git</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">repo=https://github.com/fangism/hackt.git</span>
<span class="l-Scalar-Plain">dest={{ src_dir }}/hackt</span>
<span class="l-Scalar-Plain">version=tags/SNAPSHOT-RELEASE-20141024</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Bootstrap HACKT</span>
<span class="l-Scalar-Plain">shell</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">./bootstrap</span>
<span class="l-Scalar-Plain">args</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">chdir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">hackt_dir</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">creates</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">hackt_dir</span><span class="nv"> </span><span class="s">}}/configure"</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Build HACKT</span>
<span class="l-Scalar-Plain">shell</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">item</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">args</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">chdir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">hackt_dir</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">creates</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">/usr/local/bin/hackt</span>
<span class="l-Scalar-Plain">with_items</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">env CXXFLAGS='-Wno-error' ./configure</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">make</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">sudo make install</span>
</pre></div>
</div>
</div>
<div class="section" id="final-steps-and-other-tips">
<h2>4 Final steps and other tips</h2>
<p>And that's basically it, for a very simple provisioning system. Ansible has some really cool
features that we haven't touched on here, and it's probably a good idea to breakdown the playbook
into a more hierarchical structure. But for now, here's the complete playbook:</p>
<div class="highlight"><pre><span class="nn">---</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">hosts</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">all</span>
<span class="l-Scalar-Plain">connection</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">local</span>
<span class="l-Scalar-Plain">vars</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">src_dir</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">/home/vagrant/src</span>
<span class="l-Scalar-Plain">hackt_dir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">src_dir</span><span class="nv"> </span><span class="s">}}/hackt"</span>
<span class="l-Scalar-Plain">tasks</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Update yum</span>
<span class="l-Scalar-Plain">yum</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name=* state=latest</span>
<span class="l-Scalar-Plain">sudo</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">true</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Install packages</span>
<span class="l-Scalar-Plain">yum</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name="{{ item }}" state=present</span>
<span class="l-Scalar-Plain">sudo</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">true</span>
<span class="l-Scalar-Plain">with_items</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="s">"@Development</span><span class="nv"> </span><span class="s">tools"</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">guile</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">libtool-ltdl-devel</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">wget</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">texinfo</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Remove newest bison</span>
<span class="l-Scalar-Plain">yum</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name=bison state=removed</span>
<span class="l-Scalar-Plain">sudo</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">true</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Create src directory</span>
<span class="l-Scalar-Plain">file</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">path={{ src_dir }} state=directory</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Download old bison for hackt</span>
<span class="l-Scalar-Plain">get_url</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">url=http://ftp.gnu.org/gnu/bison/bison-2.3.tar.gz dest={{ src_dir }}</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Unzip bison</span>
<span class="l-Scalar-Plain">unarchive</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">src={{ src_dir }}/bison-2.3.tar.gz dest={{ src_dir }}</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Build and install new bison</span>
<span class="l-Scalar-Plain">shell</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">item</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">args</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">chdir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">src_dir</span><span class="nv"> </span><span class="s">}}/bison-2.3"</span>
<span class="l-Scalar-Plain">with_items</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">./configure</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">make</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">sudo make install</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Get HACKT</span>
<span class="l-Scalar-Plain">git</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">repo=https://github.com/fangism/hackt.git</span>
<span class="l-Scalar-Plain">dest={{ src_dir }}/hackt</span>
<span class="l-Scalar-Plain">version=tags/SNAPSHOT-RELEASE-20141024</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Bootstrap HACKT</span>
<span class="l-Scalar-Plain">shell</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">./bootstrap</span>
<span class="l-Scalar-Plain">args</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">chdir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">hackt_dir</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">creates</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">hackt_dir</span><span class="nv"> </span><span class="s">}}/configure"</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">Build HACKT</span>
<span class="l-Scalar-Plain">shell</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">item</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">args</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">chdir</span><span class="p-Indicator">:</span> <span class="s">"{{</span><span class="nv"> </span><span class="s">hackt_dir</span><span class="nv"> </span><span class="s">}}"</span>
<span class="l-Scalar-Plain">with_items</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">env CXXFLAGS='-Wno-error' ./configure</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">make</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">sudo make install</span>
</pre></div>
<p>Some further useful commands and tips are below.</p>
<div class="section" id="utility-commands">
<h3>4.1 Utility commands</h3>
<ul class="simple">
<li><em>vagrant suspend</em> - Shuts down but saves state on the disk</li>
<li><em>vagrant resume</em> - Resumes the suspended VM</li>
<li><em>vagrant halt</em> - Shutdown</li>
<li><em>vagrant destroy</em> - Destroy all the customizations and disk</li>
<li><em>vagrant reload --provision</em> - Restart a running VM after making changes to Vagrantfile</li>
<li><em>vagrant provision</em> - Rerun the provisioning on a running VM</li>
</ul>
</div>
<div class="section" id="sharing-files">
<h3>4.2 Sharing files</h3>
<p>I wasn't able to get the NFS shares working, but just stuck to the simple rsync option where a directory
on the host can be copied to your VM on bootup/reload. Just add the following line to your Vagrantfile:</p>
<div class="highlight"><pre><span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">synced_folder</span> <span class="s2">"share"</span><span class="p">,</span> <span class="s2">"/share/dev"</span><span class="p">,</span> <span class="ss">type</span><span class="p">:</span> <span class="s2">"rsync"</span>
</pre></div>
<p>This will copy your <tt class="docutils literal">share/</tt> subdirectory to the VM's <tt class="docutils literal">/share/dev</tt> directory</p>
</div>
</div>
</body></html>Reverse engineering the remote control protocol for the Nissan LEAF (the new Nissan EV protocol)2016-01-12T10:14:00-05:00viranthatag:https://virantha.com,2016-01-12:2016/01/12/reverse-engineering-nissan-connect-ev-protocol/<html><body><div class="section" id="nissan-connect-ev">
<h2>1 Nissan Connect EV</h2>
<div class="note">
<p class="first admonition-title">Note</p>
<p class="last"><em>Update April 2016</em>: Nissan has updated their API calls to actually use the user login information to authenticate instead of just the VIN.
You can see a summary of the newer API in my new post <a class="reference external" href="https://virantha.com/2016/04/07/updated-reverse-engineering-nissan-connect-ev-protocol/">here</a>.</p>
</div>
<p>For those of you with a Nissan LEAF, the first cool things you get for free with the car is remote telematics via a web interface or a iOS or Android application.
The car has a built-in cellular modem that can be used to get all kinds of neat stats from the car (like miles driven, energy consumption, efficiency, etc), as well
as send map directions and enable/disable the charging and the climate control. In the latter part of 2015, Nissan revamped their Carwings servers which provided these
services, and they appear to be combining it with their general telematics interfaces across all their car models. The new application now talks to a different
REST endpoint and I could find scant documentation on this, so I decided to try my hand at reverse-engineering what the new Nissan EV application is doing.</p>
</div>
<div class="section" id="older-api">
<h2>2 Older API</h2>
<p>The following pages and threads have an outdated API:</p>
<ul class="simple">
<li><a class="reference external" href="http://www.mynissanleaf.com/viewtopic.php?f=27&t=2214&start=70">http://www.mynissanleaf.com/viewtopic.php?f=27&t=2214&start=70</a></li>
<li><a class="reference external" href="https://github.com/haykinson/pycarwings">https://github.com/haykinson/pycarwings</a></li>
</ul>
<p>Pycarwings actually works, but uses the original API. The forum thread then
details an updated REST endpoint (v3) but I could never get the authenticate to
take (possibly, I was using the wrong type of headers).</p>
<p>So the next step was to actually sniff the requests coming out of the Nissan EV
Connect app using a man-in-the-middle-proxy.</p>
</div>
<div class="section" id="first-steps-to-analyzing-the-nissan-ev-network-traffic">
<h2>3 First steps to analyzing the Nissan EV network traffic</h2>
<div class="section" id="installing-the-proxy-to-view-the-nissan-ev-ios-app-headers">
<h3>3.1 Installing the proxy to view the Nissan EV iOS app headers</h3>
<p>I followed the instructions on using <a class="reference external" href="http://jasdev.me/intercepting-ios-traffic/">mitmproxy here</a>.</p>
<p>Of course, just doing:</p>
<blockquote>
pip install mitmproxy</blockquote>
<p>unfortunately did not work on my mac running El Capitan, so I had to diagnose the following errors:</p>
<ul class="simple">
<li>lxml compilation failed, so I realized I had to reinstall the xcode command line tools after upgrading to El Capitan by doing <cite>xcode-select --install</cite></li>
<li>Then, it couldn't find the openssl headers to install the crytography package. So I had to read up on how El Capitan had <a class="reference external" href="https://github.com/pyca/cryptography/issues/2350">deprecated the standard openssl headers</a>. tldr; run <cite>env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install mitmproxy</cite></li>
</ul>
<p>Then, it was a matter of running <cite>mitmproxy</cite> and setting my iOS device to use a manual proxy on port 8080 pointing to my mac. Then, you browse to the mitm.it using safari and install the
appropriate certificate for your device. Then I launched the Nissan EV application and started looking at the traffic. (note to self, in order to copy text from the mitmproxy screen,
you need to hold <cite>fn+option</cite> down as you select)</p>
</div>
<div class="section" id="rest-requests">
<h3>3.2 REST requests</h3>
<p>The request/responses appear to be pretty straight-forward. On my initial reading, curiously, it seems like you just need the constant DCMID and VIN fields from the
login for subsequent requests. There doesn't appear to be any session based authentication token used.</p>
<div class="section" id="initial-login-to-nissan-servers">
<h4>3.2.1 Initial login to Nissan servers</h4>
<p>Initial logon request, where USERID is your Nissan Portal login (email address) and USERPASSWORD is the accompanying password:</p>
<blockquote>
<a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/UserLoginRequest.php?RegionCode=NNA&lg=en-US&DCMID=&VIN=&tz=&UserId=USERID&Password=USERPASSWORD">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/UserLoginRequest.php?RegionCode=NNA&lg=en-US&DCMID=&VIN=&tz=&UserId=USERID&Password=USERPASSWORD</a></blockquote>
<p>Results in:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"CustomerInfo"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Country"</span><span class="p">:</span> <span class="s2">"US"</span><span class="p">,</span>
<span class="nt">"Language"</span><span class="p">:</span> <span class="s2">"en-US"</span><span class="p">,</span>
<span class="nt">"Nickname"</span><span class="p">:</span> <span class="s2">"XXXXXXXXX"</span><span class="p">,</span>
<span class="nt">"OwnerId"</span><span class="p">:</span> <span class="s2">"XXXXXXXXXX"</span><span class="p">,</span>
<span class="nt">"RegionCode"</span><span class="p">:</span> <span class="s2">"NNA"</span><span class="p">,</span>
<span class="nt">"Timezone"</span><span class="p">:</span> <span class="s2">"America/New_York"</span><span class="p">,</span>
<span class="nt">"UserId"</span><span class="p">:</span> <span class="s2">"XXXXXX"</span><span class="p">,</span>
<span class="nt">"UserVehicleBoundDurationSec"</span><span class="p">:</span> <span class="s2">"946771200"</span><span class="p">,</span>
<span class="nt">"VehicleImage"</span><span class="p">:</span> <span class="s2">"/content/language/default/ima</span>
<span class="s2"> "</span><span class="err">VehicleInfo</span><span class="s2">": {</span>
<span class="s2"> "</span><span class="err">DCMID</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">EncryptedNAVIID</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXXXXXXX</span>
<span class="s2">"LastDCMUseTime"</span><span class="p">:</span> <span class="s2">"Jan 12, 2016 03:37 PM"</span><span class="p">,</span>
<span class="nt">"LastVehicleLoginTime"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"MSN"</span><span class="p">:</span> <span class="s2">"XXXXXXXXXXXXXXX"</span><span class="p">,</span>
<span class="nt">"NAVIID"</span><span class="p">:</span> <span class="s2">"XXXXXXXXXXXX"</span><span class="p">,</span>
<span class="nt">"SIMID"</span><span class="p">:</span> <span class="s2">"XXXXXXXXXXXXXXXXXXX"</span><span class="p">,</span>
<span class="nt">"UserVehicleBoundTime"</span><span class="p">:</span> <span class="s2">"2015-09-24T18:10:</span>
<span class="s2"> "</span><span class="err">VIN</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXX</span><span class="s2">"</span>
<span class="s2"> }</span>
<span class="s2"> },</span>
<span class="s2"> "</span><span class="err">EncAuthToken</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXXXXXXXXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">UserInfoRevisionNo</span><span class="s2">": "</span><span class="mi">1</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">VehicleInfoList</span><span class="s2">": {</span>
<span class="s2"> "</span><span class="err">VehicleInfo</span><span class="s2">": [</span>
<span class="s2"> {</span>
<span class="s2"> "</span><span class="err">charger</span><span class="mi">20066</span><span class="s2">": "</span><span class="kc">false</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">nickname</span><span class="s2">": "</span><span class="err">VLeaf</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">telematicsEnabled</span><span class="s2">": "</span><span class="kc">true</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">vin</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXX</span><span class="s2">"</span>
<span class="s2"> }</span>
<span class="s2"> ],</span>
<span class="s2"> "</span><span class="err">vehicleInfo</span><span class="s2">": [</span>
<span class="s2"> {</span>
<span class="s2"> "</span><span class="err">charger</span><span class="mi">20066</span><span class="s2">": "</span><span class="kc">false</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">nickname</span><span class="s2">": "</span><span class="err">VLeaf</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">telematicsEnabled</span><span class="s2">": "</span><span class="kc">true</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">vin</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXX</span><span class="s2">"</span>
<span class="s2"> }</span>
<span class="s2"> ]</span>
<span class="s2"> },</span>
<span class="s2"> "</span><span class="err">message</span><span class="s2">": "</span><span class="err">success</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">sessionId</span><span class="s2">": "</span><span class="mi">43978</span><span class="err">f</span><span class="mi">5</span><span class="err">b-e</span><span class="mi">45</span><span class="err">c</span><span class="mi">-498</span><span class="err">c-a</span><span class="mf">9e5</span><span class="err">-be</span><span class="mf">85e59</span><span class="err">df</span><span class="mi">3</span><span class="err">dc</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">status</span><span class="s2">": 200,</span>
<span class="s2"> "</span><span class="err">vehicle</span><span class="s2">": {</span>
<span class="s2"> "</span><span class="err">profile</span><span class="s2">": {</span>
<span class="s2"> "</span><span class="err">dcmId</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">encAuthToken</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXXXXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">gdcPassword</span><span class="s2">": "</span><span class="err">XXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">gdcUserId</span><span class="s2">": "</span><span class="err">XXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">nickname</span><span class="s2">": "</span><span class="err">XXXXXXXXX</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">status</span><span class="s2">": "</span><span class="err">ACCEPTED</span><span class="s2">",</span>
<span class="s2"> "</span><span class="err">statusDate</span><span class="s2">": "</span><span class="err">Sep</span> <span class="mi">24</span><span class="p">,</span> <span class="err">2015</span> <span class="err">12:00</span> <span class="err">AM</span><span class="nt">",</span>
<span class="nt"> "</span><span class="err">vin</span><span class="s2">": "</span><span class="err">XXXXXXXXXXXXXXXXX"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="err">}</span>
</pre></div>
</div>
<div class="section" id="request-current-status">
<h4>3.2.2 Request current status</h4>
<p>Request an update on the battery status:</p>
<blockquote>
<a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/BatteryStatusRecordsRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&TimeFrom=2015-09-24T18:10:06">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/BatteryStatusRecordsRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&TimeFrom=2015-09-24T18:10:06</a></blockquote>
<p>Response:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"BatteryStatusRecords"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"BatteryStatus"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"BatteryCapacity"</span><span class="p">:</span> <span class="s2">"12"</span><span class="p">,</span>
<span class="nt">"BatteryChargingStatus"</span><span class="p">:</span> <span class="s2">"NOT_CHARGING"</span><span class="p">,</span>
<span class="nt">"BatteryRemainingAmount"</span><span class="p">:</span> <span class="s2">"11"</span><span class="p">,</span>
<span class="nt">"BatteryRemainingAmountWH"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"BatteryRemainingAmountkWH"</span><span class="p">:</span> <span class="s2">""</span>
<span class="p">},</span>
<span class="nt">"CruisingRangeAcOff"</span><span class="p">:</span> <span class="s2">"133840.0"</span><span class="p">,</span>
<span class="nt">"CruisingRangeAcOn"</span><span class="p">:</span> <span class="s2">"122368.0"</span><span class="p">,</span>
<span class="nt">"NotificationDateAndTime"</span><span class="p">:</span> <span class="s2">"2016/01/12 15:37"</span><span class="p">,</span>
<span class="nt">"OperationDateAndTime"</span><span class="p">:</span> <span class="s2">"Jan 12, 2016 10:37 AM"</span><span class="p">,</span>
<span class="nt">"OperationResult"</span><span class="p">:</span> <span class="s2">"START"</span><span class="p">,</span>
<span class="nt">"PluginState"</span><span class="p">:</span> <span class="s2">"NOT_CONNECTED"</span><span class="p">,</span>
<span class="nt">"TargetDate"</span><span class="p">:</span> <span class="s2">"2016/01/12 15:37"</span><span class="p">,</span>
<span class="nt">"TimeRequiredToFull"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"HourRequiredToFull"</span><span class="p">:</span> <span class="s2">"3"</span><span class="p">,</span>
<span class="nt">"MinutesRequiredToFull"</span><span class="p">:</span> <span class="s2">"30"</span>
<span class="p">},</span>
<span class="nt">"TimeRequiredToFull200"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"HourRequiredToFull"</span><span class="p">:</span> <span class="s2">"2"</span><span class="p">,</span>
<span class="nt">"MinutesRequiredToFull"</span><span class="p">:</span> <span class="s2">"0"</span>
<span class="p">},</span>
<span class="nt">"TimeRequiredToFull200_6kW"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"HourRequiredToFull"</span><span class="p">:</span> <span class="s2">"2"</span><span class="p">,</span>
<span class="nt">"MinutesRequiredToFull"</span><span class="p">:</span> <span class="s2">"0"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="update-status">
<h4>3.2.3 Update status</h4>
<p>Ask for an update from the car:</p>
<blockquote>
<a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/BatteryStatusCheckRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/BatteryStatusCheckRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York</a></blockquote>
<div class="highlight"><pre><span class="p">{</span> <span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span><span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span><span class="nt">"userId"</span><span class="p">:</span><span class="s2">"XXXXXX"</span><span class="p">,</span><span class="nt">"vin"</span><span class="p">:</span><span class="s2">"XXXXXXXXX"</span><span class="p">,</span><span class="nt">"resultKey"</span><span class="p">:</span><span class="s2">"XXXXXXXXXX"</span><span class="p">}</span><span class="err">%</span>
</pre></div>
<p>The resultKey can be used to poll for when the updated value is available:</p>
<blockquote>
<a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/BatteryStatusCheckResultRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&resultKey=XXXXXXX">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/BatteryStatusCheckResultRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&resultKey=XXXXXXX</a></blockquote>
<p>If it's not ready yet:</p>
<div class="highlight"><pre><span class="p">{</span><span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span><span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span><span class="nt">"responseFlag"</span><span class="p">:</span><span class="s2">"0"</span><span class="p">}</span>
</pre></div>
<p>Once status is updated, you get:</p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span><span class="mi">200</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span><span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"responseFlag"</span><span class="p">:</span><span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"operationResult"</span><span class="p">:</span><span class="s2">"START"</span><span class="p">,</span>
<span class="nt">"timeStamp"</span><span class="p">:</span><span class="s2">"2016-02-20 20:29:33"</span><span class="p">,</span>
<span class="nt">"cruisingRangeAcOn"</span><span class="p">:</span><span class="s2">"129712.0"</span><span class="p">,</span>
<span class="nt">"cruisingRangeAcOff"</span><span class="p">:</span><span class="s2">"133584.0"</span><span class="p">,</span>
<span class="nt">"currentChargeLevel"</span><span class="p">:</span><span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"chargeMode"</span><span class="p">:</span><span class="s2">"NOT_CHARGING"</span><span class="p">,</span>
<span class="nt">"pluginState"</span><span class="p">:</span><span class="s2">"NOT_CONNECTED"</span><span class="p">,</span>
<span class="nt">"charging"</span><span class="p">:</span><span class="s2">"NO"</span><span class="p">,</span>
<span class="nt">"chargeStatus"</span><span class="p">:</span><span class="s2">"CT"</span><span class="p">,</span>
<span class="nt">"batteryDegradation"</span><span class="p">:</span><span class="s2">"11"</span><span class="p">,</span>
<span class="nt">"batteryCapacity"</span><span class="p">:</span><span class="s2">"12"</span><span class="p">,</span>
<span class="nt">"timeRequiredToFull"</span><span class="p">:{</span><span class="nt">"hours"</span><span class="p">:</span><span class="s2">""</span><span class="p">,</span><span class="nt">"minutes"</span><span class="p">:</span><span class="s2">""</span><span class="p">},</span>
<span class="nt">"timeRequiredToFull200"</span><span class="p">:{</span><span class="nt">"hours"</span><span class="p">:</span><span class="s2">""</span><span class="p">,</span><span class="nt">"minutes"</span><span class="p">:</span><span class="s2">""</span><span class="p">},</span>
<span class="nt">"timeRequiredToFull200_6kW"</span><span class="p">:{</span><span class="nt">"hours"</span><span class="p">:</span><span class="s2">""</span><span class="p">,</span><span class="nt">"minutes"</span><span class="p">:</span><span class="s2">""</span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="get-various-formats">
<h4>3.2.4 Get various formats</h4>
<p>Here some misc requests:</p>
<ul>
<li><p class="first"><a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/GetRegionSetting.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&Country=USA&cartype=15MY">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/GetRegionSetting.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&Country=USA&cartype=15MY</a></p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"ABOUT_THE_APP"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"APP_NOTICE"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"CONTACT"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"CUSTOMER_SUPPORT"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ROADSIDE_ASSISTANCE"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"CUSTOMER_SUPPORT"</span><span class="p">:</span> <span class="s2">"1-877-664-2738"</span><span class="p">,</span>
<span class="nt">"DRIVING_HELP"</span><span class="p">:</span> <span class="s2">"https://www.ev.nissanconnect.jp/EV/mycar/driving_help"</span><span class="p">,</span>
<span class="nt">"DRIVING_RECORDS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ECO_DRIVE_EVALUATION"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ECO_TREE"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ELECTRICITY_RATE_SIMULATION"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ELECTRICITY_USAGE_RECORDS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ELECTRIC_REDIRECT"</span><span class="p">:</span> <span class="s2">"https://www.ev.nissanconnect.jp/gw-redirect/gdc/auth-redirect.php"</span><span class="p">,</span>
<span class="nt">"FORGOT_PASSWORD"</span><span class="p">:</span> <span class="s2">"https://owners.nissanusa.com/nowners/"</span><span class="p">,</span>
<span class="nt">"HOME"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"NEW_REGISTRATION"</span><span class="p">:</span> <span class="s2">"https://owners.nissanusa.com/nowners/"</span><span class="p">,</span>
<span class="nt">"NOTIFICATIONS_HISTORY"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"NOTIFICATION_DETAIL"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"BATTERY_HEATER_NOTIFICATION"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"BATTERY_STATUS_NOTIFICATION"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"CHARGE_START/FINISH_NOTIFICATION"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"CHARGE_STOP_ALERT"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"CLIMATE_CONTROL_START/STOP_NOTIFICATION"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"ELECTRICITY_COST_NOTIFICATION"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"0"</span>
<span class="p">},</span>
<span class="nt">"MAINTENANCE_ALERT"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"0"</span>
<span class="p">},</span>
<span class="nt">"MY_CAR_FINDER"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"0"</span>
<span class="p">},</span>
<span class="nt">"UNPLUGGED_STATUS_ALERT"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Frequency"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Mail"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Push"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"SMS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Visible"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"NOTIFICATION_SETTINGS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"PATH_VIEWER"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"PRIVACY_LINK"</span><span class="p">:</span> <span class="s2">"https://m.nissanusa.com/privacy"</span><span class="p">,</span>
<span class="nt">"REGIONAL_RANKINGS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"REMOTE_CHARGE"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"REMOTE_CLIMATE_CONTROL"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ROADSIDE_ASSISTANCE"</span><span class="p">:</span> <span class="s2">"1-800-801-6161"</span><span class="p">,</span>
<span class="nt">"ROUTE"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"ROUTE_PLANNER"</span><span class="p">:</span> <span class="s2">"https://routepln.its-mo.com/RoutePlanner"</span><span class="p">,</span>
<span class="nt">"RegionCode"</span><span class="p">:</span> <span class="s2">"NNA"</span><span class="p">,</span>
<span class="nt">"UPDATEATSTARTUP"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"USER"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"VEHICLE_SWAP"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"WORLD_RANKINGS"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Wh"</span><span class="p">:</span> <span class="s2">"Wh"</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</li>
<li><p class="first"><a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/dateformat.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&TargetDate=2016-01-12&type=Y/m&lg=en-US">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/dateformat.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&TargetDate=2016-01-12&type=Y/m&lg=en-US</a></p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"DisplayDate"</span><span class="p">:</span> <span class="s2">"Jan/2016"</span><span class="p">,</span>
<span class="nt">"TargetDate"</span><span class="p">:</span> <span class="s2">"2016-01-12"</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</li>
<li><p class="first"><a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/GetCountrySetting.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/GetCountrySetting.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York</a></p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"setting"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"Australia"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Austria"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Belgium"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Canada"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Cyprus"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Czech Rep"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Denmark"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Estonia"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Finland"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"France"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Germany"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Greece"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Guadeloupe"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Hungary"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Iceland"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Ireland"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Israel"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Italy"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Japan"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Latvia"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Lithuania"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Luxembourg"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Malta"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Martinique"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Monaco"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Netherlands"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Norway"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Poland"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Portugal"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Reunion"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Slovakia"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Slovenia"</span><span class="p">:</span> <span class="s2">"0"</span><span class="p">,</span>
<span class="nt">"Spain"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Sweden"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"Switzerland"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"The United States"</span><span class="p">:</span> <span class="s2">"1"</span><span class="p">,</span>
<span class="nt">"United Kingdom"</span><span class="p">:</span> <span class="s2">"1"</span>
<span class="p">},</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</li>
<li><p class="first"><a class="reference external" href="https://gdcportalgw.its-mo.com/app_notice/AppNoticeRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&DeviceType=ios&DeviceTime=2016-01-12%2010:43&UserId=XXXXXXX">https://gdcportalgw.its-mo.com/app_notice/AppNoticeRequest.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&DeviceType=ios&DeviceTime=2016-01-12%2010:43&UserId=XXXXXXX</a></p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"AppNotice"</span><span class="p">:</span> <span class="p">[],</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</li>
<li><p class="first"><a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/dateformat.php?TargetDate=2016-01-12%2010:37&type=updateTime&lg=en-US">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/dateformat.php?TargetDate=2016-01-12%2010:37&type=updateTime&lg=en-US</a></p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"DisplayDate"</span><span class="p">:</span> <span class="s2">"Jan 12, 2016 10:37 AM"</span><span class="p">,</span>
<span class="nt">"TargetDate"</span><span class="p">:</span> <span class="s2">"2016-01-12 10:37"</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</li>
<li><p class="first"><a class="reference external" href="https://gdcportalgw.its-mo.com/orchestration_1111/gdc/dateformat.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&TargetDate=2016-01-12%2010:37&type=updateTime&lg=en-US">https://gdcportalgw.its-mo.com/orchestration_1111/gdc/dateformat.php?RegionCode=NNA&lg=en-US&DCMID=XXXXXXXXXXXX&VIN=XXXXXXXXXXXXXXXXX&tz=America/New_York&TargetDate=2016-01-12%2010:37&type=updateTime&lg=en-US</a></p>
<div class="highlight"><pre><span class="p">{</span>
<span class="nt">"DisplayDate"</span><span class="p">:</span> <span class="s2">"Jan 12, 2016 10:37 AM"</span><span class="p">,</span>
<span class="nt">"TargetDate"</span><span class="p">:</span> <span class="s2">"2016-01-12 10:37"</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"success"</span><span class="p">,</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
</pre></div>
</li>
</ul>
</div>
</div>
</div>
<div class="section" id="to-be-continued">
<h2>4 to be continued....</h2>
</div>
</body></html>Migrating XenServer Virtual Machine VDIs to KVM2015-03-09T12:02:00-04:00viranthatag:https://virantha.com,2015-03-09:2015/03/09/migrating-xenserver-vdi-to-kvm/<html><body><p>After having worked with Xenserver 6.2 for over a year now, I've decided to
migrate away from it. It just wasn't stable enough with SATA
pass-through on my machine, and it had some really annoying bugs with the way
snapshots and backups were handled:</p>
<ul class="simple">
<li>Under load, my SATA 8-bay enclosure would sometimes hang the entire server (not just the Windows 7 host that was the end-point)</li>
<li>Snapshots would fail with error saying not enough space, even though the disk was less than three quarters full</li>
<li>No built-in backup solution (combined with the snapshot unreliability, this is a dealbreaker)</li>
<li>Need to manually clean up disk clusters/vms (documented bug)</li>
<li>Network transfers to a remote system/disk across management NICs are throttled (good god)</li>
<li>No USB 3.0 support, so good luck backing up even to a local disk</li>
</ul>
<p>Needless to say, I've been itching to give another solution a shot. I finally decided to migrate
to KVM, and it took me a while to figure out how to seamlessly migrate my Windows virtual
machines over. The actual process is very simple and detailed below.</p>
<div class="section" id="googling-it">
<h2><a class="toc-backref" href="#id1">1 Googling it</a></h2>
<p>There are lots of tutorials on the web about using XenCenter in Windows to export disks,
but in my experience, XenCenter was extremely unreliable even for imports and I
didn't want to build another Windows VM just to export disks.</p>
</div>
<div class="section" id="the-easy-way">
<h2><a class="toc-backref" href="#id2">2 The easy way</a></h2>
<p>Turns out, and it appears not many people are aware of it, there's a simple command line interface
to the <a class="reference external" href="https://github.com/xenserver/transfervm/wiki/Usage-Examples">Transfer VMs</a> built into Xenserver 6.x. Using the Transfer VM on the
host (dom0), you can expose any offline VDI as a raw image over HTTP using
built-in commands with no additional software to install. Then, on your client
disk system, you simply use <tt class="docutils literal">curl</tt> to download the disk image over the
network. Very simple, and no third-party tools involved. The steps involved are given below:</p>
<div class="section" id="expose-the-vdi-on-your-xenserver-host">
<h3><a class="toc-backref" href="#id3">2.1 Expose the VDI on your Xenserver Host</a></h3>
<p>There are three pieces of information (UUIDs) that you're going to need for this process, and
I've summarized them in the table below, including the commands you'll run on dom0 as root. Just
find the particular UUID that identifies the resource you need in the text output that is generated by each command
below.</p>
<table border="1" class="docutils pretty compact cell-border stripe" id="uuids">
<caption>UUIDs you'll need</caption>
<colgroup>
<col width="20%"></col>
<col width="80%"></col>
</colgroup>
<thead valign="bottom">
<tr><th class="head">UUID</th>
<th class="head">Command</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>HOST-UUID</td>
<td>xe host-list</td>
</tr>
<tr><td>NETWORK-UUID</td>
<td>xe network-list</td>
</tr>
<tr><td>VDI-UUID</td>
<td>xe vdi-list</td>
</tr>
</tbody>
</table>
<p>Once you have all of the details above, shut down the VM or detach the VDI, and execute the following:</p>
<pre class="literal-block">
xe host-call-plugin host-uuid=HOST-UUID fn=expose args:vdi_uuid=VDI-UUID args:network_uuid=NETWORK-UUID args:transfer_mode=http
</pre>
<p>This will return (print out) a new record locator UUID that you should copy down. Let's call it RECORD-UUID
and use it in the following command to get the exposed URL:</p>
<pre class="literal-block">
xe host-call-plugin host-uuid=HOST-UUID plugin=transfer fn=get_record args:record_handle=RECORD-UUID
</pre>
<p>This will print out something like the following:</p>
<div class="highlight"><pre><span class="cp"><?xml version="1.0"?></span>
<span class="nt"><transfer_record</span>
<span class="na">username=</span><span class="s">"70f8e0b297fcdd94"</span>
<span class="na">status=</span><span class="s">"exposed"</span>
<span class="na">url_path=</span><span class="s">"/vdi_uuid_4b26aeff-bc12-44f0-a5b6-2e07ec37e75c"</span>
<span class="na">record_handle=</span><span class="s">"dda7a17f-2621-a77a-0464-6d95a19fbbc3"</span>
<span class="na">ip=</span><span class="s">"10.80.238.238"</span>
<span class="na">transfer_mode=</span><span class="s">"http"</span>
<span class="na">url_full=</span><span class="s">"http://70f8e0b297fcdd94:8c21f66590fcb08e@10.80.238.238:80/vdi_uuid_4b26aeff-bc12-44f0-a5b6-2e07ec37e75c"</span>
<span class="na">device=</span><span class="s">"xvdn"</span>
<span class="na">use_ssl=</span><span class="s">"false"</span>
<span class="na">password=</span><span class="s">"8c21f66590fcb08e"</span>
<span class="na">port=</span><span class="s">"80"</span>
<span class="na">vdi_uuid=</span><span class="s">"4b26aeff-bc12-44f0-a5b6-2e07ec37e75c"</span><span class="nt">></span>
<span class="nt"></transfer_record></span>
</pre></div>
<p>What you need is the <tt class="docutils literal">url_full</tt> entry; let's call it VDI_URL. This is the url where the Transfer VM will supply the raw disk image of the VDI you specified.</p>
</div>
<div class="section" id="install-the-image-on-your-kvm-host">
<h3><a class="toc-backref" href="#id4">2.2 Install the image on your KVM host</a></h3>
<p>On the destination machine, all you have to do is issue the following command to download that VDI as a raw
image:</p>
<pre class="literal-block">
curl VDI_URL -o disk.raw
</pre>
<p>Next, convert this to a .qcow2 image for KVM using the following command:</p>
<pre class="literal-block">
qemu-img convert -O qcow2 disk.raw disk.qcow2
</pre>
<p>Now, you can just attach this disk to any KVM virtual machine. If you're migrating from a Windows 7
machine, you will probably need to make sure you use the IDE drivers and not the virtio drivers for the
disk for the first boot. So, you would change something like this in your .xml configuration:</p>
<div class="highlight"><pre><span class="nt"><target</span> <span class="na">dev=</span><span class="s">'hda'</span> <span class="na">bus=</span><span class="s">'virtio'</span><span class="nt">/></span>
<span class="nt"><address</span> <span class="na">type=</span><span class="s">'pci'</span> <span class="na">domain=</span><span class="s">'0x0000'</span> <span class="na">bus=</span><span class="s">'0x00'</span> <span class="na">slot=</span><span class="s">'0x04'</span> <span class="na">function=</span><span class="s">'0x0'</span><span class="nt">/></span>
</pre></div>
<p>To something like this:</p>
<div class="highlight"><pre><span class="nt"><target</span> <span class="na">dev=</span><span class="s">'hda'</span> <span class="na">bus=</span><span class="s">'ide'</span><span class="nt">/></span>
<span class="nt"><address</span> <span class="na">type=</span><span class="s">'drive'</span> <span class="na">controller=</span><span class="s">'0'</span> <span class="na">bus=</span><span class="s">'0'</span> <span class="na">target=</span><span class="s">'0'</span> <span class="na">unit=</span><span class="s">'0'</span><span class="nt">/></span>
</pre></div>
<p>And that's it!</p>
</div>
<div class="section" id="cleanup">
<h3><a class="toc-backref" href="#id5">2.3 Cleanup</a></h3>
<p><strong>Optional</strong>: You may also want to go back to your XenServer host and un-expose your VDI, so you can reattach/restart
your VM by executing:</p>
<pre class="literal-block">
xe host-call-plugin host-uuid=HOST-UUID plugin=transfer fn=unexpose args:record_handle=RECORD-UUID
</pre>
<p>You may also want to zero some of the free space on your Windows disk image, to
take full advantage of the compressibility of the
qcow2 image format. First, in your Windows VM, download the <tt class="docutils literal">sdelete</tt> utility from <a class="reference external" href="https://technet.microsoft.com/en-us/sysinternals/bb897443.aspx">here</a>. Then, run the following command from a command prompt in Windows:</p>
<pre class="literal-block">
sdelete.exe -z c:
</pre>
<p>This will take some time to complete. Once that's done, shut down your VM, and run the following command
on your Linux KVM host:</p>
<pre class="literal-block">
qemu-img convert -c -O qcow2 original.qcow2 compressed.qcow2
</pre>
<p>Startup your VM using the new <tt class="docutils literal">compressed.qcow2</tt> and if all looks good, you can go ahead and delete the <tt class="docutils literal">original.qcow2</tt> image.</p>
</div>
</div>
<script charset="utf8" src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script><script charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script><script>
$(document).ready(function() {
$('#uuids').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script></body></html>Building a low-power home file server using Intel's Avoton platform2015-01-26T11:13:00-05:00viranthatag:https://virantha.com,2015-01-26:2015/01/26/building-a-low-power-file-server-intel-avoton/<html><body><p>In this post I'll detail the hardware I used to upgrade my 2 year old home
server that was getting a little bit long in the tooth. I wanted to replace a
Core i7 based XenServer host connected to an external 8-bay SATA enclosure that
had an idle power consumption of over 100W for machine+enclosure
with all drives spun down. In my new build, with 8 hard drives and
3 SSDs, I can get it under 40W with drives spun down but the system awake.</p>
<div class="section" id="getting-the-components">
<h2><a class="toc-backref" href="#id1">1 Getting the components</a></h2>
<p>Here's the list and summary of each part. I have more details and options
listed in the next section.</p>
<table border="1" class="docutils pretty compact cell-border stripe" id="components">
<caption>File-server components</caption>
<colgroup>
<col width="15%"></col>
<col width="10%"></col>
<col width="10%"></col>
<col width="65%"></col>
</colgroup>
<thead valign="bottom">
<tr><th class="head">Component</th>
<th class="head">Retail</th>
<th class="head">Paid</th>
<th class="head">Comments</th>
</tr>
</thead>
<tbody valign="top">
<tr><td><div class="first last"><div><a class="amazon" href="http://www.amazon.com/gp/product/B00HIDQG6E/ref=as_li_tf_il?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00HIDQG6E&linkCode=am2&tag=virantha-20&"><img border="0" src="http://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B00HIDQG6E&Format=_SL110_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=virantha-20" style="vertical-align: middle;" width="110"/>ASRock C2550I Intel Avoton Motherboard and CPU</a></div></div></td>
<td>$309</td>
<td>$223 @ newegg</td>
<td>Mini-itx 12 SATA ports (!), quad-core Intel Atom 14W TDP CPU</td>
</tr>
<tr><td><div class="first last"><div><a class="amazon" href="http://www.amazon.com/gp/product/B00IAELTAI/ref=as_li_tf_il?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00IAELTAI&linkCode=am2&tag=virantha-20&"><img border="0" src="http://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B00IAELTAI&Format=_SL110_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=virantha-20" style="vertical-align: middle;" width="110"/>Silverstone DS380B Case</a></div></div></td>
<td>$149</td>
<td>$149</td>
<td>Supports 8 hard drives with hot-swappable trays (only 2x molex power connectors for all 8 drives) plus 4x 2.5in drive cage. SFX power supply required.</td>
</tr>
<tr><td><div class="first last"><div><a class="amazon" href="http://www.amazon.com/gp/product/B00C249PEO/ref=as_li_tf_il?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00C249PEO&linkCode=am2&tag=virantha-20&"><img border="0" src="http://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B00C249PEO&Format=_SL110_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=virantha-20" style="vertical-align: middle;" width="110"/>Corsair Air Series SP120 PWM Quiet Edition Case fans</a></div></div></td>
<td>$20 each</td>
<td>~$15 @ newegg wih twin pack</td>
<td>PWM (4-pin) so speed is controllable and reportable via BIOS. Very quiet and high CFM and pressure</td>
</tr>
<tr><td><div class="first last"><div><a class="amazon" href="http://www.amazon.com/gp/product/B00H46JSVU/ref=as_li_tf_il?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00H46JSVU&linkCode=am2&tag=virantha-20&"><img border="0" src="http://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B00H46JSVU&Format=_SL110_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=virantha-20" style="vertical-align: middle;" width="110"/>Kingston 8GB DDR3 1600 Low Voltage (1.35V) with ECC</a></div></div></td>
<td>$84</td>
<td>$84</td>
<td>Used a single stick and will upgrade later. Important to note this is server memory with ECC support and 1.35V (low-voltage) vs standard 1.5V</td>
</tr>
<tr><td><div class="first last"><div><a class="amazon" href="http://www.amazon.com/gp/product/B00FA4KP8S/ref=as_li_tf_il?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00FA4KP8S&linkCode=am2&tag=virantha-20&"><img border="0" src="http://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B00FA4KP8S&Format=_SL110_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=virantha-20" style="vertical-align: middle;" width="110"/>Silverstone Tek 300W (ST30SF) Power Supply</a></div></div></td>
<td>$55</td>
<td>$55</td>
<td>SFX form factor. Temperature controlled fan that stays off under 50W load in typical usage. Non-modular</td>
</tr>
<tr><td><div class="first last"><div><a class="amazon" href="http://www.amazon.com/gp/product/B009GUXWDM/ref=as_li_tf_il?ie=UTF8&camp=1789&creative=9325&creativeASIN=B009GUXWDM&linkCode=am2&tag=virantha-20&"><img border="0" src="http://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B009GUXWDM&Format=_SL110_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=virantha-20" style="vertical-align: middle;" width="110"/>Monoprice SATA cable</a></div></div></td>
<td>$2.50</td>
<td>$2.50</td>
<td>Good length for this case/MB, and locking latches and good build quality</td>
</tr>
</tbody>
</table>
<div class="section" id="build-details">
<h3><a class="toc-backref" href="#id2">1.1 Build details</a></h3>
</div>
<div class="section" id="motherboard">
<h3><a class="toc-backref" href="#id3">1.2 Motherboard</a></h3>
<p>I went with the new Avoton series of server boards based on the Silvermont Atom
platform (the passively cooled CPU is soldered onto the motherboard). These
are surprisingly capable and extremely low power. For example, the 4-core
<a class="amazon" href="http://www.amazon.com/gp/product/B00LS34SJM?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">ASRock C2550D4l</a> mini-ITX motherboard has 14W TDP, and is
usually under 10W. This server-class motherboard has no problem playing back
my high-bit-rate 1080P Plex streams, and supports 64GB ECC RAM (regular RAM
also works), 12 SATA ports, and has an IPMI interface. IPMI uses a dedicated
ethernet port and lets you get a console via a web interface over the network,
letting you physically place this machine anywhere you like and do everything
without a keyboard and monitor. You get access to all the sensors, BIOS, and
even BIOS updates through this web interface, which makes everything
hassle-free, and makes me wonder why I didn't splurge for this feature years
ago.</p>
<p>If you're planning on doing any kind of virtualization or multiple Plex
streams, then I recommend jumping up to the 8-core version, the
<a class="amazon" href="http://www.amazon.com/gp/product/B00HIDQG6E?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">ASRock C2750D4I</a>.</p>
</div>
<div class="section" id="case">
<h3><a class="toc-backref" href="#id4">1.3 Case</a></h3>
<p>I decided to go with the new NAS desktop <a class="amazon" href="http://www.amazon.com/gp/product/B00IAELTAI?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">DS380B case</a> from
Silverstone that has a built-in 8x3.5in bay hot-swappable backplane, plus an
internal bracket for mounting up to 4x2.5in hard drives or SSDs. It comes with
3x120mm fans that are relatively quiet, but I upgraded to 3 [amazon B00C249PEO
Corsair SP120 Quiet fans] that support fan speed control via PWM and also
report RPMs to the BIOS.</p>
</div>
<div class="section" id="memory">
<h3><a class="toc-backref" href="#id5">1.4 Memory</a></h3>
<p>Here is a list of all the Kingston memory that will work with this board:
<a class="reference external" href="http://www.kingston.com/us/memory/search?DeviceType=8&Mfr=ASR&Line=Server%20Board&Model=87350&Description=Kingston_ValueRam_Memory_Server_Premier_Memory_HyperX_Memory_for_ASRock_Server_Board_C2550D4I">Kingston ECC memory</a>
I decided to pay the slight premium for a stick of ECC low-voltage RAM.</p>
<p>See <a class="reference external" href="http://www.servethehome.com/testing-power-savings-low-voltage-135v-kingston-memory-intel-avoton/">this article</a> for the benefits of using 1.35V vs 1.5V RAM. You'd probably end up saving about 1 to 2W on a 16GB system, or almost 10% on a ~20W TDP system like the Avoton.</p>
</div>
<div class="section" id="power-supply">
<h3><a class="toc-backref" href="#id6">1.5 Power supply</a></h3>
<p>I originally considered getting a <a class="amazon" href="http://www.amazon.com/gp/product/B005TWE6B8?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">PicoPSU-160-XT</a> with 16A
brick, but the power brick was sold out, and I wasn't too keen in the end on
running the 12V directly to my system. Plus, it was tough to find an adapter
plate to have a nice DC-plugin come into the case. Moreover, I was looking
at combined over $100 for this combination, so I ended up settling for a
<a class="amazon" href="http://www.amazon.com/gp/product/B00FA4KP8S?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">300W power supply</a> that's fanless under 50W draw from
Silverstone listed below. I probably could've saved a few watts with the
PicoPSU with a good adapter, but this was the easier solution. Keep in mind
that the Silvestone DS308B NAS case uses a SFX form-factor power supply, like
this power supply, and not a standard ATX sized power supply.</p>
</div>
</div>
<script charset="utf8" src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script><script charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script><script>
$(document).ready(function() {
$('#components').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script></body></html>Setting up the Edgemax EdgeRouter Lite (ERLite-3) in a Home Environment2014-08-22T09:46:00-04:00viranthatag:https://virantha.com,2014-08-22:2014/08/22/setting-up-edgemax-edgerouter-lite-erlite-3/<html><body><p>This is just my cheat sheet on setting up the <a class="amazon" href="http://www.amazon.com/gp/product/B00CPRVF5K?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">EdgeRouter ERLite-3</a>
in a home environment. I recently migrated to this after a long
search for a low-power firewall and router solution. Prior to getting this
beauty, I'd migrated my stand-alone PC running Ipcop/Smoothwall to a Xen
virtual machine firewall, which unfortunately had some usability issues.</p>
<div class="section" id="command-line-interface">
<h2><a class="toc-backref" href="#id1">1 Command-line interface</a></h2>
<p>Although the gui is fairly responsive, a lot of the features are only exposed through the
underlying command line. The Ubiquiti EdgeOS products run a fork of the open-source
Vyatta firewall OS, and you can usually find what you're looking for with a quick web search.</p>
<p>You can either access the CLI through the GUI (seems to have issues running on Safari though), or
directly through ssh. The former has issues with timeouts, however, for long-running jobs.</p>
<p>Once you're in the command-line, you are in regular mode. To switch to changing the configuration,
type:</p>
<pre class="literal-block">
configure
</pre>
<p>Once you are done making configuration changes with 'set' commands, you'll want to commit the changes to
the current firewall state. After doing that, if everything looks good, you'll want to save the changes
to /config/config.boot</p>
<pre class="literal-block">
commit
save
exit
</pre>
</div>
<div class="section" id="setting-up-dyndns">
<h2><a class="toc-backref" href="#id2">2 Setting up DynDNS</a></h2>
<p>If you use No-IP.com you have to rely on the built-in dyndns service but modify the server setting
as detailed in this <a class="reference external" href="http://www.ubnt.com/t5/EdgeMAX/DDNS-for-NO-IP/td-p/492809">post</a>.</p>
<pre class="literal-block">
set service dns dynamic interface eth1 service dyndns server dynupdate.no-ip.com
set service dns dynamic interface eth1 service dyndns host-name <host>
set service dns dynamic interface eth1 service dyndns login <username>
set service dns dynamic interface eth1 service dyndns password <password>
</pre>
<p>To force update the dynamic interface:</p>
<pre class="literal-block">
update dns dynamic interface eth1
</pre>
<p>To get the status (from regular mode)</p>
<pre class="literal-block">
show dns dynamic status
</pre>
<p>To schedule this using the built-in task scheduler (1.5.0 firmware)</p>
<pre class="literal-block">
set system task-scheduler task dyndns_update executable path /opt/vyatta/bin/vyatta-op-cmd-wrapper
set system task-scheduler task dyndns_update executable arguments "update dns dynamic interface eth1"
set system task-scheduler task dyndns_update interval 7d
</pre>
</div>
<div class="section" id="setting-up-web-filtering-using-squidguard">
<h2><a class="toc-backref" href="#id3">3 Setting up web filtering using SquidGuard</a></h2>
<p>First, get the blacklists</p>
<pre class="literal-block">
update webproxy blacklists
</pre>
<p>Will take a long time to build the adult database. You probably want to do it in a ssh terminal
and not in the web CLI, because the latter will timeout.</p>
<pre class="literal-block">
set service webproxy listen-address 192.168.9.1
set service webproxy url-filtering squidguard allow-ipaddr-url
set service webproxy url-filtering squidguard block-category ads
set service webproxy url-filtering squidguard block-category malware
set service webproxy url-filtering squidguard block-category phishing
set service webproxy url-filtering squidguard auto-update update-hour 15
set service webproxy url-filtering squidguard redirect-url "http://192.168.9.1/cgi-bin/squidGuard-simple.cgi?targetclass=%t&url=%u"
</pre>
<p>I'm not convinced I'll stick with the ad blocks, as the redirects cause my browsing sessions to sometimes
fail.</p>
</div>
<div class="section" id="qos-for-voip-calls">
<h2><a class="toc-backref" href="#id4">4 QOS for VOIP calls</a></h2>
<p>Taken directly from <a class="reference external" href="https://community.ubnt.com/t5/EdgeMAX-Configuration-Examples/EdgeMAX-Quality-of-Service-for-Voice-Over-IP-QoS-for-VoIP/ta-p/529077">this page</a></p>
<p>Assumes your LAN is on eth0, Internet is eth1, and 30/6Mbps connection.</p>
<pre class="literal-block">
# Set-up the details of the DownStream Policy
set traffic-policy shaper DownStream description "DownStream QoS policy"
set traffic-policy shaper DownStream bandwidth 30000kbit
set traffic-policy shaper DownStream class 10 description "RTP"
set traffic-policy shaper DownStream class 10 bandwidth 25%
set traffic-policy shaper DownStream class 10 ceiling 100%
set traffic-policy shaper DownStream class 10 match VOIP-RTP ip dscp 46
set traffic-policy shaper DownStream class 20 description "SIP"
set traffic-policy shaper DownStream class 20 bandwidth 5%
set traffic-policy shaper DownStream class 20 ceiling 100%
set traffic-policy shaper DownStream class 20 match VOIP-SIP ip dscp 24
set traffic-policy shaper DownStream default bandwidth 70%
set traffic-policy shaper DownStream default ceiling 100%
# Set-up the details of the UpStream Policy
set traffic-policy shaper UpStream description "UpStream QoS policy"
set traffic-policy shaper UpStream bandwidth 6000kbit
set traffic-policy shaper UpStream class 10 description "RTP"
set traffic-policy shaper UpStream class 10 bandwidth 50%
set traffic-policy shaper UpStream class 10 ceiling 100%
set traffic-policy shaper UpStream class 10 match VOIP-RTP ip dscp 46
set traffic-policy shaper UpStream class 20 description "SIP"
set traffic-policy shaper UpStream class 20 bandwidth 10%
set traffic-policy shaper UpStream class 20 ceiling 100%
set traffic-policy shaper UpStream class 20 match VOIP-SIP ip dscp 24
set traffic-policy shaper UpStream default bandwidth 40%
set traffic-policy shaper UpStream default ceiling 100%
# Apply the policies to the interfaces
set interfaces ethernet eth0 traffic-policy out DownStream
set interfaces ethernet eth1 traffic-policy out UpStream
# Commit, Save, and Exit
</pre>
</div>
<div class="section" id="installing-pound-reverse-proxy">
<h2><a class="toc-backref" href="#id5">5 Installing Pound Reverse Proxy</a></h2>
<p>First, follow the <a class="reference external" href="http://wiki.ubnt.com/Add_Other_Debian_Packages_to_EdgeOS">instructions here</a> to add Debian packages
with apt.</p>
<pre class="literal-block">
configure
set system package repository wheezy components 'main contrib non-free'
set system package repository wheezy distribution squeeze
set system package repository wheezy url http://http.us.debian.org/debian
set system package repository wheezy-security components main
set system package repository wheezy-security distribution squeeze/updates
set system package repository wheezy-security url http://security.debian.org
commit
save
exit
sudo apt-get update
</pre>
<p>Then, install Pound:</p>
<pre class="literal-block">
sudo apt-get install pound
</pre>
<p>You'll need to set the bind port to something other than 80, and follow the rest of the instructions <a class="reference external" href="http://virantha.com/2014/03/08/pound-reverse-proxy-easy-to-use-hostnames/">here</a>.</p>
</div>
<div class="section" id="setting-up-vpn-for-example-your-ios-device">
<h2><a class="toc-backref" href="#id6">6 Setting up VPN (for example, your iOS device)</a></h2>
<p>Follow the <a class="reference external" href="https://help.ubnt.com/hc/en-us/articles/205146070-EdgeMAX-L2TP-Server">procedure here</a> . Here is the relevant configuration after following all the commands:</p>
<pre class="literal-block">
admin@ubnt# show vpn ipsec
auto-firewall-nat-exclude disable
ipsec-interfaces {
interface eth1
}
nat-networks {
allowed-network 0.0.0.0/0 {
}
}
nat-traversal enable
admin@ubnt# show vpn l2tp
remote-access {
authentication {
local-users {
username virantha {
password **************
}
}
mode local
}
client-ip-pool {
start 192.168.101.2
stop 192.168.101.100
}
dhcp-interface eth1
dns-servers {
server-1 192.168.9.1
}
ipsec-settings {
authentication {
mode pre-shared-secret
pre-shared-secret **************
}
ike-lifetime 3600
}
mtu 1492
}
admin@ubnt# show firewall name WAN_LOCAL
name WAN_LOCAL {
default-action drop
description "WAN to router"
rule 1 {
action accept
description "Allow L2tp"
destination {
port 500,1701,4500
}
log disable
protocol udp
}
rule 2 {
action accept
description ESP
log disable
protocol esp
}
.
.
.
}
</pre>
<p>If for some reason you forget all the passwords and secrets, you can get a list of them by doing:</p>
<pre class="literal-block">
configure
show vpn l2tp
</pre>
</div>
</body></html>Example of how to authenticate using Oauth2 in Ember.js using Implicit Grant (client-side)2014-05-07T21:12:00-04:00viranthatag:https://virantha.com,2014-05-07:2014/05/07/ember-js-oauth2-implicit-grant-example/<html><body><p>This post will show a quick and complete example on how to authenticate someone
using their Google credentials in your <a class="reference external" href="http://emberjs.com">Ember.js</a> web
application. This uses the implicit-grant scheme in Oauth2, so everything
happens on the client side, without exposing the user's Google credentials or
requiring a back-end server. Obviously, at some point you'll have a server to
store the authentication token and persist other user information, but the
whole authentication infrastructure will stay on the client.</p>
<div class="section" id="what-we-re-building">
<h2><a class="toc-backref" href="#id1">1 What we're building</a></h2>
<p>I'm going to use a toy example that displays a list of events to an authenticated user.
When you first go to the app, it will show you nothing:</p>
<div class="figure" style="width: 789px; height: auto; max-width: 100%;">
<img alt="Logged out view" height="365" src="/images/2014/ember-oauth/loggedout.jpg" style="width: 789px; height: auto; max-width: 100%;" width="789"/>
</div>
<p>Any attempt to access it without being authenticated will redirect to the login
page, which will pop-up a window to enter your Google credentials.</p>
<div class="figure" style="width: 789px; height: auto; max-width: 100%;">
<img alt="Login page" height="365" src="/images/2014/ember-oauth/login_page.jpg" style="width: 789px; height: auto; max-width: 100%;" width="789"/>
</div>
<p>Once the credentials are entered succesfully, the app will automatically redirect to your events list.</p>
<div class="figure" style="width: 789px; height: auto; max-width: 100%;">
<img alt="Logged in view" height="365" src="/images/2014/ember-oauth/loggedin.jpg" style="width: 789px; height: auto; max-width: 100%;" width="789"/>
</div>
</div>
<div class="section" id="getting-the-code-and-running-it">
<h2><a class="toc-backref" href="#id2">2 Getting the code and running it</a></h2>
<div class="section" id="pre-requisites">
<h3><a class="toc-backref" href="#id3">2.1 Pre-requisites</a></h3>
<p>This method relies on a couple of great third-party extensions for Ember, which do all the heavy lifting
in my example.</p>
<ol class="arabic simple">
<li><a class="reference external" href="http://ember-simple-auth.simplabs.com/">Ember Simple Auth</a> - This really flexible and simple library
provides the basic framework for authentication and restricting certain routes to authenticated users.</li>
<li><a class="reference external" href="https://github.com/amkirwan/ember-oauth2">Ember Oauth2</a> - A simple but cool library to handle the communication
with Oauth2 services like Google or Facebook.</li>
</ol>
</div>
<div class="section" id="source-code">
<h3><a class="toc-backref" href="#id4">2.2 Source code</a></h3>
<p>You can find the source on <a class="reference external" href="https://github.com/virantha/ember-oauth-example">github</a>. Just
<cite>git clone</cite> this repository, and start a simple web server in that directory and you should be good
to go. For example, if you're on a Mac, you can just do:</p>
<div class="highlight"><pre>git clone https://github.com/virantha/ember-oauth-example.git
<span class="nb">cd </span>ember-oauth-example
python -m SimpleHttpServer 5000
</pre></div>
<p>and then go visit <a class="reference external" href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> in your browser. Remember, when dealing with Oauth, for local
testing, always use 127.0.0.1 instead of localhost, because a lot of the service providers don't like
providing authentication tokens to localhost.</p>
</div>
</div>
<div class="section" id="code-discussion">
<h2><a class="toc-backref" href="#id5">3 Code discussion</a></h2>
<div class="section" id="ember-index-html">
<h3><a class="toc-backref" href="#id6">3.1 Ember index.html</a></h3>
<p>The html is pretty trivial, but here's a quick walk through:</p>
<div class="highlight"><pre><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></span>
<span class="nt"><title></span>Events<span class="nt"></title></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span><span class="nt">></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">nav</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"navbar navbar-default"</span> <span class="nx">role</span><span class="o">=</span><span class="s2">"navigation"</span><span class="o">></span>
<span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"container-fluid"</span><span class="o">></span>
<span class="o"><</span><span class="nx">ul</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"nav navbar-nav"</span><span class="o">></span>
<span class="o"><</span><span class="nx">li</span><span class="o">></span> <span class="p">{{</span><span class="err">#</span><span class="nx">link</span><span class="o">-</span><span class="nx">to</span> <span class="s1">'events'</span><span class="p">}}</span><span class="nx">List</span> <span class="nx">of</span> <span class="nx">Events</span><span class="p">{{</span><span class="err">/link-to}}</li></span>
<span class="o"><</span><span class="err">/ul></span>
<span class="p">{{</span><span class="err">#</span><span class="k">if</span> <span class="nx">session</span><span class="p">.</span><span class="nx">isAuthenticated</span><span class="p">}}</span>
<span class="o"><</span><span class="nx">a</span> <span class="p">{{</span> <span class="nx">action</span> <span class="s1">'invalidateSession'</span> <span class="p">}}</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"btn btn-danger navbar-btn navbar-left"</span> <span class="o">></span><span class="nx">Logout</span><span class="o"><</span><span class="err">/a></span>
<span class="o"><</span><span class="nx">p</span> <span class="kr">class</span><span class="o">=</span><span class="s1">'navbar-right'</span><span class="o">></span><span class="nx">Logged</span> <span class="k">in</span> <span class="nx">as</span> <span class="p">{{</span><span class="nx">session</span><span class="p">.</span><span class="nx">userFn</span><span class="p">}}</span> <span class="p">{{</span><span class="nx">session</span><span class="p">.</span><span class="nx">userLn</span><span class="p">}}</span> <span class="o"><</span><span class="err">/p</span>
<span class="p">{{</span><span class="k">else</span><span class="p">}}</span>
<span class="p">{{</span><span class="err">#</span><span class="nx">link</span><span class="o">-</span><span class="nx">to</span> <span class="s1">'login'</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"btn navbar-btn navbar-left"</span> <span class="p">}}</span><span class="nx">Login</span><span class="p">{{</span><span class="err">/link-to}}</span>
<span class="p">{{</span><span class="err">/if}}</span>
<span class="o"><</span><span class="err">/div></span>
<span class="o"><</span><span class="err">/nav></span>
<span class="p">{{</span><span class="nx">outlet</span><span class="p">}}</span>
<span class="nt"></script></span>
</pre></div>
<p>The first script is our main application outlet for Ember. Obviously, it's a <a class="reference external" href="http://handlebarsjs.com">handlebars</a> template, and
it defines a quick <a class="reference external" href="http://getbootstrap.com">Bootstrap</a> navigation bar that has a link to our Events page. After that,
it has a conditional statement that checks if the session is authenticated. If it is, it displays
the logout button as well as some user information retrieved from Google after a successful
authentication. Otherwise, it just links to our Login route.</p>
<p>Here's the next template that handles the listing of the Events:</p>
<div class="highlight"><pre><span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span> <span class="na">data-template-name=</span><span class="s">"events"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">div</span><span class="o">></span>
<span class="o"><</span><span class="nx">table</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"table table-striped table-bordered table-condensed"</span><span class="o">></span>
<span class="o"><</span><span class="nx">thead</span><span class="o">></span>
<span class="o"><</span><span class="nx">tr</span><span class="o">></span>
<span class="o"><</span><span class="nx">th</span><span class="o">></span><span class="nb">Date</span><span class="o"><</span><span class="err">/th></span>
<span class="o"><</span><span class="nx">th</span><span class="o">></span><span class="nx">Event</span><span class="o"><</span><span class="err">/th></span>
<span class="o"><</span><span class="err">/tr></span>
<span class="o"><</span><span class="err">/thead></span>
<span class="o"><</span><span class="nx">tbody</span><span class="o">></span>
<span class="p">{{</span><span class="err">#</span><span class="nx">each</span> <span class="nx">model</span><span class="p">}}</span>
<span class="o"><</span><span class="nx">tr</span><span class="o">></span>
<span class="o"><</span><span class="nx">td</span><span class="o">></span> <span class="p">{{</span><span class="nx">date</span><span class="p">}}</span> <span class="o"><</span><span class="err">/td></span>
<span class="o"><</span><span class="nx">td</span><span class="o">></span> <span class="p">{{</span><span class="nx">title</span><span class="p">}}</span> <span class="o"><</span><span class="err">/td></span>
<span class="o"><</span><span class="err">/tr></span>
<span class="p">{{</span><span class="err">/each}}</span>
<span class="o"><</span><span class="err">/tbody></span>
<span class="o"><</span><span class="err">/table></span>
<span class="o"><</span><span class="err">/div></span>
<span class="nt"></script></span>
</pre></div>
<p>In this <tt class="docutils literal">events</tt> template, we just display a table and for each row, we iterate through an
array of events returned by the <tt class="docutils literal">EventsModel</tt> in our Ember app, and display the date and title per
event. Nothing in this template has any reference to authentication; all the protecting is done
in the app itself as we'll see shortly.</p>
<p>Finally, we need a <tt class="docutils literal">Login</tt> template as shown below:</p>
<div class="highlight"><pre><span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span> <span class="na">data-template-name=</span><span class="s">"login"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"page-header"</span><span class="o">></span>
<span class="o"><</span><span class="nx">h1</span><span class="o">></span><span class="nx">Login</span><span class="o"><</span><span class="err">/h1></span>
<span class="o"><</span><span class="err">/div></span>
<span class="o"><</span><span class="nx">div</span><span class="o">></span>
<span class="o"><</span><span class="nx">a</span> <span class="nx">href</span><span class="o">=</span><span class="s2">"#"</span> <span class="p">{{</span><span class="nx">action</span> <span class="s2">"authenticate"</span><span class="p">}}</span><span class="o">></span><span class="nx">Login</span> <span class="nx">using</span> <span class="nx">Google</span><span class="o"><</span><span class="err">/a></span>
<span class="o"><</span><span class="err">/div></span>
<span class="p">{{</span><span class="err">#</span><span class="k">if</span> <span class="nx">errorMessage</span><span class="p">}}</span>
<span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"alert alert-danger"</span><span class="o">></span>
<span class="o"><</span><span class="nx">p</span><span class="o">></span>
<span class="o"><</span><span class="nx">strong</span><span class="o">></span><span class="nx">Login</span> <span class="nx">failed</span><span class="o">:<</span><span class="err">/strong></span>
<span class="o"><</span><span class="nx">code</span><span class="o">></span><span class="p">{{</span><span class="nx">errorMessage</span><span class="p">}}</span><span class="o"><</span><span class="err">/code></span>
<span class="o"><</span><span class="err">/p></span>
<span class="o"><</span><span class="err">/div></span>
<span class="p">{{</span><span class="err">/if}}</span>
<span class="nt"></script></span>
</pre></div>
<p>This just shows link with an <tt class="docutils literal">authenticate</tt> action that will be triggered in our
Authenticator in our app. In addition, it has an additional rendering part to display
any authentication errors (e.g. if the user decided not to authorize access to their information. )</p>
<p>Finally, we have all the javascript includes. I've gone ahead and bundled the dependencies in the
example code, but usually this would be more modular and perhaps pointing to a CDN:</p>
<div class="highlight"><pre><span class="nt"><script </span><span class="na">src=</span><span class="s">"jquery.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"handlebars.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"ember.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"ember-simple-auth.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"ember-oauth2.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"//netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"app.js"</span><span class="nt">></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</pre></div>
</div>
<div class="section" id="ember-app-js">
<h3><a class="toc-backref" href="#id7">3.2 Ember app.js</a></h3>
<p>Now, let's discuss the second part of our app, the main javascript code that implements
the Ember App, located in <tt class="docutils literal">app.js</tt>.</p>
<div class="section" id="initialize">
<h4><a class="toc-backref" href="#id8">3.2.1 Initialize</a></h4>
<p>Here, we initialize our authentication plugins as well as the authorizer. This is taken
straight from the Ember-Simple-Auth guides:</p>
<div class="highlight"><pre> <span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">Ember</span><span class="p">.</span><span class="nx">Application</span><span class="p">.</span><span class="nx">initializer</span><span class="p">({</span>
<span class="nx">name</span><span class="o">:</span> <span class="s1">'authentication'</span><span class="p">,</span>
<span class="nx">initialize</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">container</span><span class="p">,</span> <span class="nx">application</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">container</span><span class="p">.</span><span class="nx">register</span><span class="p">(</span><span class="s1">'authenticator:custom'</span><span class="p">,</span> <span class="nx">App</span><span class="p">.</span><span class="nx">GoogleAuthenticator</span><span class="p">);</span>
<span class="nx">container</span><span class="p">.</span><span class="nx">register</span><span class="p">(</span><span class="s1">'authorizer:custom'</span><span class="p">,</span> <span class="nx">App</span><span class="p">.</span><span class="nx">CustomAuthorizer</span><span class="p">);</span>
<span class="nx">Ember</span><span class="p">.</span><span class="nx">SimpleAuth</span><span class="p">.</span><span class="nx">setup</span><span class="p">(</span><span class="nx">container</span><span class="p">,</span> <span class="nx">application</span><span class="p">,</span> <span class="p">{</span>
<span class="nx">authorizerFactory</span><span class="o">:</span> <span class="s1">'authorizer:custom'</span><span class="p">,</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">...</span>
<span class="p">})</span> <span class="p">();</span>
</pre></div>
<p>Next, let's turn on some debugging so we can see what happens inside Ember as
we transition from route to route:</p>
<div class="highlight"><pre><span class="nx">App</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Application</span><span class="p">.</span><span class="nx">create</span><span class="p">({</span>
<span class="nx">LOG_TRANSITIONS</span><span class="o">:</span><span class="kc">true</span> <span class="p">,</span>
<span class="nx">LOG_TRANSITIONS_INTERNAL</span><span class="o">:</span><span class="kc">true</span> <span class="p">,</span>
<span class="nx">LOG_ACTIVE_GENERATION</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">});</span>
</pre></div>
</div>
<div class="section" id="define-routes">
<h4><a class="toc-backref" href="#id9">3.2.2 Define routes</a></h4>
<p>Now, here's the usual Ember code that defines our routes. We only have two in this app,
a login page, and an events page. Notice that the EventsRoute is now using
a special mixin <tt class="docutils literal">AuthenticatedRouteMixin</tt> that marks it as requiring authentication.</p>
<div class="highlight"><pre><span class="c1">// Add some routes</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">Router</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">route</span><span class="p">(</span><span class="s1">'login'</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">resource</span><span class="p">(</span><span class="s1">'events'</span><span class="p">,</span> <span class="p">{</span><span class="nx">path</span><span class="o">:</span> <span class="s1">'events'</span><span class="p">},</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">EventsRoute</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Route</span><span class="p">.</span><span class="nx">extend</span><span class="p">(</span><span class="nx">Ember</span><span class="p">.</span><span class="nx">SimpleAuth</span><span class="p">.</span><span class="nx">AuthenticatedRouteMixin</span><span class="p">,{</span>
<span class="nx">model</span><span class="o">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">[</span> <span class="p">{</span><span class="nx">date</span><span class="o">:</span> <span class="s2">"Apr 26th"</span><span class="p">,</span> <span class="nx">title</span><span class="o">:</span> <span class="s2">"Science festival"</span><span class="p">},</span>
<span class="p">{</span><span class="nx">date</span><span class="o">:</span> <span class="s2">"Apr 27th"</span><span class="p">,</span> <span class="nx">title</span><span class="o">:</span> <span class="s2">"Swim class"</span><span class="p">}</span>
<span class="p">];</span>
<span class="p">},</span>
<span class="p">});</span>
</pre></div>
</div>
<div class="section" id="simpleauth-boiler-plate">
<h4><a class="toc-backref" href="#id10">3.2.3 SimpleAuth boiler-plate</a></h4>
<p>Next up is the code that is common to any application that uses SimpleAuth, whether you're
using the default password authentication or the custom Oauth authenticator we're building. All
this code is taken from the examples and documentation of this library.</p>
<div class="highlight"><pre><span class="c1">// Simple authentication</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">Router</span><span class="p">.</span><span class="nx">reopen</span><span class="p">({</span>
<span class="nx">rootURL</span><span class="o">:</span> <span class="s1">'index.html'</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">ApplicationRoute</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Route</span><span class="p">.</span><span class="nx">extend</span><span class="p">(</span><span class="nx">Ember</span><span class="p">.</span><span class="nx">SimpleAuth</span><span class="p">.</span><span class="nx">ApplicationRouteMixin</span><span class="p">);</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">LoginRoute</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Route</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">setupController</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">controller</span><span class="p">,</span> <span class="nx">model</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">controller</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">'errorMessage'</span><span class="p">,</span> <span class="kc">null</span><span class="p">);</span>
<span class="p">},</span>
<span class="nx">actions</span><span class="o">:</span> <span class="p">{</span>
<span class="nx">sessionAuthenticationFailed</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">controller</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">'errorMessage'</span><span class="p">,</span> <span class="nx">error</span><span class="p">);</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">LoginController</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Controller</span><span class="p">.</span><span class="nx">extend</span><span class="p">(</span><span class="nx">Ember</span><span class="p">.</span><span class="nx">SimpleAuth</span><span class="p">.</span><span class="nx">LoginControllerMixin</span><span class="p">,</span> <span class="p">{</span>
<span class="nx">authenticatorFactory</span><span class="o">:</span> <span class="s1">'authenticator:custom'</span>
<span class="p">});</span>
<span class="c1">// The authorizer that injects the auth token into every api request</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">CustomAuthorizer</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">SimpleAuth</span><span class="p">.</span><span class="nx">Authorizers</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">authorize</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">jqXHR</span><span class="p">,</span> <span class="nx">requestOptions</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">'session.isAuthenticated'</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="nx">Ember</span><span class="p">.</span><span class="nx">isEmpty</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">'session.token'</span><span class="p">)))</span> <span class="p">{</span>
<span class="nx">jqXHR</span><span class="p">.</span><span class="nx">setRequestHeader</span><span class="p">(</span><span class="s1">'Authorization'</span><span class="p">,</span> <span class="s1">'Token: '</span> <span class="o">+</span> <span class="k">this</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">'session.token'</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
</pre></div>
<p>You can see in the <tt class="docutils literal">LoginRoute</tt> we have a authentication failure action handler that sets
the <tt class="docutils literal">errorMessage</tt> for the <tt class="docutils literal">login</tt> handlebars template in our <tt class="docutils literal">index.html</tt>. The <tt class="docutils literal">setupController</tt>
is there to clear out any stale/old error messages.</p>
<div class="note">
<p class="first admonition-title">Note</p>
<p class="last">The final block shows the <tt class="docutils literal">Authorizer</tt> that will inject a custom header
into any back-end ajax calls with the session token (or authentication token,
if you want to be that crazy :-) ). We don't need this here since we're not
even dealing with a back-end server, but I thought I'd leave this code in to
show you how easy it is to transfer the token to your REST API.</p>
</div>
</div>
<div class="section" id="oauth2-authenticator">
<h4><a class="toc-backref" href="#id11">3.2.4 Oauth2 Authenticator</a></h4>
<p>Now we come to the main portion of the example that leverages both the SimpleAuth library
as well as the Oauth library. First there's a bit of configuration that needs to happen in
your app that specifies the oauth server.</p>
<div class="highlight"><pre><span class="nx">Ember</span><span class="p">.</span><span class="nx">OAuth2</span><span class="p">.</span><span class="nx">config</span> <span class="o">=</span> <span class="p">{</span>
<span class="nx">google</span><span class="o">:</span> <span class="p">{</span>
<span class="nx">clientId</span><span class="o">:</span> <span class="s2">"XXXXXXXX.apps.googleusercontent.com"</span><span class="p">,</span>
<span class="nx">authBaseUri</span><span class="o">:</span> <span class="s1">'https://accounts.google.com/o/oauth2/auth'</span><span class="p">,</span>
<span class="nx">redirectUri</span><span class="o">:</span> <span class="s1">'http://127.0.0.1:5000/redirect.html'</span><span class="p">,</span>
<span class="nx">scope</span><span class="o">:</span> <span class="s1">'https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">oauth</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">OAuth2</span><span class="p">.</span><span class="nx">create</span><span class="p">({</span><span class="nx">providerId</span><span class="o">:</span> <span class="s1">'google'</span><span class="p">});</span>
</pre></div>
<p>You need to go to the <a class="reference external" href="https://console.developers.google.com/project">developer google panel</a>, and setup a new application
as detailed in <a class="reference external" href="https://virantha.com/2014/05/08/setting-up-new-project-using-google-oauth2-api/">this post</a>,
where you will enter things like application
name, authorized domain (use <a class="reference external" href="http://127.0.0.1:5000">http://127.0.0.1:5000</a>), and the redirect URL where
Google will send you the token via a GET request.</p>
<p>In this case, we're going to use <tt class="docutils literal">redirect.html</tt> file that comes straight from
the docs for <tt class="docutils literal"><span class="pre">ember-oauth2</span></tt>:</p>
<div class="highlight"><pre><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>Authorize<span class="nt"></title></span>
<span class="nt"><script></span>
<span class="kd">var</span> <span class="nx">hash</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">location</span><span class="p">.</span><span class="nx">hash</span><span class="p">;</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">opener</span><span class="p">.</span><span class="nx">App</span><span class="p">.</span><span class="nx">oauth</span><span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="s1">'redirect'</span><span class="p">,</span> <span class="nx">hash</span><span class="p">);</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">close</span><span class="p">();</span>
<span class="nt"></script></span>
<span class="nt"></head></span>
<span class="nt"></html></span>
</pre></div>
<p>Finally, here's the Ember code where we actually do the authentication:</p>
<div class="highlight"><pre><span class="nx">App</span><span class="p">.</span><span class="nx">GoogleAuthenticator</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">SimpleAuth</span><span class="p">.</span><span class="nx">Authenticators</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">restore</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">RSVP</span><span class="p">.</span><span class="nx">Promise</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">resolve</span><span class="p">,</span> <span class="nx">reject</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">Ember</span><span class="p">.</span><span class="nx">isEmpty</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">token</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">resolve</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">reject</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="nx">get_email</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">access_token</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Call the google api with our token to get the user info</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">RSVP</span><span class="p">.</span><span class="nx">Promise</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">resolve</span><span class="p">,</span> <span class="nx">reject</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">Ember</span><span class="p">.</span><span class="nx">$</span><span class="p">.</span><span class="nx">ajax</span><span class="p">({</span>
<span class="nx">url</span><span class="o">:</span> <span class="s1">'https://www.googleapis.com/oauth2/v2/userinfo?access_token='</span><span class="o">+</span><span class="nx">access_token</span><span class="p">,</span>
<span class="nx">type</span><span class="o">:</span> <span class="s1">'GET'</span><span class="p">,</span>
<span class="nx">contentType</span><span class="o">:</span> <span class="s1">'application/json'</span>
<span class="p">}).</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">response</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">resolve</span> <span class="p">(</span><span class="nx">response</span><span class="p">);</span>
<span class="p">},</span> <span class="kd">function</span><span class="p">(</span><span class="nx">xhr</span><span class="p">,</span> <span class="nx">status</span><span class="p">,</span> <span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">error</span><span class="p">);</span>
<span class="nx">reject</span><span class="p">(</span><span class="nx">error</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="nx">authenticate</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">credentials</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">_this</span> <span class="o">=</span> <span class="k">this</span><span class="p">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">RSVP</span><span class="p">.</span><span class="nx">Promise</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">resolve</span><span class="p">,</span> <span class="nx">reject</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Setup handlers</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">oauth</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'success'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">stateObj</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Setup the callback to resolve this function</span>
<span class="nx">token</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">getAccessToken</span><span class="p">();</span>
<span class="c1">// Get all the user info</span>
<span class="nx">_this</span><span class="p">.</span><span class="nx">get_email</span><span class="p">(</span><span class="nx">token</span><span class="p">).</span><span class="nx">then</span><span class="p">(</span>
<span class="kd">function</span><span class="p">(</span><span class="nx">resp</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">resolve</span><span class="p">({</span> <span class="nx">token</span><span class="o">:</span> <span class="nx">token</span><span class="p">,</span>
<span class="nx">userEmail</span><span class="o">:</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">email</span><span class="p">,</span>
<span class="nx">userFn</span><span class="o">:</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">given_name</span><span class="p">,</span>
<span class="nx">userLn</span><span class="o">:</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">family_name</span><span class="p">,</span>
<span class="nx">userPic</span><span class="o">:</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">picture</span><span class="p">,</span>
<span class="nx">userGender</span><span class="o">:</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">gender</span><span class="p">,</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="kd">function</span><span class="p">(</span><span class="nx">rej</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">reject</span><span class="p">(</span><span class="nx">rej</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="p">});</span><span class="c1">// oauth.on</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">oauth</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'error'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span> <span class="nx">reject</span><span class="p">(</span><span class="nx">err</span><span class="p">.</span><span class="nx">error</span><span class="p">);});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">oauth</span><span class="p">.</span><span class="nx">authorize</span><span class="p">();</span>
<span class="p">});</span><span class="c1">// return</span>
<span class="p">},</span>
<span class="nx">invalidate</span><span class="o">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">_this</span> <span class="o">=</span> <span class="k">this</span><span class="p">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">RSVP</span><span class="p">.</span><span class="nx">Promise</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">resolve</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Do something with your API</span>
<span class="nx">resolve</span><span class="p">();</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="p">});</span>
</pre></div>
<p>The main entry point is <tt class="docutils literal">authenticate</tt> which gets triggered with the action in your
handlebars template. Normally, it will submit the form with <tt class="docutils literal">identifiation</tt> (username)
and <tt class="docutils literal">password</tt> fields, which will be sent in via the <tt class="docutils literal">credentials</tt> hash, but for our purposes
we don't need any parameters.</p>
<p>In this function, we basically return a new promise that does the following.</p>
<ol class="arabic simple">
<li>Register a handler for <tt class="docutils literal"><span class="pre">App.oauth.on('success')</span></tt> that gets called by <tt class="docutils literal"><span class="pre">ember-oauth2</span></tt> if the
Google authentication succeeds. At this point, we get the access token, and then using this
token, we call the Google API to get the user information that has been authorized to us. This
call to Google is done in the <tt class="docutils literal">get_email</tt> method that returns a promise. With this information
we set more variables in our session object, such as first name, last name, email, etc.</li>
<li>Register a handler for <tt class="docutils literal"><span class="pre">App.oauth.on('error')</span></tt> where we reject the promise with the error text
that will bubble its way into the <tt class="docutils literal">errorMessage</tt> in the handlebars template on our login route.</li>
</ol>
<p>And finally, we have the <tt class="docutils literal">invalidate</tt> function that gets called when the user hits the <tt class="docutils literal">Logout</tt>
button that clears out the session information.</p>
</div>
</div>
<div class="section" id="summary">
<h3><a class="toc-backref" href="#id12">3.3 Summary</a></h3>
<p>And there you have it: A really concise example of how to use Oauth2 authentication using
implicit grant, all handled in the client-side Ember application! Hope you find this useful!</p>
<div class="note">
<p class="first admonition-title">Note</p>
<p class="last">Please do let me know if you find any mistakes or more efficient ways of doing things. I'm
still fairly new to Ember, so I may have trodden over certain best-known-methods.</p>
</div>
</div>
</div>
</body></html>Detecting color vs greyscale and blank pages during scanning2014-03-30T23:39:00-04:00viranthatag:https://virantha.com,2014-03-30:2014/03/30/images-color-and-blank-detection/<html><body><p>Recently, I <a class="reference external" href="http://virantha.com/category/scanpdf.html">needed</a> a way to detect if a color scanned image was actually color or just contained greyscale/BW
content. I also wanted to detect if an image was "blank", or just a continuous shade of one color. I couldn't
really find a simple solution on the web, so I pieced together my own using ImageMagick, as described below. I
am not an image processing person by any means, but these quick hacks worked well in practice for my needs.</p>
<div class="section" id="detecting-color-vs-greyscale">
<h2>Detecting color vs greyscale</h2>
<p>There are a lot of questions on forums about this (<a class="reference external" href="http://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=19580">1</a>, <a class="reference external" href="http://programmers.stackexchange.com/questions/131067/image-color-grayscale-classification">2</a>), but no solution seemed to work for me. I basically
needed to take a scanned color image, and detect if the original source was just a B&W document that I could
down-convert into a dithered 1-bit document to save space/processing time.</p>
<p>The solution I came up with was similar to <a class="reference external" href="http://www.imagemagick.org/Usage/compare/#type_general">this post</a>
in the ImageMagick documentation</p>
<blockquote>
Another technique is to do a direct 'best fit' of a 3 dimensional line to
all the colors (or a simplified Color Matrix of metrics) in the image. The
error of the fit (generally average of the squares of the errors) gives you
a very good indication about how well the image fits to that line.</blockquote>
<p>The observation is that a pure grey scale image will have colors that
individually have RGB components that are the same (RGB: #222222), so a plot of
greys in 3D space should lie on a single line going through the origin. A
monochromatic tint (sepia) similarly will also lie on a line with different
slope/intercept, and if there are any shade variations, there will be some
errors off the line.</p>
<p>Finding this best-linear-fit in 3D space is overkill for my problem. I don't
need the line, and only need to deal with very small tints from grey (intercept
of the line will be very close to the origin). So I just need to know if some
simple error metric that correlates to my image's colors deviation from the
best fit line is within some threshold. Here's the solution I came up with
that I've verified works well at scanning different types of paper documents:</p>
<ol class="arabic simple">
<li>Quantize the number of colors in the image to some small number like 8.</li>
<li>Quantize to 8-bits in each component.</li>
<li>Generate a frequency table of those colors (optional, I didn't really end up using this)</li>
<li>Take each of the top N colors, and calculate the mean of the differences between the RGB components.</li>
<li>If the mean for any color is larger than some threshold (I used 20), then classify this image as color; if it is less, then I can safely assume the original image was greyscale or black and white.</li>
</ol>
<p>I use ImageMagick for the first 3 actions, using the following command:</p>
<pre class="literal-block">
convert IMAGE -colors 8 -depth 8 -format %c histogram:info:-
</pre>
<p>This generates output like the following (small differences may arise depending on the image format):</p>
<pre class="literal-block">
10831: ( 24, 26, 26,255) #181A1A srgba(24,26,26,1)
4836: ( 55, 87, 79,255) #37574F srgba(55,87,79,1)
6564: ( 77,138,121,255) #4D8A79 srgba(77,138,121,1)
4997: ( 86, 96, 93,255) #56605D srgba(86,96,93,1)
7005: ( 92,153,139,255) #5C998B srgba(92,153,139,1)
2479: (143,118,123,255) #8F767B srgba(143,118,123,1)
8870: (169,176,170,255) #A9B0AA srgba(169,176,170,1)
442906: (254,254,254,255) #FEFEFE srgba(254,254,254,1)
1053: ( 0, 0, 0,255) #000000 black
484081: (255,255,255,255) #FFFFFF white
</pre>
<p>And here's my python code for parsing it (you can find this being used in <a class="reference external" href="https://github.com/virantha/scanpdf/blob/master/scanpdf/scanpdf.py">scanpdf</a>). Note that I just use all
the colors instead of limiting it to the top N.</p>
<div class="highlight"><pre><span class="n">cmd</span> <span class="o">=</span> <span class="s">"convert </span><span class="si">%s</span><span class="s"> -colors 8 -depth 8 -format </span><span class="si">%%</span><span class="s">c histogram:info:-"</span> <span class="o">%</span> <span class="n">filename</span>
<span class="n">out</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cmd</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
<span class="n">mLine</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s">r"""\s*(?P<count>\d+):\s*\(\s*(?P<R>\d+),\s*(?P<G>\d+),\s*(?P<B>\d+).+"""</span><span class="p">)</span>
<span class="n">colors</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">out</span><span class="o">.</span><span class="n">splitlines</span><span class="p">():</span>
<span class="n">matchLine</span> <span class="o">=</span> <span class="n">mLine</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">matchLine</span><span class="p">:</span>
<span class="n">color</span> <span class="o">=</span> <span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="p">(</span><span class="n">matchLine</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="s">'count'</span><span class="p">),</span>
<span class="n">matchLine</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="s">'R'</span><span class="p">),</span>
<span class="n">matchLine</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="s">'G'</span><span class="p">),</span>
<span class="n">matchLine</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="s">'B'</span><span class="p">),</span>
<span class="p">)</span>
<span class="p">]</span>
<span class="n">colors</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">color</span><span class="p">)</span>
<span class="c"># sort</span>
<span class="n">colors</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">reverse</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">key</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">is_color</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span>
<span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span>
<span class="c"># Calculate the mean differences between the RGB components</span>
<span class="c"># Shades of grey will be very close to zero in this metric...</span>
<span class="n">diff</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">sum</span><span class="p">([</span><span class="nb">abs</span><span class="p">(</span><span class="n">color</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="o">-</span><span class="n">color</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span>
<span class="nb">abs</span><span class="p">(</span><span class="n">color</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="o">-</span><span class="n">color</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span>
<span class="nb">abs</span><span class="p">(</span><span class="n">color</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="o">-</span><span class="n">color</span><span class="p">[</span><span class="mi">2</span><span class="p">]),</span>
<span class="p">]))</span><span class="o">/</span><span class="mi">3</span>
<span class="k">if</span> <span class="n">diff</span> <span class="o">></span> <span class="mi">20</span><span class="p">:</span>
<span class="n">is_color</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">return</span> <span class="n">is_color</span>
</pre></div>
</div>
<div class="section" id="detecting-blank-pages">
<h2>Detecting blank pages</h2>
<p>For this, I originally started off with <a class="reference external" href="http://dave.frop.net/blank_page_detection_command_line_imagemagick">post</a>
and used the following command line:</p>
<pre class="literal-block">
convert %s -shave 1%x1% -format "%[fx:mean]" info:
</pre>
<p>For a BW image (with one channel), This returns the average white value after
trimming off the edges of the image. So if the percentage is >0.97, I would say
the page was blank. Unfortunately, this doesn't work for color images as the
channels will have information.</p>
<p>My final idea was to use the image information in each channel, and
specifically the standard deviation, to identify images with information. If
the standard deviation is higher, that means there is more variation in a
particular RGB channel, or in other words, more features in the image. So, I
just look at the standard deviation of each channel and if it's higher than a
threshold, I mark it as a non-blank page. I use the ImageMagick <tt class="docutils literal">identify
<span class="pre">-verbose</span></tt> tool to get this value:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">is_blank</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">c</span> <span class="o">=</span> <span class="s">'identify -verbose </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">filename</span>
<span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cmd</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="n">mStdDev</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s">"""\s*standard deviation:\s*\d+\.\d+\s*\((?P<percent>\d+\.\d+)\).*"""</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">result</span><span class="o">.</span><span class="n">splitlines</span><span class="p">():</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">mStdDev</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">match</span><span class="p">:</span>
<span class="n">stdev</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="s">'percent'</span><span class="p">))</span>
<span class="k">if</span> <span class="n">stdev</span> <span class="o">></span> <span class="mf">0.1</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">False</span>
<span class="k">return</span> <span class="bp">True</span>
</pre></div>
<p>Obviously, any noise (high frequency variation) will also contribute to a
higher stdev, so at some point in the future, I might run some denoise filters
first, but for now I can live with the false positive non-blanks.</p>
<p>That's it for this post. Again, you can see how I used this in my own pdf scanning script, <a class="reference external" href="https://github.com/virantha/scanpdf/blob/master/scanpdf/scanpdf.py">ScanPDF</a>.</p>
</div>
</body></html>One-touch scan enabling in Ubuntu Linux for the Fujitsu ScanSnap S15002014-03-17T12:11:00-04:00viranthatag:https://virantha.com,2014-03-17:2014/03/17/one-touch-scanning-with-fujitsu-scansnap-in-linux/<html><body>
<p>Here's how I managed to get Linux to recognise scan button presses on the
S1500, and automatically initiate a scan to a pdf file. You first
need to download <a class="reference external" href="http://sourceforge.net/projects/scanbd/">Scanbd</a> and install it. Unfortunately, there doesn't seem to be any
apt repository for Scanbd, so I had to download the source and build it. Here's the
minimal setup needed to get this built, based on instructions on the web
<a class="reference external" href="http://thehomeserverhandbook.com/2012/03/03/scanbd_part1">here</a>, followed by updated
info that I pieced together on getting this enabled that didin't require xinetd, etc.</p>
<div class="section" id="building-scanbd">
<h2><a class="toc-backref" href="#id1">1 Building Scanbd</a></h2>
<p>First, as root (<tt class="docutils literal">sudo su -</tt>), install the following packages to allow you to compile Scanbd:</p>
<div class="highlight"><pre>apt-get install build-essential
apt-get install libconfuse-dev libusb-dev libudev-dev libdbus-1-dev dbus libsane-dev
</pre></div>
<p>Next, get the source code for Scanbd and build it:</p>
<div class="highlight"><pre><span class="nb">cd</span> /root
mkdir src
<span class="nb">cd </span>src
wget http://sourceforge.net/projects/scanbd/files/releases/scanbd--1.3.1.zip/download -O scanbd.zip
unzip scanbd.zip
<span class="nb">cd</span> <UNZIPPED DIRECTORY>
./configure
make
make install
</pre></div>
</div>
<div class="section" id="configuring-scanbd">
<h2><a class="toc-backref" href="#id2">2 Configuring Scanbd</a></h2>
<p>Scanbd is now installed into <tt class="docutils literal">/usr/local/sbin/scanbd</tt>, and its configuration files are in
<tt class="docutils literal">/usr/local/etc/scanbd</tt>. Follow similar instructions to <a class="reference external" href="https://wiki.archlinux.org/index.php/Scanner_Button_Daemon">ArchLinux</a>.</p>
<div class="highlight"><pre>cp -R /etc/sane.d /usr/local/etc/scanbd
</pre></div>
<p>Modify <tt class="docutils literal">/etc/sane.d/dll.conf</tt> to only have the <tt class="docutils literal">net</tt> line uncommented, then update the <tt class="docutils literal">/etc/sane.d/net.conf</tt> to
have the following</p>
<pre class="literal-block">
connect_timeout = 3
localhost
</pre>
<p>Now, do the following to startup the daemon that responds to button presses on the S1500</p>
<div class="highlight"><pre><span class="nb">export </span><span class="nv">SANE_CONFIG_DIR</span><span class="o">=</span>/usr/local/etc/scanbd/sane.d/
/usr/local/sbin/scanbd -df -c scanbd.conf
</pre></div>
<p>You can see messages in the syslog <tt class="docutils literal">/var/log/syslog</tt>. After you run the command, it will take about 10 to 15 seconds
to detect and start listening to your scanner.</p>
</div>
<div class="section" id="custom-scan-script">
<h2><a class="toc-backref" href="#id3">3 Custom scan script</a></h2>
<p>The default script that gets executed when the button is pushed simply logs a message to syslog. (the script is
<tt class="docutils literal">/usr/local/etc/scanbd/test.script</tt>). If you look in <tt class="docutils literal">/usr/local/etc/scanbd/scanbd.conf</tt>, you can set test.script
to your own shell script. Here's my trigger script that is called <tt class="docutils literal">scanadf.script</tt>:</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
logger -t <span class="s2">"scanbd: $0"</span> <span class="s2">"Begin of $SCANBD_ACTION for device $SCANBD_DEVICE"</span>
<span class="nv">mydate</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"BLAH"</span> | awk <span class="s1">'{ print strftime("%Y%m%d_%H%M%S") }'</span><span class="k">)</span>
scanpdf --dpi<span class="o">=</span>300 -d scan pdf /home/virantha/scans/scan_<span class="k">${</span><span class="nv">mydate</span><span class="k">}</span>.pdf 2>&1 > /tmp/output.log
logger -t <span class="s2">"scanbd: $0"</span> <span class="s2">"End of $SCANBD_ACTION for device $SCANBD_DEVICE"</span>
</pre></div>
<p>First, it logs the start of the scan to the syslog, and then it gets the current date using awk/strftime
to use for the final pdf filename, and calls my <tt class="docutils literal">scanpdf</tt> python program that handles all the scanning/post processing
to pdf. You can find <a class="reference external" href="http://virantha.github.io/scanpdf/html/">scanpdf</a> documentation online, with source on <a class="reference external" href="https://github.com/virantha/scanpdf">github</a>, and download on <a class="reference external" href="https://pypi.python.org/pypi/scanpdf">PyPI</a>.</p>
<p>The script as written is slightly suboptimal, in that scanpdf does some processing (removing blank pages, etc) that can
take some time, blocking the button response until it's done. A better way would be to spawn a background job to
do this instead:</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
logger -t <span class="s2">"scanbd: $0"</span> <span class="s2">"Begin of $SCANBD_ACTION for device $SCANBD_DEVICE"</span>
<span class="nv">mydate</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"BLAH"</span> | awk <span class="s1">'{ print strftime("%Y%m%d_%H%M%S") }'</span><span class="k">)</span>
<span class="nv">myfile</span><span class="o">=</span><span class="s2">"/home/virantha/scans/scan_${mydate}.pdf"</span>
scanpdf --dpi<span class="o">=</span>300 --tmpdir<span class="o">=</span>/tmp/<span class="k">${</span><span class="nv">mydate</span><span class="k">}</span> scan
scanpdf --dpi<span class="o">=</span>300 --tmpdir<span class="o">=</span>/tmp/<span class="k">${</span><span class="nv">mydate</span><span class="k">}</span> pdf <span class="k">${</span><span class="nv">myfile</span><span class="k">}</span> &
logger -t <span class="s2">"scanbd: $0"</span> <span class="s2">"End of $SCANBD_ACTION for device $SCANBD_DEVICE"</span>
</pre></div>
<p>This way, as soon as the paper is out of the scanner, you can load the next job and hit the scan button immediately;
the long-running post-processing job is run in the background.</p>
</div>
<div class="section" id="starting-scanbd-as-a-service">
<h2><a class="toc-backref" href="#id4">4 Starting Scanbd as a service</a></h2>
<p>Ubuntu uses upstart, so just create a conf file in /etc/init/scanbd like so:</p>
<div class="highlight"><pre><span class="c1"># Scanbd</span>
<span class="err">description</span> <span class="err">"Scanbd</span> <span class="err">service"</span>
<span class="err">author</span> <span class="err">"Virantha</span> <span class="err">Ekanayake"</span>
<span class="c1"># When to start the service</span>
<span class="err">start</span> <span class="err">on</span> <span class="err">runlevel</span> <span class="k">[2345]</span>
<span class="c1"># When to stop the service</span>
<span class="err">stop</span> <span class="err">on</span> <span class="err">runlevel</span> <span class="k">[016]</span>
<span class="c1"># Automatically restart process if crashed</span>
<span class="err">respawn</span>
<span class="c1"># Essentially lets upstart know the process will detach itself to the background</span>
<span class="err">expect</span> <span class="err">fork</span>
<span class="c1"># Specify the process/command to start, e.g.</span>
<span class="na">exec sudo SANE_CONFIG_DIR</span><span class="o">=</span><span class="s">/usr/local/etc/scanbd/sane.d/ /usr/local/sbin/scanbd -df -c /usr/local/etc/scanbd/scanbd.conf</span>
</pre></div>
<p>Then, run the following command to check this service:</p>
<div class="highlight"><pre>initctl reload-configuration
initctl list | grep scanbd
service scanbd start
</pre></div>
<p>Wait about 20 seconds and check:</p>
<div class="highlight"><pre>tail -f /var/log/syslog
</pre></div>
<p>ro make sure it started up. Now, after every boot, your scanbd service will be up and running, watching for button
presses from your scanner.</p>
</div>
</body></html>Starting a simple Flask app with Heroku2013-11-14T21:03:00-05:00viranthatag:https://virantha.com,2013-11-14:2013/11/14/starting-a-simple-flask-app-with-heroku/<html><body><div class="figure align-right" style="width: 383px; height: auto; max-width: 100%;">
<img alt="Flask and Heroku" height="250" src="/images/2013/python_flask_cloud_scaled.jpg" style="width: 383px; height: auto; max-width: 100%;" width="383"/>
</div>
<p>Inspired by <a class="reference external" href="http://shea.io/lightweight-python-apps-with-flask-twitter-bootstrap-and-heroku/">this article</a> on Python Weekly, I thought I'd look into
deploying a simple hello world application to Heroku, built on the Flask
micro framework.</p>
<p>Here's my workflow:</p>
<ul>
<li><p class="first">Create a new directory and Python virtualenv (make sure
virtualenvwrapper is installed!):</p>
<pre class="literal-block">
cd ~/dev
mkdir proj
cd proj
mkvirtualenv proj
</pre>
</li>
<li><p class="first">At this point, I usually edit my virtualenv proj/bin/postactivate
script to cd into proj, since I usually have multiple projects going
on at the same time.</p>
<pre class="literal-block">
#!/bin/tcsh
# This hook is run after this virtualenv is activated.
cd ~/dev/proj
</pre>
</li>
<li><p class="first">Next, install the dependencies:</p>
<ol class="arabic simple">
<li>pip install flask</li>
<li><a class="reference external" href="https://www.heroku.com">Get a Heroku account</a></li>
<li>Install the <a class="reference external" href="https://toolbelt.heroku.com">Heroku tools</a>. This gives you the command-line
`heroku` tool to deploy and run your web app</li>
</ol>
</li>
<li><p class="first">Create the following hello.py web server:</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_1"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12</pre></div></td><td class="code"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="nd">@app.route</span><span class="p">(</span><span class="s">"/"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello</span><span class="p">():</span>
<span class="k">return</span> <span class="s">"Hello world!"</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">port</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"PORT"</span><span class="p">,</span> <span class="mi">5000</span><span class="p">))</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s">'0.0.0.0'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="n">port</span><span class="p">)</span>
</pre></div>
</td></tr></table><p></p><p>Line 11 where we define `port` is very important for
running on Heroku. The default port of 5000 that a Flask app starts
on does not work for a Heroku deployment. Instead, you need to get
the PORT environment variable (that is set in each Heroku deployment
environment, or dyno) and start the app listening on that port. For
running locally for testing, if there is no PORT variable, then we
use 5000</p>
</li>
<li><p class="first">Test your Flask install by running this webserver:</p>
<pre class="literal-block">
python hello.py
</pre>
<p>You should see this in your terminal:</p>
<pre class="literal-block">
* Running on http://0.0.0.0:5000/
</pre>
<p>Now, navigate to localhost:5000 in your browser and make sure you
can see "hello world" printed
out in the browser window.</p>
</li>
<li><p class="first"><strong>Files needed to deploy to Heroku</strong>: Now we're going to add some
files that are required to help Heroku deploy your Flask powered
web-app to its system:</p>
<ol class="arabic">
<li><p class="first"><strong>Python requirements file</strong>: Run the following to output a list
of all your installed python packages</p>
<pre class="literal-block">
pip freeze >! requirements.txt
</pre>
<p>This should generate a file like the following (note, if you did
not have a clean python package installation environment, you may
have more packages listed below):</p>
<pre class="literal-block">
Flask==0.10.1
Jinja2==2.7.1
MarkupSafe==0.18
Werkzeug==0.9.4
itsdangerous==0.23
wsgiref==0.1.2
</pre>
<p></p><p>Although it's usually good practice to change the `==` to
`>=`, so that you're not tied to one particular version of a
package, we'll leave this file as it is now.</p>
</li>
<li><p class="first"><strong>Procfile file</strong>: This is required for Heroku to <a class="reference external" href="https://devcenter.heroku.com/articles/procfile">deploy</a> your
app. Use the following:</p>
<pre class="literal-block">
web: python dealscraper/hello.py
</pre>
<p></p><p>You can test this by running ``foreman web`` and making sure
your web server starts up again.</p>
</li>
</ol>
</li>
<li><p class="first"><strong>Deploy to the cloud!</strong>:
You can now deploy your webserver to Heroku and have it run by
them. Use the following commands to setup the deployment for the
first time:</p>
<pre class="literal-block">
git init
git add hello.py requirements.txt Procfile
git commit -m "Initial commit"
heroku create
heroku apps:rename proj
git push heroku master
</pre>
<p>Note that when you run ``heroku create`` it gives your app a
random name like "serene-meadow-3827", so we want to rename it to our
project name. If all goes well, the first time you do the create, it
will ask your for your Heroku login and password, and then when you
go the git push, it should show something like the following:</p>
<pre class="literal-block">
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 472 bytes | 0 bytes/s, done.
Total 4 (delta 1), reused 0 (delta 0)
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Using Python runtime (python-2.7.4)
-----> Installing dependencies using Pip (1.3.1)
Cleaning up...
-----> Discovering process types
Procfile declares types -> web
-----> Compiled slug size: 28.4MB
-----> Launching... done, v5
http://proj.herokuapp.com deployed to Heroku
To git@heroku.com:proj.git
0ed96f3..dcd13ba master -> master
</pre>
<p></p><p>Whenever you want to deploy a change in your code, just do the
``git commit -m "Message"`` and ``git push heroku master`` to
push your changes to the Heroku infrastructure and restart your
server.</p>
</li>
<li><p class="first">Now, run <tt class="docutils literal">heroku ps:scale web=1</tt> to set your server to use one dyno
or instance. You can check the status of your instance by doing
<tt class="docutils literal">heroku ps</tt> which should yield</p>
<pre class="literal-block">
Scaling web dynos... done, now running 1
</pre>
<p>Check that your process is running by doing ``heroku ps`` which
might yield something like:</p>
<pre class="literal-block">
=== web (1X): `python hello.py`
web.1: up 2013/11/14 15:58:28 (~ 22s ago)
</pre>
<p></p><p>You can check the server logs by doing ``heroku logs``</p>
</li>
<li><p class="first"><strong>Test the server</strong>: Just go to <a class="reference external" href="http://proj.herokuapp.com">http://proj.herokuapp.com</a> to see your
web server in action!</p>
</li>
</ul>
<script charset="utf8" src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script><script charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script><script>
$(document).ready(function() {
$('#unknown_1').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script></body></html>Creating new notes and attaching files using Evernote's Python API2013-11-04T17:53:00-05:00viranthatag:https://virantha.com,2013-11-04:2013/11/04/creating-new-notes-and-attaching-files-using-evernotes-python-api/<html><body><div class="figure align-right" style="width: 282px; height: auto; max-width: 100%;">
<img alt="Documents to Evernote" height="303" src="/images/2013/cloud.jpg" style="width: 282px; height: auto; max-width: 100%;" width="282"/>
</div>
<p>I managed to piece together how to attach a new PDF file to an Evernote
note using their Python API, so I thought it might be useful to have a
post that has all of this information together in one place. I've put
together a complete example that will:</p>
<ol class="arabic simple">
<li>Authenticate to Evernote using a developer token (Oauth2 is a topic
for another day)</li>
<li>Check if a notebook exists; if not, it will create it for you.</li>
<li>Create a new note in the notebook</li>
<li>Attach a PDF file to that note (including calculating the necessary
MD5 hash)</li>
<li>Upload the new note</li>
</ol>
<p>The complete file is shown at the end of this post, and I'll go through
each function separately in the next sections.</p>
<div class="section" id="imports">
<h2><a class="toc-backref" href="#id1">1 Imports</a></h2>
<p>Here's the complete set of imports that took me a while to track down,
even from Evernote's own examples:</p>
<div class="gist">
<script src='https://gist.github.com/7294365.js?file=_imports.py'></script>
<noscript>
<pre><code>import os
import hashlib
import sys
from evernote.api.client import EvernoteClient
import evernote.edam.type.ttypes as Types
import evernote.edam.userstore.constants as UserStoreConstants
from evernote.edam.error.ttypes import EDAMUserException
from evernote.edam.error.ttypes import EDAMSystemException
from evernote.edam.error.ttypes import EDAMNotFoundException
from evernote.edam.error.ttypes import EDAMErrorCode
</code></pre>
</noscript>
</div>
</div>
<div class="section" id="authentication">
<h2><a class="toc-backref" href="#id2">2 Authentication</a></h2>
<p>And here's how to do the authentication using a developer token (Go to
the following places to get a token: <a class="reference external" href="https://sandbox.evernote.com/api/DeveloperToken.action">Sandbox evernote server</a> or <a class="reference external" href="https://www.evernote.com/api/DeveloperToken.action">Production Evernote server</a></p>
<div class="gist">
<script src='https://gist.github.com/7294365.js?file=_auth.py'></script>
<noscript>
<pre><code> def _connect_to_evernote(self, dev_token):
user = None
try:
self.client = EvernoteClient(token=dev_token)
self.user_store = self.client.get_user_store()
user = self.user_store.getUser()
except EDAMUserException as e:
err = e.errorCode
print("Error attempting to authenticate to Evernote: %s - %s" % (EDAMErrorCode._VALUES_TO_NAMES[err], e.parameter))
return False
except EDAMSystemException as e:
err = e.errorCode
print("Error attempting to authenticate to Evernote: %s - %s" % (EDAMErrorCode._VALUES_TO_NAMES[err], e.message))
sys.exit(-1)
if user:
print("Authenticated to evernote as user %s" % user.username)
return True
else:
return False</code></pre>
</noscript>
</div>
<p>The important thing is to keep the EvernoteClient object around in
<tt class="docutils literal">self.client</tt>, as this will proved the authenticated access to the
note stores.</p>
</div>
<div class="section" id="handle-notebooks">
<h2><a class="toc-backref" href="#id3">3 Handle notebooks</a></h2>
<p>The next step is to check whether the required notebook is available,
or if we need to make it. See the _check_and_make_notebook function.</p>
<div class="gist">
<script src='https://gist.github.com/7294365.js?file=_notebook.py'></script>
<noscript>
<pre><code>def _get_notebooks(self):
note_store = self.client.get_note_store()
notebooks = note_store.listNotebooks()
return {n.name:n for n in notebooks}
def _create_notebook(self, notebook):
note_store = self.client.get_note_store()
return note_store.createNotebook(notebook)
def _update_notebook(self, notebook):
note_store = self.client.get_note_store()
note_store.updateNotebook(notebook)
return
def _check_and_make_notebook(self, notebook_name, stack=None):
notebooks = self._get_notebooks()
if notebook_name in notebooks:
# Existing notebook, so just update the stack if needed
notebook = notebooks[notebook_name]
if stack:
notebook.stack = stack
self._update_notebook(notebook)
return notebook
else:
# Need to create a new notebook
notebook = Types.Notebook()
notebook.name = notebook_name
if stack:
notebook.stack = stack
notebook = self._create_notebook(notebook)
return notebook</code></pre>
</noscript>
</div>
<p>We use the <tt class="docutils literal">get_note_store</tt> API call to get all the
notebooks, and return a dict with the notebook name mapping to the
notebook, in function <tt class="docutils literal">_get_notebooks</tt>. Then, if the desired
notebook is present, we update the stack (in Evernote, a notebook can be
in a collection called a "stack" of notebooks) and return the notebook
pointer. If not, we create a new notebook using the <tt class="docutils literal">Types.Notebook()</tt>
call, and store it using the <tt class="docutils literal">createNotebook</tt> API call in the
<tt class="docutils literal">note_store</tt>.</p>
</div>
<div class="section" id="create-the-new-note-with-attachment">
<h2><a class="toc-backref" href="#id4">4 Create the new note with attachment</a></h2>
<p>Next is the real meat of this example, where we create the note with
the attachment:</p>
<div class="gist">
<script src='https://gist.github.com/7294365.js?file=_note.py'></script>
<noscript>
<pre><code> def _create_evernote_note(self, notebook, filename):
# Create the new note
note = Types.Note()
note.title = os.path.basename(filename)
note.notebookGuid = notebook.guid
note.content = '<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">'
note.content += '<en-note>My first PDF upload<br/>'
# Calculate the md5 hash of the pdf
md5 = hashlib.md5()
with open(filename,'rb') as f:
pdf_bytes = f.read()
md5.update(pdf_bytes)
md5hash = md5.hexdigest()
# Create the Data type for evernote that goes into a resource
pdf_data = Types.Data()
pdf_data.bodyHash = md5hash
pdf_data.size = len(pdf_bytes)
pdf_data.body = pdf_bytes
# Add a link in the evernote boy for this content
link = '<en-media type="application/pdf" hash="%s"/>' % md5hash
note.content += link
note.content += '</en-note>'
# Create a resource for the note that contains the pdf
pdf_resource = Types.Resource()
pdf_resource.data = pdf_data
pdf_resource.mime = "application/pdf"
# Create a resource list to hold the pdf resource
resource_list = []
resource_list.append(pdf_resource)
# Set the note's resource list
note.resources = resource_list
return note</code></pre>
</noscript>
</div>
<p>We create a new note using <tt class="docutils literal">Types.Note()</tt>, and set its containing
notebook using the <tt class="docutils literal">GUID</tt> of the notebook. We then start setting the
contents of the note using the Evernote markup language. All the text
and attachment links must be inside the "en-note" tag. The content is
then built up as follows:</p>
<ul class="simple">
<li>Read in the PDF file to attach</li>
<li>Calculate the MD5 hash</li>
<li>Create a new Data container for Evernote and store the hash, size,
and data from the file</li>
<li>Create a link to this file to insert into the content of the note</li>
<li>Create a <tt class="docutils literal">Resource</tt> type to hold the PDF <tt class="docutils literal">Data</tt>, and put it into
a <tt class="docutils literal">Resource</tt> list</li>
<li>Append the resource list to the note</li>
<li>Return this newly formed note</li>
</ul>
</div>
<div class="section" id="uploading-the-note">
<h2><a class="toc-backref" href="#id5">5 Uploading the note</a></h2>
<p>The final step is to upload the note:</p>
<div class="gist">
<script src='https://gist.github.com/7294365.js?file=_upload.py'></script>
<noscript>
<pre><code> def upload_to_notebook(self, filename, notebookname):
# Check if the evernote notebook exists
print ("Checking for notebook named %s" % notebookname)
notebook = self._check_and_make_notebook(notebookname, "my_stack")
print("Uploading %s to %s" % (filename, notebookname))
note = self._create_evernote_note(notebook, filename)
# Store the note in evernote
note_store = self.client.get_note_store()
note = note_store.createNote(note)</code></pre>
</noscript>
</div>
</div>
<div class="section" id="complete-example">
<h2><a class="toc-backref" href="#id6">6 Complete example</a></h2>
<div class="gist">
<script src='https://gist.github.com/7294365.js?file=evernote_pdf_upload.py'></script>
<noscript>
<pre><code>import os
import hashlib
import sys
from evernote.api.client import EvernoteClient
import evernote.edam.type.ttypes as Types
import evernote.edam.userstore.constants as UserStoreConstants
from evernote.edam.error.ttypes import EDAMUserException
from evernote.edam.error.ttypes import EDAMSystemException
from evernote.edam.error.ttypes import EDAMNotFoundException
from evernote.edam.error.ttypes import EDAMErrorCode
class EvernoteUpload(object):
def __init__(self, dev_token):
self._connect_to_evernote(dev_token)
def _connect_to_evernote(self, dev_token):
user = None
try:
self.client = EvernoteClient(token=dev_token)
self.user_store = self.client.get_user_store()
user = self.user_store.getUser()
except EDAMUserException as e:
err = e.errorCode
print("Error attempting to authenticate to Evernote: %s - %s" % (EDAMErrorCode._VALUES_TO_NAMES[err], e.parameter))
return False
except EDAMSystemException as e:
err = e.errorCode
print("Error attempting to authenticate to Evernote: %s - %s" % (EDAMErrorCode._VALUES_TO_NAMES[err], e.message))
sys.exit(-1)
if user:
print("Authenticated to evernote as user %s" % user.username)
return True
else:
return False
def _get_notebooks(self):
note_store = self.client.get_note_store()
notebooks = note_store.listNotebooks()
return {n.name:n for n in notebooks}
def _create_notebook(self, notebook):
note_store = self.client.get_note_store()
return note_store.createNotebook(notebook)
def _update_notebook(self, notebook):
note_store = self.client.get_note_store()
note_store.updateNotebook(notebook)
return
def _check_and_make_notebook(self, notebook_name, stack=None):
notebooks = self._get_notebooks()
if notebook_name in notebooks:
# Existing notebook, so just update the stack if needed
notebook = notebooks[notebook_name]
if stack:
notebook.stack = stack
self._update_notebook(notebook)
return notebook
else:
# Need to create a new notebook
notebook = Types.Notebook()
notebook.name = notebook_name
if stack:
notebook.stack = stack
notebook = self._create_notebook(notebook)
return notebook
def _create_evernote_note(self, notebook, filename):
# Create the new note
note = Types.Note()
note.title = os.path.basename(filename)
note.notebookGuid = notebook.guid
note.content = '<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">'
note.content += '<en-note>My first PDF upload<br/>'
# Calculate the md5 hash of the pdf
md5 = hashlib.md5()
with open(filename,'rb') as f:
pdf_bytes = f.read()
md5.update(pdf_bytes)
md5hash = md5.hexdigest()
# Create the Data type for evernote that goes into a resource
pdf_data = Types.Data()
pdf_data.bodyHash = md5hash
pdf_data.size = len(pdf_bytes)
pdf_data.body = pdf_bytes
# Add a link in the evernote boy for this content
link = '<en-media type="application/pdf" hash="%s"/>' % md5hash
note.content += link
note.content += '</en-note>'
# Create a resource for the note that contains the pdf
pdf_resource = Types.Resource()
pdf_resource.data = pdf_data
pdf_resource.mime = "application/pdf"
# Create a resource list to hold the pdf resource
resource_list = []
resource_list.append(pdf_resource)
# Set the note's resource list
note.resources = resource_list
return note
def upload_to_notebook(self, filename, notebookname):
# Check if the evernote notebook exists
print ("Checking for notebook named %s" % notebookname)
notebook = self._check_and_make_notebook(notebookname, "my_stack")
print("Uploading %s to %s" % (filename, notebookname))
note = self._create_evernote_note(notebook, filename)
# Store the note in evernote
note_store = self.client.get_note_store()
note = note_store.createNote(note)
if __name__ == '__main__':
dev_token = "YOUR_DEV_TOKEN"
p = EvernoteUpload(dev_token)
p.upload_to_notebook('test_sherlock.pdf', 'boom')
</code></pre>
</noscript>
</div>
</div>
</body></html>Quick web app with Go, Ember.js, and MongoDB2013-09-29T19:17:00-04:00viranthatag:https://virantha.com,2013-09-29:2013/09/29/quick-web-app-with-go-ember-js-and-mongodb/<html><body><div class="figure align-left" style="width: 236px">
<img alt=" Screenshot" src="/images/2013/kittens_app_screenshot_scaled.png" style="width: 236px;"/>
</div>
<p>As part of my recent foray into the <a class="reference external" href="http://golang.org/">Go</a> language, I stumbled across
this excellent example by Nerdyworm <a class="reference external" href="http://nerdyworm.com/blog/2013/05/21/building-an-app-with-ember-dot-js-and-go/">part 1</a>, <a class="reference external" href="http://nerdyworm.com/blog/2013/05/24/building-an-app-with-ember-dot-js-and-go-part-2/">part 2</a>, <a class="reference external" href="http://nerdyworm.com/blog/2013/05/27/building-an-app-with-ember-dot-js-and-go-part-3/">part 3</a> that
shows how to build a small web application using the <a class="reference external" href="http://emberjs.com/">ember.js</a>
javascript framework and Go providing the backend server. I took this
example, fixed up some incompatibilites with the latest version of
ember-data.js, and added support for a backing store using MongoDB. You
can find all the code for this at my <a class="reference external" href="http://github.com/virantha/kittens_ember_go">github</a>, while I describe some of
the more interesting stuff below.</p>
<div class="section" id="getting-started">
<h2><a class="toc-backref" href="#id1">1 Getting started</a></h2>
<div class="section" id="get-my-code">
<h3><a class="toc-backref" href="#id2">1.1 Get my code</a></h3>
<div class="highlight"><pre>git clone https://github.com/virantha/kittens_ember_go.git
<span class="nb">cd </span>kittens_ember_go
setenv GOPATH <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/go/src
</pre></div>
</div>
<div class="section" id="depedencies">
<h3><a class="toc-backref" href="#id3">1.2 Depedencies</a></h3>
<p>I'm assuming you have Go already installed:</p>
<div class="section" id="mongodb">
<h4><a class="toc-backref" href="#id4">1.2.1 MongoDB</a></h4>
<p>Follow the instructions at <a class="reference external" href="http://docs.mongodb.org/manual/installation/">Mongo</a> to install for your platform. Make
sure you can start the Mongo server afterwards and insert some records
into a test database to validate your install.</p>
<p>Then, you need to install the Go driver for Mongodb. First, you need to
get <a class="reference external" href="http://docs.mongodb.org/manual/installation/">Canonical's Bazaar</a> revision control system. Make sure you can
type ''bzr'' at the command-line to verify install. Then, run the
following command to install <a class="reference external" href="http://labix.org/mgo">Mgo</a></p>
<pre class="literal-block">
# Install mgo, the mongodb driver for go
# Make sure your GOPATH is set as above
go get labix.org/v2/mgo
</pre>
</div>
<div class="section" id="gorilla">
<h4><a class="toc-backref" href="#id5">1.2.2 Gorilla</a></h4>
<p>Install this muxing package for Go:</p>
<pre class="literal-block">
go get github.com/gorilla/mux
</pre>
</div>
</div>
</div>
<div class="section" id="what-we-re-going-to-be-building">
<h2><a class="toc-backref" href="#id6">2 What we're going to be building</a></h2>
<p>We'll build a small web application that you can open directly in your
browser. It will do exactly what Nerdyworm built in his original, except
that this version adds data persistence in a database. The app presents
the following interface:</p>
<div class="figure" style="width: 847px; height: auto; max-width: 100%;">
<img alt="Screenshot of kittens app" height="554" src="/images/2013/kittens_app_screenshot.png" style="width: 847px; height: auto; max-width: 100%;" width="847"/>
</div>
<p>It will let you create a "kitten", linking
to a random kitten image. You can then either edit the kitten name, or
delete the kitten, with all kittens stored in the backing MongoDB
database. The web interface is done in Ember.js, and Go provides the
web-server/database access.</p>
<div class="section" id="running-the-app">
<h3><a class="toc-backref" href="#id7">2.1 Running the app</a></h3>
<dl class="docutils">
<dt>Once you have everything installed, you can just do the following to start</dt>
<dd>the server:</dd>
</dl>
<pre class="literal-block">
cd site
go run ../go/src/github.com/virantha/server_mongo.go
</pre>
<p>Then navigate to <a class="reference external" href="http://localhost:8081/#/create">http://localhost:8081</a></p>
</div>
</div>
<div class="section" id="code-discussion">
<h2><a class="toc-backref" href="#id8">3 Code discussion</a></h2>
<p>First, I'll discuss the server-side Go program that accesses the Mongo
database and provides JSON objects for the front-end. That's not to say
the client-side is not interesting; quite the contrary, it's pretty
impressive what one can do with ember.js, but most of that has already
been covered in Nerdyworm's posts.</p>
<div class="section" id="the-backend-go-server-server-mongo-go">
<h3><a class="toc-backref" href="#id9">3.1 The Backend Go Server (server_mongo.go)</a></h3>
<p>The server is just one single go file; here's the list of imports:</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_2"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11</pre></div></td><td class="code"><div class="highlight"><pre><span class="kn">package</span> <span class="nx">main</span>
<span class="kn">import</span> <span class="p">(</span>
<span class="s">"net/http"</span>
<span class="s">"log"</span>
<span class="s">"github.com/gorilla/mux"</span>
<span class="s">"encoding/json"</span>
<span class="s">"math/rand"</span>
<span class="s">"fmt"</span>
<span class="s">"labix.org/v2/mgo"</span>
<span class="s">"labix.org/v2/mgo/bson"</span>
<span class="p">)</span>
</pre></div>
</td></tr></table><div class="section" id="main">
<h4><a class="toc-backref" href="#id10">3.1.1 Main</a></h4>
<p>Here's the main function where we set up the URL routes for the
application, connect to the database, and start the web server.</p>
<p>Lines 6 to 12 setup the routes for "getting" kittens for display,
"creating" a new kitten, "updating" an existing kitten, and "deleting" a
kitten. This is all unchanged from the original source code.</p>
<p>Lines 16 to 20 are new, and handle the MongoDB connection. We open a new
session and then connect to the kittens collection. These are global
variables for this app, explained in the next section.</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_3"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24</pre></div></td><td class="code"><div class="highlight"><pre><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
<span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"Starting Server"</span><span class="p">)</span>
<span class="nx">r</span> <span class="o">:=</span> <span class="nx">mux</span><span class="p">.</span><span class="nx">NewRouter</span><span class="p">()</span>
<span class="nx">r</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api/kittens"</span><span class="p">,</span> <span class="nx">KittensHandler</span><span class="p">).</span><span class="nx">Methods</span><span class="p">(</span><span class="s">"GET"</span><span class="p">)</span>
<span class="nx">r</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api/kittens"</span><span class="p">,</span> <span class="nx">CreateKittenHandler</span><span class="p">).</span><span class="nx">Methods</span><span class="p">(</span><span class="s">"POST"</span><span class="p">)</span>
<span class="nx">r</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api/kittens/{id}"</span><span class="p">,</span> <span class="nx">UpdateKittenHandler</span><span class="p">).</span><span class="nx">Methods</span><span class="p">(</span><span class="s">"PUT"</span><span class="p">)</span>
<span class="nx">r</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api/kittens/{id}"</span><span class="p">,</span> <span class="nx">DeleteKittenHandler</span><span class="p">).</span><span class="nx">Methods</span><span class="p">(</span><span class="s">"DELETE"</span><span class="p">)</span>
<span class="nx">http</span><span class="p">.</span><span class="nx">Handle</span><span class="p">(</span><span class="s">"/api/"</span><span class="p">,</span> <span class="nx">r</span><span class="p">)</span>
<span class="nx">http</span><span class="p">.</span><span class="nx">Handle</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="nx">http</span><span class="p">.</span><span class="nx">FileServer</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Dir</span><span class="p">(</span><span class="s">"."</span><span class="p">)))</span>
<span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"Starting mongo db session"</span><span class="p">)</span>
<span class="nx">session</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">mgo</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"localhost"</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="nx">panic</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">}</span>
<span class="k">defer</span> <span class="nx">session</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">SetMode</span><span class="p">(</span><span class="nx">mgo</span><span class="p">.</span><span class="nx">Monotonic</span><span class="p">,</span> <span class="kc">true</span><span class="p">)</span>
<span class="nx">collection</span> <span class="p">=</span> <span class="nx">session</span><span class="p">.</span><span class="nx">DB</span><span class="p">(</span><span class="s">"Kittens"</span><span class="p">).</span><span class="nx">C</span><span class="p">(</span><span class="s">"kittens"</span><span class="p">)</span>
<span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"Listening on 8080"</span><span class="p">)</span>
<span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="s">":8080"</span><span class="p">,</span> <span class="kc">nil</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
<div class="section" id="globals-and-types">
<h4><a class="toc-backref" href="#id11">3.1.2 Globals and types</a></h4>
<p>Here's the list of custom types defined for our application. First, we
have the database pointers to the session and collection.</p>
<p>Then we have the struct for a single Kitten, which contains an Id of
type ObjectId (used as unique identifier in the MongoDB), the name of
the kitten, and a string with the URL for the picture. Note the string
literals (the backticks are just to avoid backslashing the
double-quotes) at the end of each member type that specifies a <strong>tag</strong>
for each member that will be used as a name for JSON or BSON
representations instead of the member name. Also note how we can use
_id for the BSON representation for mongo (which is the default unique
identifier in Mongo for each element), while still keeping id for the
JSON, and capitalized Id for making it an exported member in Go.</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_4"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19</pre></div></td><td class="code"><div class="highlight"><pre><span class="kd">var</span> <span class="p">(</span>
<span class="nx">session</span> <span class="o">*</span><span class="nx">mgo</span><span class="p">.</span><span class="nx">Session</span>
<span class="nx">collection</span> <span class="o">*</span><span class="nx">mgo</span><span class="p">.</span><span class="nx">Collection</span>
<span class="p">)</span>
<span class="kd">type</span> <span class="nx">Kitten</span> <span class="kd">struct</span> <span class="p">{</span>
<span class="nx">Id</span> <span class="nx">bson</span><span class="p">.</span><span class="nx">ObjectId</span> <span class="s">`bson:"_id" json:"id"`</span>
<span class="nx">Name</span> <span class="kt">string</span> <span class="s">`json:"name"`</span>
<span class="nx">Picture</span> <span class="kt">string</span> <span class="s">`json:"picture"`</span>
<span class="p">}</span>
<span class="kd">type</span> <span class="nx">KittenJSON</span> <span class="kd">struct</span> <span class="p">{</span>
<span class="nx">Kitten</span> <span class="nx">Kitten</span> <span class="s">`json:"kitten"`</span>
<span class="p">}</span>
<span class="kd">type</span> <span class="nx">KittensJSON</span> <span class="kd">struct</span> <span class="p">{</span>
<span class="nx">Kittens</span> <span class="p">[]</span><span class="nx">Kitten</span> <span class="s">`json:"kittens"`</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
<div class="section" id="the-handlers">
<h4><a class="toc-backref" href="#id12">3.1.3 The handlers</a></h4>
<p>We have a set of 4 handlers for each of the actions the client-side
application might send our way:</p>
<div class="section" id="create-a-kitten">
<h5><a class="toc-backref" href="#id13">3.1.3.1 Create a kitten</a></h5>
<p>In lines 5 to 9, we decode the incoming kitten from JSON. Next, in lines
10 to 15, we pick a random image width/height and to get the
corresponding kitten image from placekitten.com.</p>
<p>Finally, in linees 19 to 26, we create a new Id, and insert this kitten
instance into our database, after which we return a JSON object of the
kitten to render in the client.</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_5"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33</pre></div></td><td class="code"><div class="highlight"><pre><span class="kd">func</span> <span class="nx">CreateKittenHandler</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">r</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">kittenJSON</span> <span class="nx">KittenJSON</span>
<span class="nx">err</span> <span class="o">:=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewDecoder</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">Body</span><span class="p">).</span><span class="nx">Decode</span><span class="p">(</span><span class="o">&</span><span class="nx">kittenJSON</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">}</span>
<span class="nx">kitten</span> <span class="o">:=</span> <span class="nx">kittenJSON</span><span class="p">.</span><span class="nx">Kitten</span>
<span class="c1">// Generate a random dimension for the kitten</span>
<span class="nx">width</span> <span class="o">:=</span> <span class="nx">rand</span><span class="p">.</span><span class="nx">Int</span><span class="p">()</span> <span class="o">%</span> <span class="mi">400</span>
<span class="nx">height</span> <span class="o">:=</span> <span class="nx">rand</span><span class="p">.</span><span class="nx">Int</span><span class="p">()</span> <span class="o">%</span> <span class="mi">400</span>
<span class="k">if</span> <span class="nx">width</span> <span class="p"><</span> <span class="mi">100</span> <span class="p">{</span> <span class="nx">width</span> <span class="o">+=</span> <span class="mi">100</span> <span class="p">}</span>
<span class="k">if</span> <span class="nx">height</span> <span class="p"><</span> <span class="mi">100</span> <span class="p">{</span> <span class="nx">height</span> <span class="o">+=</span> <span class="mi">100</span><span class="p">}</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">Picture</span> <span class="p">=</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"http://placekitten.com/%d/%d"</span><span class="p">,</span> <span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">)</span>
<span class="c1">// Store the new kitten in the database</span>
<span class="c1">// First, let's get a new id</span>
<span class="nx">obj_id</span> <span class="o">:=</span> <span class="nx">bson</span><span class="p">.</span><span class="nx">NewObjectId</span><span class="p">()</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">Id</span> <span class="p">=</span> <span class="nx">obj_id</span>
<span class="nx">err</span> <span class="p">=</span> <span class="nx">collection</span><span class="p">.</span><span class="nx">Insert</span><span class="p">(</span><span class="o">&</span><span class="nx">kitten</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"Inserted new kitten %s with name %s"</span><span class="p">,</span> <span class="nx">kitten</span><span class="p">.</span><span class="nx">Id</span><span class="p">,</span> <span class="nx">kitten</span><span class="p">.</span><span class="nx">Name</span><span class="p">)</span>
<span class="p">}</span>
<span class="nx">j</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">KittenJSON</span><span class="p">{</span><span class="nx">Kitten</span><span class="p">:</span> <span class="nx">kitten</span><span class="p">})</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">}</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span> <span class="s">"application/json"</span><span class="p">)</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">Write</span><span class="p">(</span><span class="nx">j</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
<div class="section" id="display-the-kittens">
<h5><a class="toc-backref" href="#id14">3.1.3.2 Display the kittens</a></h5>
<p>Here, we just return an array of kitten objects from the database for
display in the client. We iterate through the kittens collection and
build up a local slice of kittens (I admit, this is pretty inefficient
and we'd probably want to cache this in a real application) that then
gets marshaled into JSON for display.</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_6"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18</pre></div></td><td class="code"><div class="highlight"><pre><span class="kd">func</span> <span class="nx">KittensHandler</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">r</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Let's build up the kittens slice</span>
<span class="kd">var</span> <span class="nx">mykittens</span> <span class="p">[]</span><span class="nx">Kitten</span>
<span class="nx">iter</span> <span class="o">:=</span> <span class="nx">collection</span><span class="p">.</span><span class="nx">Find</span><span class="p">(</span><span class="kc">nil</span><span class="p">).</span><span class="nx">Iter</span><span class="p">()</span>
<span class="nx">result</span> <span class="o">:=</span> <span class="nx">Kitten</span><span class="p">{}</span>
<span class="k">for</span> <span class="nx">iter</span><span class="p">.</span><span class="nx">Next</span><span class="p">(</span><span class="o">&</span><span class="nx">result</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">mykittens</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">mykittens</span><span class="p">,</span> <span class="nx">result</span><span class="p">)</span>
<span class="p">}</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span> <span class="s">"application/json"</span><span class="p">)</span>
<span class="nx">j</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">KittensJSON</span><span class="p">{</span><span class="nx">Kittens</span><span class="p">:</span> <span class="nx">mykittens</span><span class="p">})</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="nx">panic</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">}</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">Write</span><span class="p">(</span><span class="nx">j</span><span class="p">)</span>
<span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"Provided json"</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
<div class="section" id="delete-a-kitten">
<h5><a class="toc-backref" href="#id15">3.1.3.3 Delete a kitten</a></h5>
<p>This is probably the most trivial to write. The only "gotcha" (if you
can even call it that) is to convert the incoming string id into an
actual ObjectId object for lookup in the MongoDB.</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_7"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11</pre></div></td><td class="code"><div class="highlight"><pre><span class="kd">func</span> <span class="nx">DeleteKittenHandler</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">r</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Grab the kitten's id from the incoming url</span>
<span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
<span class="nx">vars</span> <span class="o">:=</span> <span class="nx">mux</span><span class="p">.</span><span class="nx">Vars</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
<span class="nx">id</span> <span class="o">:=</span> <span class="nx">vars</span><span class="p">[</span><span class="s">"id"</span><span class="p">]</span>
<span class="c1">// Remove it from database</span>
<span class="nx">err</span> <span class="p">=</span> <span class="nx">collection</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">bson</span><span class="p">.</span><span class="nx">M</span><span class="p">{</span><span class="s">"_id"</span><span class="p">:</span><span class="nx">bson</span><span class="p">.</span><span class="nx">ObjectIdHex</span><span class="p">(</span><span class="nx">id</span><span class="p">)})</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"Could not find kitten %s to delete"</span><span class="p">,</span> <span class="nx">id</span><span class="p">)}</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">WriteHeader</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusNoContent</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
<div class="section" id="update-a-kitten-name">
<h5><a class="toc-backref" href="#id16">3.1.3.4 Update a kitten name</a></h5>
<p>Again, pretty straightforward:</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_8"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22</pre></div></td><td class="code"><div class="highlight"><pre><span class="kd">func</span> <span class="nx">UpdateKittenHandler</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">r</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
<span class="c1">// Grab the kitten's id from the incoming url</span>
<span class="nx">vars</span> <span class="o">:=</span> <span class="nx">mux</span><span class="p">.</span><span class="nx">Vars</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
<span class="nx">id</span> <span class="o">:=</span> <span class="nx">bson</span><span class="p">.</span><span class="nx">ObjectIdHex</span><span class="p">(</span><span class="nx">vars</span><span class="p">[</span><span class="s">"id"</span><span class="p">])</span>
<span class="c1">// Decode the incoming kitten json</span>
<span class="kd">var</span> <span class="nx">kittenJSON</span> <span class="nx">KittenJSON</span>
<span class="nx">err</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewDecoder</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">Body</span><span class="p">).</span><span class="nx">Decode</span><span class="p">(</span><span class="o">&</span><span class="nx">kittenJSON</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)}</span>
<span class="c1">// Update the database</span>
<span class="nx">err</span> <span class="p">=</span> <span class="nx">collection</span><span class="p">.</span><span class="nx">Update</span><span class="p">(</span><span class="nx">bson</span><span class="p">.</span><span class="nx">M</span><span class="p">{</span><span class="s">"_id"</span><span class="p">:</span><span class="nx">id</span><span class="p">},</span>
<span class="nx">bson</span><span class="p">.</span><span class="nx">M</span><span class="p">{</span><span class="s">"name"</span><span class="p">:</span><span class="nx">kittenJSON</span><span class="p">.</span><span class="nx">Kitten</span><span class="p">.</span><span class="nx">Name</span><span class="p">,</span>
<span class="s">"_id"</span><span class="p">:</span> <span class="nx">id</span><span class="p">,</span>
<span class="s">"picture"</span><span class="p">:</span> <span class="nx">kittenJSON</span><span class="p">.</span><span class="nx">Kitten</span><span class="p">.</span><span class="nx">Picture</span><span class="p">,</span>
<span class="p">})</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"Updated kitten %s name to %s"</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">kittenJSON</span><span class="p">.</span><span class="nx">Kitten</span><span class="p">.</span><span class="nx">Name</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">}</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">WriteHeader</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusNoContent</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td></tr></table><p>And that's pretty much it for the back-end server. Go is pretty concise!</p>
</div>
</div>
</div>
<div class="section" id="frontend-ember-js-app">
<h3><a class="toc-backref" href="#id17">3.2 Frontend ember.js app</a></h3>
<div class="section" id="html-output-template">
<h4><a class="toc-backref" href="#id18">3.2.1 HTML output template</a></h4>
<p>First, here's the html that the ember framework will write into. A lot
of it is just boilerplate and there's nothing really new here from the
original source that's been discussed by nerdyworm:</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_9"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63</pre></div></td><td class="code"><div class="highlight"><pre><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></span>
<span class="nt"><title></span>Kittens<span class="nt"></title></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"css/normalize.css"</span><span class="nt">></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"css/style.css"</span><span class="nt">></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/css/bootstrap-combined.no-icons.min.css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span><span class="nt">></span>
<span class="nt"></head></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"container"</span><span class="o">></span>
<span class="o"><</span><span class="nx">h1</span><span class="o">></span><span class="nx">Create</span> <span class="nx">your</span> <span class="nx">own</span> <span class="nx">kitten</span><span class="o"><</span><span class="err">/h1></span>
<span class="o"><</span><span class="nx">ul</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"nav nav-tabs"</span><span class="o">></span>
<span class="o"><</span><span class="nx">li</span><span class="o">></span> <span class="p">{{</span><span class="err">#</span><span class="nx">linkTo</span> <span class="s1">'index'</span><span class="p">}}</span><span class="nx">Index</span><span class="p">{{</span><span class="err">/linkTo}}</li></span>
<span class="o"><</span><span class="nx">li</span><span class="o">></span> <span class="p">{{</span><span class="err">#</span><span class="nx">linkTo</span> <span class="s1">'create'</span><span class="p">}}</span><span class="nx">Create</span><span class="p">{{</span><span class="err">/linkTo}}</li></span>
<span class="o"><</span><span class="err">/ul></span>
<span class="p">{{</span><span class="nx">outlet</span><span class="p">}}</span>
<span class="o"><</span><span class="err">/div></span>
<span class="nt"></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span> <span class="na">data-template-name=</span><span class="s">"index"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">ul</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"thumbnails"</span><span class="o">></span>
<span class="p">{{</span><span class="err">#</span><span class="nx">each</span> <span class="nx">controller</span><span class="p">}}</span>
<span class="o"><</span><span class="nx">li</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"span3"</span><span class="o">></span>
<span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"thumbnail"</span><span class="o">></span>
<span class="o"><</span><span class="nx">img</span> <span class="p">{{</span><span class="nx">bindAttr</span> <span class="nx">src</span><span class="o">=</span><span class="s2">"picture"</span><span class="p">}}</span><span class="o">/></span>
<span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"caption"</span><span class="o">></span>
<span class="o"><</span><span class="nx">h3</span><span class="o">></span><span class="p">{{</span><span class="nx">name</span><span class="p">}}</span><span class="o"><</span><span class="err">/h3></span>
<span class="p">{{</span><span class="err">#</span><span class="nx">linkTo</span> <span class="s1">'edit'</span> <span class="k">this</span><span class="p">}}</span><span class="nx">Edit</span><span class="p">{{</span><span class="err">/linkTo}}</span>
<span class="o"><</span><span class="nx">button</span> <span class="p">{{</span><span class="nx">action</span> <span class="nx">deleteKitten</span> <span class="k">this</span><span class="p">}}</span><span class="o">></span><span class="nx">Delete</span><span class="o"><</span><span class="err">/button></span>
<span class="o"><</span><span class="err">/div></span>
<span class="o"><</span><span class="err">/div></span>
<span class="o"><</span><span class="err">/li></span>
<span class="p">{{</span><span class="err">/each}}</span>
<span class="o"><</span><span class="err">/ul></span>
<span class="nt"></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span> <span class="na">data-template-name=</span><span class="s">"_form"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">form</span> <span class="p">{{</span><span class="nx">action</span> <span class="nx">save</span> <span class="nx">on</span><span class="o">=</span><span class="s2">"submit"</span><span class="p">}}</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"form-inline"</span><span class="o">></span>
<span class="p">{{</span><span class="nx">input</span> <span class="nx">type</span><span class="o">=</span><span class="s2">"text"</span> <span class="nx">value</span><span class="o">=</span><span class="nx">name</span><span class="p">}}</span>
<span class="o"><</span><span class="nx">button</span> <span class="nx">type</span><span class="o">=</span><span class="s2">"submit"</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"btn btn-primary"</span><span class="o">></span><span class="nx">Save</span><span class="o"><</span><span class="err">/button></span>
<span class="o"><</span><span class="err">/form></span>
<span class="nt"></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span> <span class="na">data-template-name=</span><span class="s">"create"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">h1</span><span class="o">></span><span class="nx">Create</span> <span class="nx">kitten</span><span class="o"><</span><span class="err">/h1></span>
<span class="p">{{</span><span class="nx">partial</span> <span class="s1">'form'</span><span class="p">}}</span>
<span class="nt"></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/x-handlebars"</span> <span class="na">data-template-name=</span><span class="s">"edit"</span><span class="nt">></span>
<span class="o"><</span><span class="nx">h1</span><span class="o">></span><span class="nx">Edit</span> <span class="nx">kitten</span><span class="o"><</span><span class="err">/h1></span>
<span class="p">{{</span><span class="nx">partial</span> <span class="s1">'form'</span><span class="p">}}</span>
<span class="nt"></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"js/lib/jquery.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"js/lib/handlebars.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"js/lib/ember.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"js/lib/ember-data.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"js/app_kittens.js"</span><span class="nt">></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</pre></div>
</td></tr></table></div>
<div class="section" id="ember-js-application">
<h4><a class="toc-backref" href="#id19">3.2.2 Ember.js application</a></h4>
<p>Now, here is the actual client-side logic that's been updated to work
with the latest version of ember and ember-data:</p>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_10"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62</pre></div></td><td class="code"><div class="highlight"><pre><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">App</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Application</span><span class="p">.</span><span class="nx">create</span><span class="p">();</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">Store</span> <span class="o">=</span> <span class="nx">DS</span><span class="p">.</span><span class="nx">Store</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">revision</span><span class="o">:</span> <span class="mi">12</span><span class="p">,</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">Kitten</span> <span class="o">=</span> <span class="nx">DS</span><span class="p">.</span><span class="nx">Model</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">name</span><span class="o">:</span> <span class="nx">DS</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'string'</span><span class="p">),</span>
<span class="nx">picture</span><span class="o">:</span> <span class="nx">DS</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'string'</span><span class="p">),</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">KittenAdapter</span> <span class="o">=</span> <span class="nx">DS</span><span class="p">.</span><span class="nx">RESTAdapter</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">namespace</span><span class="o">:</span> <span class="s1">'api'</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">IndexRoute</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Route</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">model</span><span class="o">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">store</span><span class="p">.</span><span class="nx">find</span><span class="p">(</span><span class="s1">'kitten'</span><span class="p">);</span>
<span class="p">},</span>
<span class="nx">actions</span><span class="o">:</span> <span class="p">{</span>
<span class="nx">deleteKitten</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">kitten</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">deleteRecord</span><span class="p">();</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">save</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="c1">// Add some routes</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">Router</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">route</span><span class="p">(</span><span class="s1">'create'</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">route</span><span class="p">(</span><span class="s1">'edit'</span><span class="p">,</span> <span class="p">{</span><span class="nx">path</span><span class="o">:</span> <span class="s1">'/edit/:kitten_id'</span><span class="p">});</span>
<span class="p">});</span>
<span class="c1">// Add a controller to dispatch create commands</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">CreateController</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">Controller</span><span class="p">.</span><span class="nx">extend</span><span class="p">(</span> <span class="p">{</span>
<span class="nx">name</span><span class="o">:</span> <span class="kc">null</span><span class="p">,</span>
<span class="nx">actions</span><span class="o">:</span> <span class="p">{</span>
<span class="nx">save</span><span class="o">:</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">kitten</span> <span class="o">=</span><span class="k">this</span><span class="p">.</span><span class="nx">store</span><span class="p">.</span><span class="nx">createRecord</span><span class="p">(</span><span class="s1">'kitten'</span><span class="p">);</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">'name'</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">'name'</span><span class="p">));</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">save</span><span class="p">().</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">transitionToRoute</span><span class="p">(</span><span class="s1">'index'</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">''</span><span class="p">);</span>
<span class="p">}.</span><span class="nx">bind</span><span class="p">(</span><span class="k">this</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">App</span><span class="p">.</span><span class="nx">EditController</span> <span class="o">=</span> <span class="nx">Ember</span><span class="p">.</span><span class="nx">ObjectController</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="nx">actions</span><span class="o">:</span> <span class="p">{</span>
<span class="nx">save</span><span class="o">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">kitten</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">'model'</span><span class="p">);</span>
<span class="nx">kitten</span><span class="p">.</span><span class="nx">save</span><span class="p">().</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">transitionToRoute</span><span class="p">(</span><span class="s1">'index'</span><span class="p">);</span>
<span class="p">}.</span><span class="nx">bind</span><span class="p">(</span><span class="k">this</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">})();</span>
</pre></div>
</td></tr></table></div>
</div>
</div>
<script charset="utf8" src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script><script charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script><script>
$(document).ready(function() {
$('#unknown_2').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_3').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_4').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_5').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_6').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_7').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_8').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_9').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script><script>
$(document).ready(function() {
$('#unknown_10').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script></body></html>Reading and writing Microsoft Word docx files with Python2013-08-16T03:16:00-04:00viranthatag:https://virantha.com,2013-08-16:2013/08/16/reading-and-writing-microsoft-word-docx-files-with-python/<html><body><div class="figure align-right">
<img alt=" python" src="/images/2013/python_scaled.png" style="width: 250px;"/>
</div>
<p>I've been wanting to script simple text scanning and substitution in
Microsoft Word documents for a while now, and after a little digging, it
turns out, it's fairly straight-forward to read and edit .docx (OpenXML)
or the <a class="reference external" href="http://www.ecma-international.org/publications/standards/Ecma-376.htm">ECMA-376</a> original standard, and now under ISO as <a class="reference external" href="http://www.iso.org/iso/catalogue_detail?csnumber=51463">ISO/IEC
29500</a>. Although I couldn't find a general python library that provides
a nice API for this, I was thankfully able to follow the examples in the
<a class="reference external" href="https://github.com/mikemaccana/python-docx">python-docx</a> to understand what was going on and get my script done.
In this post, I'll describe the structure of this file format and how to
access it easily in python.</p>
<p>I've also used these techniques in my other project, <a class="reference external" href="https://virantha.com/2015/04/07/oneresume-a-data-driven-resume-generator-for-microsoft-word/">OneResumé</a>, a data-driven
resume generator for MS Word documents.</p>
<div class="section" id="getting-the-text-content">
<h2><a class="toc-backref" href="#id1">1 Getting the text content</a></h2>
<p>At its heart, a docx file is just a zip file (try running unzip on it!)
containing a bunch of well defined XML and collateral files. The main
textual content and structure is defined in the following XML file:</p>
<pre class="literal-block">
word/document.xml
</pre>
<p>So the first step is to read this zip container and get the xml:</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">zipfile</span>
<span class="k">def</span> <span class="nf">get_word_xml</span><span class="p">(</span><span class="n">docx_filename</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">docx_filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="nb">zip</span> <span class="o">=</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">xml_content</span> <span class="o">=</span> <span class="nb">zip</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s">'word/document.xml'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">xml_content</span>
</pre></div>
<p>Next, we need to parse this string containing XML into a usable tree.
For this, we use the lxml package (pip install lxml):</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">lxml</span> <span class="kn">import</span> <span class="n">etree</span>
<span class="k">def</span> <span class="nf">get_xml_tree</span><span class="p">(</span><span class="n">xml_string</span><span class="p">):</span>
<span class="k">return</span> <span class="n">etree</span><span class="o">.</span><span class="n">fromstring</span><span class="p">(</span><span class="n">xml_string</span><span class="p">)</span>
</pre></div>
<p>And now you have the tree. Let's take a look at how this XML is
structured.</p>
</div>
<div class="section" id="word-xml-document-structure">
<h2><a class="toc-backref" href="#id2">2 Word XML document structure</a></h2>
<p>For a basic document that consists of paragraphs of text with some
styles/formatting applied, the XML structure is fairly straightforward.
Here's the example document:</p>
<div class="figure" style="width: 577px; height: auto; max-width: 100%;">
<img alt="Example word doc" height="450" src="/images/2013/Word.png" style="width: 577px; height: auto; max-width: 100%;" width="577"/>
</div>
<p>And here's the resulting xml in word/document.xml (you can get this by
simply doing</p>
<div class="highlight"><pre><span class="n">etree</span><span class="o">.</span><span class="n">tostring</span><span class="p">(</span><span class="n">xmltree</span><span class="p">,</span> <span class="n">pretty_print</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
<div class="highlight"><pre><span class="nt"><w:document</span> <span class="na">xmlns:wpc=</span><span class="s">"http://schemas.microsoft.com/office/word/2010/wordprocessingCanvas"</span> <span class="na">xmlns:mo=</span><span class="s">"http://schemas.microsoft.com/office/mac/office/2008/main"</span> <span class="na">xmlns:mc=</span><span class="s">"http://schemas.openxmlformats.org/markup-compatibility/2006"</span> <span class="na">xmlns:mv=</span><span class="s">"urn:schemas-microsoft-com:mac:vml"</span> <span class="na">xmlns:o=</span><span class="s">"urn:schemas-microsoft-com:office:office"</span> <span class="na">xmlns:r=</span><span class="s">"http://schemas.openxmlformats.org/officeDocument/2006/relationships"</span> <span class="na">xmlns:m=</span><span class="s">"http://schemas.openxmlformats.org/officeDocument/2006/math"</span> <span class="na">xmlns:v=</span><span class="s">"urn:schemas-microsoft-com:vml"</span> <span class="na">xmlns:wp14=</span><span class="s">"http://schemas.microsoft.com/office/word/2010/wordprocessingDrawing"</span> <span class="na">xmlns:wp=</span><span class="s">"http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing"</span> <span class="na">xmlns:w10=</span><span class="s">"urn:schemas-microsoft-com:office:word"</span> <span class="na">xmlns:w=</span><span class="s">"http://schemas.openxmlformats.org/wordprocessingml/2006/main"</span> <span class="na">xmlns:w14=</span><span class="s">"http://schemas.microsoft.com/office/word/2010/wordml"</span> <span class="na">xmlns:wpg=</span><span class="s">"http://schemas.microsoft.com/office/word/2010/wordprocessingGroup"</span> <span class="na">xmlns:wpi=</span><span class="s">"http://schemas.microsoft.com/office/word/2010/wordprocessingInk"</span> <span class="na">xmlns:wne=</span><span class="s">"http://schemas.microsoft.com/office/word/2006/wordml"</span> <span class="na">xmlns:wps=</span><span class="s">"http://schemas.microsoft.com/office/word/2010/wordprocessingShape"</span> <span class="na">mc:Ignorable=</span><span class="s">"w14 wp14"</span><span class="nt">></span>
<span class="nt"><w:body></span>
<span class="nt"><w:p</span> <span class="na">w:rsidR=</span><span class="s">"00192975"</span> <span class="na">w:rsidRDefault=</span><span class="s">"00450526"</span> <span class="na">w:rsidP=</span><span class="s">"00450526"</span><span class="nt">></span>
<span class="nt"><w:pPr></span>
<span class="nt"><w:pStyle</span> <span class="na">w:val=</span><span class="s">"Heading1"</span><span class="nt">/></span>
<span class="nt"></w:pPr></span>
<span class="nt"><w:r></span>
<span class="nt"><w:t></span>Test<span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"></w:p></span>
<span class="nt"><w:p</span> <span class="na">w:rsidR=</span><span class="s">"00450526"</span> <span class="na">w:rsidRDefault=</span><span class="s">"00450526"</span> <span class="na">w:rsidP=</span><span class="s">"00450526"</span><span class="nt">></span>
<span class="nt"><w:r></span>
<span class="nt"><w:t></span>The quick brown fox jumped over the lazy dog.<span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"></w:p></span>
<span class="nt"><w:p</span> <span class="na">w:rsidR=</span><span class="s">"00450526"</span> <span class="na">w:rsidRPr=</span><span class="s">"00450526"</span> <span class="na">w:rsidRDefault=</span><span class="s">"00450526"</span> <span class="na">w:rsidP=</span><span class="s">"00450526"</span><span class="nt">></span>
<span class="nt"><w:bookmarkStart</span> <span class="na">w:id=</span><span class="s">"0"</span> <span class="na">w:name=</span><span class="s">"_GoBack"</span><span class="nt">/></span>
<span class="nt"><w:bookmarkEnd</span> <span class="na">w:id=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></w:p></span>
<span class="nt"><w:sectPr</span> <span class="na">w:rsidR=</span><span class="s">"00450526"</span> <span class="na">w:rsidRPr=</span><span class="s">"00450526"</span> <span class="na">w:rsidSect=</span><span class="s">"00192975"</span><span class="nt">></span>
<span class="nt"><w:pgSz</span> <span class="na">w:w=</span><span class="s">"12240"</span> <span class="na">w:h=</span><span class="s">"15840"</span><span class="nt">/></span>
<span class="nt"><w:pgMar</span> <span class="na">w:top=</span><span class="s">"1440"</span> <span class="na">w:right=</span><span class="s">"1800"</span> <span class="na">w:bottom=</span><span class="s">"1440"</span> <span class="na">w:left=</span><span class="s">"1800"</span> <span class="na">w:header=</span><span class="s">"720"</span> <span class="na">w:footer=</span><span class="s">"720"</span> <span class="na">w:gutter=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"><w:cols</span> <span class="na">w:space=</span><span class="s">"720"</span><span class="nt">/></span>
<span class="nt"><w:docGrid</span> <span class="na">w:linePitch=</span><span class="s">"360"</span><span class="nt">/></span>
<span class="nt"></w:sectPr></span>
<span class="nt"></w:body></span>
<span class="nt"></w:document></span>
</pre></div>
<p>Some salient points:</p>
<ul class="simple">
<li>The top tag is <w:document>, followed by the <w:body> tag</li>
<li>The body is then split up into paragraphs, demarcated by <w:p></li>
<li>Each paragraph may contain paragraph styles</li>
<li>Within each paragraph, there also exist runs of content <w:r></li>
<li>These runs then end up having text blocks inside them, with the text
enclosed by <w:t> tags</li>
<li>Multiple pieces of text can be contained within a run <w:r> tag</li>
</ul>
<p>The last point is not immediately apparent, but consider what happened
when I edited the file in the next section.</p>
</div>
<div class="section" id="complications-in-the-xml">
<h2><a class="toc-backref" href="#id3">3 Complications in the XML</a></h2>
<p>Let's edit the docx to uppercase the 'fox' into 'FOX' and look at the
XML contents:</p>
<div class="highlight"><pre><span class="nt"><w:p</span> <span class="na">w14:paraId=</span><span class="s">"5192A87F"</span> <span class="na">w14:textId=</span><span class="s">"77777777"</span> <span class="na">w:rsidR=</span><span class="s">"00450526"</span> <span class="na">w:rsidRDefault=</span><span class="s">"00450526"</span> <span class="na">w:rsidP=</span><span class="s">"00450526"</span><span class="nt">></span>
<span class="nt"><w:r></span>
<span class="nt"><w:t</span> <span class="na">xml:space=</span><span class="s">"preserve"</span><span class="nt">></span>The quick brown <span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"><w:r</span> <span class="na">w:rsidR=</span><span class="s">"004C173F"</span><span class="nt">></span>
<span class="nt"><w:t></span>FOX<span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"><w:bookmarkStart</span> <span class="na">w:id=</span><span class="s">"0"</span> <span class="na">w:name=</span><span class="s">"_GoBack"</span><span class="nt">/></span>
<span class="nt"><w:bookmarkEnd</span> <span class="na">w:id=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"><w:r></span>
<span class="nt"><w:t</span> <span class="na">xml:space=</span><span class="s">"preserve"</span><span class="nt">></span> jumped over the lazy dog.<span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"></w:p></span>
</pre></div>
<p>MS Word has now split up the previously single piece of text inside
one run, into 3 separate runs of text, with some meta-data (probably to
keep track of undo actions). So, any of your text parsing and
substitution needs to take into account the fact that your contiguous
text may in fact be split up into separate sub-trees in the XML file.
Also, see below what happens when I now bold the 'FOX':</p>
<div class="highlight"><pre><span class="nt"><w:p</span> <span class="na">w14:paraId=</span><span class="s">"5192A87F"</span> <span class="na">w14:textId=</span><span class="s">"77777777"</span> <span class="na">w:rsidR=</span><span class="s">"00450526"</span> <span class="na">w:rsidRDefault=</span><span class="s">"00450526"</span> <span class="na">w:rsidP=</span><span class="s">"00450526"</span><span class="nt">></span>
<span class="nt"><w:r></span>
<span class="nt"><w:t</span> <span class="na">xml:space=</span><span class="s">"preserve"</span><span class="nt">></span>The quick brown <span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"><w:r</span> <span class="na">w:rsidR=</span><span class="s">"004C173F"</span><span class="nt">></span>
<span class="nt"><w:rPr></span>
<span class="nt"><w:b/></span>
<span class="nt"></w:rPr></span>
<span class="nt"><w:t></span>FOX<span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"><w:bookmarkStart</span> <span class="na">w:id=</span><span class="s">"0"</span> <span class="na">w:name=</span><span class="s">"_GoBack"</span><span class="nt">/></span>
<span class="nt"><w:bookmarkEnd</span> <span class="na">w:id=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"><w:r></span>
<span class="nt"><w:t</span> <span class="na">xml:space=</span><span class="s">"preserve"</span><span class="nt">></span> jumped over the lazy dog.<span class="nt"></w:t></span>
<span class="nt"></w:r></span>
<span class="nt"></w:p></span>
</pre></div>
<p>Notice that now there is a format tag inside your run tag as well.</p>
</div>
<div class="section" id="extracting-text">
<h2><a class="toc-backref" href="#id4">4 Extracting text</a></h2>
<p>lxml has some nice functions for traversing the XML tree, but I usually
had to wrap these in my own iterators to get the most functionality. For
instance, here's a class-based iterator that will traverse every node
given a starting node ''my_etree'', and return every text node and it's
containing text.</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">_itertext</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">my_etree</span><span class="p">):</span>
<span class="sd">"""Iterator to go through xml tree's text nodes"""</span>
<span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">my_etree</span><span class="o">.</span><span class="n">iter</span><span class="p">(</span><span class="n">tag</span><span class="o">=</span><span class="n">etree</span><span class="o">.</span><span class="n">Element</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_check_element_is</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="s">'t'</span><span class="p">):</span>
<span class="k">yield</span> <span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">node</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">_check_element_is</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">type_char</span><span class="p">):</span>
<span class="n">word_schema</span> <span class="o">=</span> <span class="s">'http://schemas.openxmlformats.org/wordprocessingml/2006/main'</span>
<span class="k">return</span> <span class="n">element</span><span class="o">.</span><span class="n">tag</span> <span class="o">==</span> <span class="s">'{</span><span class="si">%s</span><span class="s">}</span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">word_schema</span><span class="p">,</span><span class="n">type_char</span><span class="p">)</span>
</pre></div>
<p>So, in order to get all the text out of the document, you could use the
earlier function to get the xml tree and then iterate over it like this:</p>
<div class="highlight"><pre><span class="o">...</span>
<span class="n">xml_from_file</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_word_xml</span><span class="p">(</span><span class="n">wod_filename</span><span class="p">)</span>
<span class="n">xml_tree</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_xml_tree</span><span class="p">(</span><span class="n">xml_from_file</span><span class="p">)</span>
<span class="k">for</span> <span class="n">node</span><span class="p">,</span> <span class="n">txt</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_itertext</span><span class="p">(</span><span class="n">xml_tree</span><span class="p">):</span>
<span class="k">print</span> <span class="n">txt</span>
</pre></div>
</div>
<div class="section" id="modifying-text">
<h2><a class="toc-backref" href="#id5">5 Modifying text</a></h2>
<p>Here's a quick example on how to modify the XML tree to change the Word
content. In one of my other project's (that I'll blog about soon) I
needed to replace special pieces of text with some other text (think of
it as a mail-merge or a templating function). I define a special piece
of text as being enclosed inside square brackets like [my_tag] in my
original Word document. Now, the problem is that this tag could be split
among multiple XML nodes because of the way Word splits up runs of text.
So, to make my text-substitution function easier, I go through the XML
tree and collapse all these tags into a single text node first. I can do
this without too much worry because I know there will never be any
special formatting or structures inside the brackets. So here's my
function that does it:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">_join_tags</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">my_etree</span><span class="p">):</span>
<span class="n">chars</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">openbrac</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">inside_openbrac_node</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">for</span> <span class="n">node</span><span class="p">,</span><span class="n">text</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_itertext</span><span class="p">(</span><span class="n">my_etree</span><span class="p">):</span>
<span class="c"># Scan through every node with text</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span><span class="n">c</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">text</span><span class="p">):</span>
<span class="c"># Go through each node's text character by character</span>
<span class="k">if</span> <span class="n">c</span> <span class="o">==</span> <span class="s">'['</span><span class="p">:</span>
<span class="n">openbrac</span> <span class="o">=</span> <span class="bp">True</span> <span class="c"># Within a tag</span>
<span class="n">inside_openbrac_node</span> <span class="o">=</span> <span class="bp">True</span> <span class="c"># Tag was opened in this node</span>
<span class="n">openbrac_node</span> <span class="o">=</span> <span class="n">node</span> <span class="c"># Save ptr to open bracket containing node</span>
<span class="n">chars</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">elif</span> <span class="n">c</span><span class="o">==</span> <span class="s">']'</span><span class="p">:</span>
<span class="k">assert</span> <span class="n">openbrac</span>
<span class="k">if</span> <span class="n">inside_openbrac_node</span><span class="p">:</span>
<span class="c"># Open and close inside same node, no need to do anything</span>
<span class="k">pass</span>
<span class="k">else</span><span class="p">:</span>
<span class="c"># Open bracket in earlier node, now it's closed</span>
<span class="c"># So append all the chars we've encountered since the openbrac_node '['</span>
<span class="c"># to the openbrac_node</span>
<span class="n">chars</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">']'</span><span class="p">)</span>
<span class="n">openbrac_node</span><span class="o">.</span><span class="n">text</span> <span class="o">+=</span> <span class="s">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">chars</span><span class="p">)</span>
<span class="c"># Also, don't forget to remove the characters seen so far from current node</span>
<span class="n">node</span><span class="o">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">:]</span>
<span class="n">openbrac</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">inside_openbrac_node</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="c"># Normal text character</span>
<span class="k">if</span> <span class="n">openbrac</span> <span class="ow">and</span> <span class="n">inside_openbrac_node</span><span class="p">:</span>
<span class="c"># No need to copy text</span>
<span class="k">pass</span>
<span class="k">elif</span> <span class="n">openbrac</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">inside_openbrac_node</span><span class="p">:</span>
<span class="n">chars</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c"># outside of a open/close</span>
<span class="k">pass</span>
<span class="k">if</span> <span class="n">openbrac</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">inside_openbrac_node</span><span class="p">:</span>
<span class="c"># Went through all text that is part of an open bracket/close bracket</span>
<span class="c"># in other nodes</span>
<span class="c"># need to remove this text completely</span>
<span class="n">node</span><span class="o">.</span><span class="n">text</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">inside_openbrac_node</span> <span class="o">=</span> <span class="bp">False</span>
</pre></div>
</div>
<div class="section" id="saving-the-edited-word-file">
<h2><a class="toc-backref" href="#id6">6 Saving the edited word file</a></h2>
<p>Now, if you want to save this modified content, you just need to extract
all the files in the docx, overwrite the content file, and then zip it
back up:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">_write_and_close_docx</span> <span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">xml_content</span><span class="p">,</span> <span class="n">output_filename</span><span class="p">):</span>
<span class="sd">""" Create a temp directory, expand the original docx zip.</span>
<span class="sd"> Write the modified xml to word/document.xml</span>
<span class="sd"> Zip it up as the new docx</span>
<span class="sd"> """</span>
<span class="n">tmp_dir</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">mkdtemp</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">zipfile</span><span class="o">.</span><span class="n">extractall</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span><span class="s">'word/document.xml'</span><span class="p">),</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">xmlstr</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">tostring</span> <span class="p">(</span><span class="n">xml_content</span><span class="p">,</span> <span class="n">pretty_print</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">xmlstr</span><span class="p">)</span>
<span class="c"># Get a list of all the files in the original docx zipfile</span>
<span class="n">filenames</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">zipfile</span><span class="o">.</span><span class="n">namelist</span><span class="p">()</span>
<span class="c"># Now, create the new zip file and add all the filex into the archive</span>
<span class="n">zip_copy_filename</span> <span class="o">=</span> <span class="n">output_filename</span>
<span class="k">with</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="n">zip_copy_filename</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">docx</span><span class="p">:</span>
<span class="k">for</span> <span class="n">filename</span> <span class="ow">in</span> <span class="n">filenames</span><span class="p">:</span>
<span class="n">docx</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span><span class="n">filename</span><span class="p">),</span> <span class="n">filename</span><span class="p">)</span>
<span class="c"># Clean up the temp dir</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">rmtree</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="summary">
<h2><a class="toc-backref" href="#id7">7 Summary</a></h2>
<p>Done! I hope I've helped explain how to get a the contents of MS Word
docx file and do some simple modifications to it.
Again, if you need more examples on how to process .docx files, please check
out how I've used these methods to implement a data-driven resume generator
called <a class="reference external" href="https://virantha.com/2015/04/07/oneresume-a-data-driven-resume-generator-for-microsoft-word/">OneResumé</a> for multiple file formats including MS Word.</p>
<p>Until then, happy coding!</p>
</div>
</body></html>Python auto sort of OCR'ed PDFs2013-04-20T20:25:00-04:00viranthatag:https://virantha.com,2013-04-20:2013/04/20/python-auto-sort-of-ocred-pdfs/<html><body><p>I'd <a class="reference external" href="http://virantha.com/2012/10/25/getting-rid-of-paper-clutter/">previously written about</a> how I was using a <a class="amazon" href="http://www.amazon.com/gp/product/B00ATZ9QMO?ie=UTF8&linkCode=as2&camp=1634&tag=virantha-20">Fujitsu ScanSnap 1500</a> to reduce paper clutter
and move to a paperless workflow at home. So far, this system has been
working great for me, with every scanned document getting OCR'ed and
uploaded to my default Evernote notebook as a searchable PDF. However,
I realized that I was still spending more time than I wanted to in
manually sorting these documents into Evernote notebooks.</p>
<p>It seemed silly that there was no way for Evernote to automatically sort
pdfs based on some kind of tag or keyword search. So I ended up
spending a few hours developing the Python script described below to do
this sorting for automatically. At some point, I'll package this up on
github or bitbucket.</p>
<p>I'm calling this program <strong>scanever</strong> because I didn't have more than
30 seconds to come up with something besides pdf2evernote. It replaces
the kludgy batchfile and watch4folder I had used previously, and is
OS-independent.</p>
<div class="section" id="use">
<h2>Use</h2>
<p>scanever takes a configuration file that lists all the folders I want to
sort into, as well as the keywords that fall into each folder. Here's
an example (it uses the YAML syntax):</p>
<div class="highlight"><pre><span class="l-Scalar-Plain">watch_folder</span><span class="p-Indicator">:</span> <span class="s">"M:/Incoming"</span>
<span class="l-Scalar-Plain">evernote_folder</span><span class="p-Indicator">:</span> <span class="s">"M:/To</span><span class="nv"> </span><span class="s">Evernote"</span>
<span class="l-Scalar-Plain">default_folder</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">default</span>
<span class="l-Scalar-Plain">folders</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">finances</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">chase visa</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">american express</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">bank of america</span>
<span class="l-Scalar-Plain">home</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">mortgage</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">property tax</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">city of oz</span>
<span class="l-Scalar-Plain">health</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">explanation of benefits</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">healthcare</span>
</pre></div>
<p>The <em>folders</em> section defines a list of folders and their associated
keywords. For example, any PDF that gets placed in "M:/Incoming" by
Abbyy FineReader that has "chase visa" or "american express" or "bank of
america" in its first page gets filed into a subfolder of "M:/To
Evernote" called "finances". The <em>default_folder</em> sets where the pdf
goes if there is no match found.</p>
<p>So how does my script work? The main logic is divided into three
classes:</p>
<ul class="simple">
<li>Main class that reads in the configuration file and runs everything</li>
<li>Folder watching logic</li>
<li>File matching and filing logic</li>
</ul>
</div>
<div class="section" id="watching-the-incoming-folder">
<h2>Watching the incoming folder</h2>
<p>For this, I'm relying on the excellent cross-platform package
<a class="reference external" href="http://pythonhosted.org/watchdog/">watchdog</a>. Here's my custom event handler class that I use to
interface with this API, and the way it works is as follows:</p>
<ol class="arabic simple">
<li>On any file created/modified/moved event, it will check if the file
ends with .pdf. (This is the first pdf that the ScanSnap does before
it sends it off to Abbyy for OCR'ing). We add this filename to the
events list (inside <em>check_for_new_pdf) </em>if not already present.</li>
<li>Abbyy FineReader at some point will finish writing the _OCR.pdf file
and delete the original .pdf. We key off this delete event and start
our analysis of the final _OCR file, described in the next section.</li>
</ol>
<table class="highlighttable pretty compact cell-border stripe" id="unknown_11"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39</pre></div></td><td class="code"><div class="highlight"><pre><span class="k">class</span> <span class="nc">ChangeHandler</span><span class="p">(</span><span class="n">FileSystemEventHandler</span><span class="p">):</span>
<span class="n">events</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pdfsearcher</span><span class="p">):</span>
<span class="n">FileSystemEventHandler</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pdfsearcher</span> <span class="o">=</span> <span class="n">pdfsearcher</span>
<span class="k">def</span> <span class="nf">check_for_new_pdf</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">ev_path</span><span class="p">):</span>
<span class="k">if</span> <span class="n">ev_path</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s">".pdf"</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">ev_path</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s">"_OCR.pdf"</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">ev_path</span> <span class="ow">in</span> <span class="n">ChangeHandler</span><span class="o">.</span><span class="n">events</span><span class="p">:</span>
<span class="n">ChangeHandler</span><span class="o">.</span><span class="n">events</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ev_path</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"Adding </span><span class="si">%s</span><span class="s"> to event queue"</span> <span class="o">%</span> <span class="n">ev_path</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"</span><span class="si">%s</span><span class="s"> alread in event queue"</span> <span class="o">%</span> <span class="n">ev_path</span>
<span class="k">def</span> <span class="nf">on_created</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">print</span> <span class="p">(</span><span class="s">"on_created: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">event</span><span class="o">.</span><span class="n">src_path</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">check_for_new_pdf</span><span class="p">(</span><span class="n">event</span><span class="o">.</span><span class="n">src_path</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_moved</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">print</span> <span class="p">(</span><span class="s">"on_moved: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">event</span><span class="o">.</span><span class="n">src_path</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">check_for_new_pdf</span><span class="p">(</span><span class="n">event</span><span class="o">.</span><span class="n">dest_path</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_modified</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">print</span> <span class="p">(</span><span class="s">"on_modified: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">event</span><span class="o">.</span><span class="n">src_path</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">check_for_new_pdf</span><span class="p">(</span><span class="n">event</span><span class="o">.</span><span class="n">src_path</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_deleted</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">event</span><span class="p">):</span>
<span class="k">print</span> <span class="p">(</span><span class="s">"on_deleted: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">event</span><span class="o">.</span><span class="n">src_path</span><span class="p">)</span>
<span class="n">ev_src_path</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="n">src_path</span>
<span class="k">if</span> <span class="n">ev_src_path</span> <span class="ow">in</span> <span class="n">ChangeHandler</span><span class="o">.</span><span class="n">events</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Deleting </span><span class="si">%s</span><span class="s"> in event queue"</span> <span class="o">%</span> <span class="n">ev_src_path</span>
<span class="n">ChangeHandler</span><span class="o">.</span><span class="n">events</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">ev_src_path</span><span class="p">)</span>
<span class="c"># Now, check that the OCR version is present</span>
<span class="n">ocr_path</span> <span class="o">=</span> <span class="n">ev_src_path</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">".pdf"</span><span class="p">,</span> <span class="s">"_OCR.pdf"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">ocr_path</span><span class="p">):</span>
<span class="k">print</span> <span class="s">"Analyzing OCR'ed file </span><span class="si">%s</span><span class="s">!"</span> <span class="o">%</span> <span class="n">ocr_path</span>
<span class="n">pdf</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pdfsearcher</span>
<span class="n">text</span> <span class="o">=</span> <span class="n">pdf</span><span class="o">.</span><span class="n">readPdfFirstPage</span><span class="p">(</span><span class="n">ocr_path</span><span class="p">)</span>
<span class="n">pdf</span><span class="o">.</span><span class="n">moveToFolders</span><span class="p">()</span>
</pre></div>
</td></tr></table></div>
<div class="section" id="analyzing-the-pdf-and-sorting-it">
<h2>Analyzing the PDF and sorting it</h2>
<p>The next step is to read in the PDF (the first page) and search for
matching text. For this, I rely on <a class="reference external" href="http://knowah.github.io/PyPDF2/">PyPDF2</a></p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">PdfSearcher</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">evernote</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">folderTargets</span> <span class="o">=</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pdfText</span> <span class="o">=</span> <span class="s">""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">evernote_folder</span> <span class="o">=</span> <span class="n">evernote</span>
<span class="bp">self</span><span class="o">.</span><span class="n">default_folder</span> <span class="o">=</span> <span class="n">default</span>
<span class="k">def</span> <span class="nf">readPdfFirstPage</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span>
<span class="n">reader</span> <span class="o">=</span> <span class="n">PdfFileReader</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">text</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">getPage</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">extractText</span><span class="p">()</span>
<span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'ascii'</span><span class="p">,</span> <span class="s">'ignore'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pdfText</span> <span class="o">=</span> <span class="n">text</span>
<span class="k">return</span> <span class="n">text</span>
<span class="k">def</span> <span class="nf">addFolderTarget</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dirname</span><span class="p">,</span> <span class="n">matchStrings</span><span class="p">):</span>
<span class="c"># Used externally to add in the keywords/folders</span>
<span class="k">assert</span> <span class="n">dirname</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">folderTargets</span><span class="p">,</span> <span class="s">"Target folder already defined! (</span><span class="si">%s</span><span class="s">)"</span> <span class="o">%</span> <span class="p">(</span><span class="n">dirname</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">folderTargets</span><span class="p">[</span><span class="n">dirname</span><span class="p">]</span> <span class="o">=</span> <span class="n">matchStrings</span>
<span class="k">def</span> <span class="nf">_getMatchingFolder</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c"># Return the folder that matches any of the keywords in self.pdfText</span>
<span class="n">searchText</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pdfText</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
<span class="k">for</span> <span class="n">folder</span><span class="p">,</span><span class="n">strings</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">folderTargets</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">strings</span><span class="p">:</span>
<span class="k">if</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">searchText</span><span class="p">:</span>
<span class="k">print</span> <span class="n">s</span>
<span class="k">return</span> <span class="n">folder</span>
<span class="c"># No match found, so return</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">moveToFolders</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">tgt_folder</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getMatchingFolder</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">tgt_folder</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"No match found, using default folder"</span><span class="p">)</span>
<span class="n">tgt_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">evernote_folder</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">default_folder</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">tgt_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">evernote_folder</span><span class="p">,</span><span class="n">tgt_folder</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">tgt_path</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">tgt_path</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Making path </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">tgt_path</span><span class="p">)</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">move</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="n">tgt_path</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="main-class">
<h2>Main class</h2>
<p>Now, here's the main class that reads the configuration and sets up the
event loop:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">ScanEver</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span> <span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">maxlength</span> <span class="o">=</span> <span class="mi">500</span>
<span class="k">def</span> <span class="nf">getOptions</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">argv</span><span class="p">):</span>
<span class="n">usage</span> <span class="o">=</span> <span class="s">'ScanEver '</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">OptionParser</span><span class="p">(</span><span class="n">usage</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s">'-d'</span><span class="p">,</span> <span class="s">'--debug'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s">'store_true'</span><span class="p">,</span>
<span class="n">default</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">dest</span><span class="o">=</span><span class="s">'debug'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'Turn on debugging'</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s">'-v'</span><span class="p">,</span> <span class="s">'--verbose'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s">'store_true'</span><span class="p">,</span>
<span class="n">default</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">dest</span><span class="o">=</span><span class="s">'verbose'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'Turn on verbose mode'</span><span class="p">)</span>
<span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="n">parse_args</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">debug</span> <span class="o">=</span> <span class="n">opt</span><span class="o">.</span><span class="n">debug</span>
<span class="bp">self</span><span class="o">.</span><span class="n">verbose</span> <span class="o">=</span> <span class="n">opt</span><span class="o">.</span><span class="n">verbose</span>
<span class="k">if</span> <span class="n">opt</span><span class="o">.</span><span class="n">debug</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">,</span> <span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(message)s</span><span class="s">'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">opt</span><span class="o">.</span><span class="n">verbose</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">,</span> <span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(message)s</span><span class="s">'</span><span class="p">)</span>
<span class="n">fstream</span> <span class="o">=</span> <span class="nb">file</span><span class="p">(</span><span class="s">"paths.yaml"</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span>
<span class="n">myopts</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">fstream</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">evernote_folder</span> <span class="o">=</span> <span class="n">myopts</span><span class="p">[</span><span class="s">'evernote_folder'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">default_folder</span> <span class="o">=</span> <span class="n">myopts</span><span class="p">[</span><span class="s">'default_folder'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">searcher</span> <span class="o">=</span> <span class="n">PdfSearcher</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">evernote_folder</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">default_folder</span><span class="p">)</span>
<span class="k">for</span> <span class="n">folder</span><span class="p">,</span><span class="n">strings</span> <span class="ow">in</span> <span class="n">myopts</span><span class="p">[</span><span class="s">"folders"</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="bp">self</span><span class="o">.</span><span class="n">searcher</span><span class="o">.</span><span class="n">addFolderTarget</span><span class="p">(</span><span class="n">folder</span><span class="p">,</span><span class="n">strings</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">monitor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">event_handler</span> <span class="o">=</span> <span class="n">ChangeHandler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">searcher</span><span class="p">)</span>
<span class="n">observer</span> <span class="o">=</span> <span class="n">Observer</span><span class="p">()</span>
<span class="c">#observer.schedule(event_handler, os.path.abspath("test"))</span>
<span class="n">observer</span><span class="o">.</span><span class="n">schedule</span><span class="p">(</span><span class="n">event_handler</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="s">"""M:\Incoming"""</span><span class="p">))</span>
<span class="n">observer</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">observer</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
<span class="n">observer</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">go</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">argv</span><span class="p">):</span>
<span class="c"># Read the command line options</span>
<span class="bp">self</span><span class="o">.</span><span class="n">getOptions</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">monitor</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">script</span> <span class="o">=</span> <span class="n">ScanEver</span><span class="p">()</span>
<span class="n">script</span><span class="o">.</span><span class="n">go</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
</pre></div>
</div>
<script charset="utf8" src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script><script charset="utf8" src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script><script>
$(document).ready(function() {
$('#unknown_11').dataTable( {
"sPaginationType": "full_numbers" ,
"sDom": "rtipf",
"order": [],
"iDisplayLength": 25,
"oLanguage": {
"oPaginate": {
"sNext": ">",
"sPrevious": "<",
"sFirst": "<<",
"sLast": ">>"
}
}
});
} );</script></body></html>Better VIM for Python2013-03-01T19:13:00-05:00viranthatag:https://virantha.com,2013-03-01:2013/03/01/better-vim-for-python/<html><body><p>As someone who spends a large fraction of their day editing text and
code, I've often thought about just investing a few days learning the
more advanced time-saving features of my text-editor, VIM.
Unfortunately, "a few days" just doesn't happen, and the few times I did
learn some new tips, I ended up forgetting them after a few days through
disuse. So I'm going to start a new series of blog posts, mainly for my
own reference, that I'll try to come back to often and refresh my
memory.</p>
<p>Most of this is based on the excellent resources already out there on
the web, most notably these posts:</p>
<ul class="simple">
<li><a class="reference external" href="http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/"> Vim as Python IDE</a></li>
<li><a class="reference external" href="http://sontek.net/blog/detail/turning-vim-into-a-modern-python-ide">Turning Vim into a modern Python IDE</a></li>
</ul>
<div class="section" id="alignment">
<h2>Alignment</h2>
<p>At work, we try to enforce pretty strict guidelines on the readability
of python code You only write a specific piece of code a few times, but
you or someone else will have to read it many times over, so it makes
sense to optimize readability over getting it written as quickly as
possible. The <a class="reference external" href="http://www.vim.org/scripts/script.php?script_id=294">Align plugin</a> makes some of these readability
guidelines simple to implement and maintain, by allowing you to align
function arguments, variable assignments, etc.</p>
<p>For example, if you have code like in the following nonsensical
example:</p>
<div class="highlight"><pre><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span> <span class="c"># set a</span>
<span class="n">nextvar</span> <span class="o">=</span> <span class="s">"kdjf"</span> <span class="c"># hmmm</span>
<span class="n">cdk</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
<p>Selecting ("V") and typing <tt class="docutils literal">Align = #</tt> will reformat your text as:</p>
<div class="highlight"><pre><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span> <span class="c"># set a</span>
<span class="n">nextvar</span> <span class="o">=</span> <span class="s">"kdjf"</span> <span class="c"># hmmm</span>
<span class="n">cdk</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
</div>
</body></html>Class-based decorators in Python2012-11-02T01:22:00-04:00viranthatag:https://virantha.com,2012-11-02:2012/11/02/class-based-decorators-in-python/<html><body><p>I recently started using decorators in python (2.7) to clean up some
existing code, and one big hurdle I had to surmount was the dearth of
accurate information on using class-based decorators. The few examples I
found were quite buggy, and it seemed that most people did not use
decorator classes, which is a shame because in addition to being very
flexible, they expose the underlying mechanics of method/function
decoration better than functions.</p>
<p>Here's an example of one such use: in EDA tool flows, we often do many
text file transformations (in fact, it's been argued that all EDA work
basically boils down to transforming information from one file format to
another :-) ). Sometimes, a lot of intermediate files get generated on
the way to the final output, such as when taking a structural circuit
description and outputting a spice netlist that has the proper name
mangling for the end simulator or netlist comparison engine. In these
instances, it would be far preferable to do all the work in a tmp
directory to reduce file clutter in our source directories, so a lot of
functions end up with code that looks like the following:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">someMunge</span> <span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">currentDir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">tmpDir</span><span class="p">)</span>
<span class="o">....</span> <span class="n">actual</span> <span class="n">work</span> <span class="o">....</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">currentDir</span><span class="p">)</span>
</pre></div>
<p>In fact, I've omitted a lot of the error-checking and cleanup code that
we would end up using. Once we start having a lot of different munge
functions, all this boilerplate code needs to be factored out to keep it
maintainable, so we might end up doing the following:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">someMunge</span> <span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">enterTempDir</span><span class="p">()</span>
<span class="o">....</span> <span class="n">actual</span> <span class="n">work</span> <span class="o">....</span>
<span class="bp">self</span><span class="o">.</span><span class="n">leaveTempDir</span><span class="p">()</span>
</pre></div>
<p>Seems reasonable, although it's still cluttering up the munge function
with extraneous detail unrelated to the actual munge. A much cleaner
solution is being able to do the following:</p>
<div class="highlight"><pre><span class="nd">@tmpdir</span>
<span class="k">def</span> <span class="nf">someMunge</span> <span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="o">....</span> <span class="n">actual</span> <span class="n">work</span> <span class="o">....</span>
</pre></div>
<p>The <tt class="docutils literal">@tmpdir</tt> applies the tmpdir decorator to this class
method. And here is what the tmpdir decorator class looks like:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">tmpdir</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">f</span><span class="p">):</span>
<span class="c"># f is the method being decorated, so save it so we can call it</span>
<span class="n">later</span><span class="err">!</span>
<span class="bp">self</span><span class="o">.</span><span class="n">f</span> <span class="o">=</span> <span class="n">f</span>
<span class="k">def</span> <span class="nf">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">instance</span><span class="p">,</span> <span class="n">owner</span><span class="p">):</span>
<span class="c"># Save a ptr to the object being decorated</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cls</span> <span class="o">=</span> <span class="n">owner</span>
<span class="bp">self</span><span class="o">.</span><span class="n">obj</span> <span class="o">=</span> <span class="n">instance</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__call__</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c"># The actual meat of the decorator</span>
<span class="n">currentDir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">obj</span><span class="o">.</span><span class="n">tmpDir</span><span class="p">)</span>
<span class="c"># Call the original method being decorated</span>
<span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">f</span><span class="o">.</span><span class="n">__call__</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">obj</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">currentDir</span><span class="p">)</span>
<span class="k">return</span> <span class="n">r</span>
</pre></div>
<p>A more detailed explanation of how it works is coming up soon!</p>
</body></html>Getting rid of paper clutter2012-10-25T14:30:00-04:00viranthatag:https://virantha.com,2012-10-25:2012/10/25/getting-rid-of-paper-clutter/<html><body><div class="note">
<p class="first admonition-title">Note</p>
<p class="last">A lot of this information is deprecated now. Please see my PyPDFOCR
package for how I do everything described in this article</p>
</div>
<div class="figure align-right" style="width: 50%">
<img alt=" Picture" src="/images/2012/picture2life_05067_original.jpg" style="width: 90%;"/>
</div>
<p>I've tried to go paperless for a long time now, but the overhead of
starting my laptop, scanning, importing, and the filing in a folder
always made me go back to just keeping stacks of paper around. But now
I've finally found a solution that works well enough that I haven't
abandoned it after a few months. The key is a stand-alone scanner that
does OCR, converts to PDF, and then uploads to Evernote automatically.
Details below:</p>
<div class="section" id="hardware-required">
<h2>Hardware required</h2>
<ol class="arabic simple">
<li><a class="reference external" href="http://www.amazon.com/gp/product/B001V9LQH0/ref=as_li_tf_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B001V9LQH0&linkCode=as2&tag=virantha-20">Fujitsu ScanSnap 1500</a> - This was the game-changer; it's a little
pricey but after having gone through a bunch of inferior flat-bed and
Neatworks scanners, this stand-alone scanner really makes things
simple.</li>
<li>Dedicated PC running on your network - You could just use your
laptop, but I have a bunch of different desktops running file
servers etc that makes things simpler</li>
</ol>
</div>
<div class="section" id="software-required">
<h2> Software required</h2>
<ol class="arabic simple">
<li><a class="reference external" href="http://evernote.com">Evernote</a> - I'm a big fan of this service, and use it for all my
notes/receipts/documents/lists. Get the premium service, it's
definitely worth it!</li>
<li>Abby FineReader for OCR (this ships with the Fujitsu ScanSnap)</li>
<li>Download a program that can watch a folder for file changes and run a
script. Since the server I'm using at home runs Windows, I
use <a class="reference external" href="http://www.guidingtech.com/9861/automate-folder-actions-windows-watch-4-folder/">Watch 4 Folder</a>, but this should be much easier on Mac OS X.</li>
</ol>
</div>
<div class="section" id="workflow-scripts">
<h2>Workflow Scripts</h2>
<ul class="simple">
<li>Setup up a profile on your ScanSnap image scanner that scans to Abby
FineReader (OCR that comes with the software) and does a searchable
PDF, that writes the file to a specific folder which I'll refer to as
"Incoming". The raw scan will first show up as
"YYYY_MM_DD_HH_MM_SS.pdf", and once Abby finishes the OCR in the
background, it will replace it with
"YYYY_MM_DD_HH_MM_SS_OCR.pdf"</li>
<li>Setup the following batch file "move.bat". This will watch the
"Incoming" folder for any file ending with "_OCR.pdf", wait 5
minutes, and then copy it to a folder called "To evernote"</li>
</ul>
<div class="line-block">
<div class="line">[ccw lang="dos" width="100%" strict="true"]</div>
<div class="line-block">
<div class="line">set noext=%file:~1,-9%</div>
<div class="line">set ocr="%noext%_OCR.pdf"</div>
<div class="line">IF EXIST %ocr% (</div>
<div class="line">echo "Found %ocr%! Waiting 5 minutes before doing anything"</div>
<div class="line">PING 1.1.1.1 -n 1 -w 300000 >NUL</div>
<div class="line">move %ocr% "c:\users\virantha\Documents\ScanSnap\To evernote"</div>
<div class="line">)</div>
<div class="line">exit</div>
<div class="line">[/cc]</div>
</div>
</div>
<ul class="simple">
<li>Since we want multiple instances of this batch file running, we need
to create another batch file "start.bat" to invoke this as a process:</li>
<li>[cc lang="dos"] start c:\Users\virantha\Documents\move.bat %1
[/cc]</li>
<li>Configure Watch 4 Folder to run at startup minimized, and monitor
your "Incoming" folder for any changes, and then execute "start.bat"</li>
<li>Configure Evernote "Tools -> Import Folders" and add your "To
evernote" directory to the list of folders Evernote watches for file
imports.</li>
<li>You're done! One press on your scanner, and your OCR'ed PDF
documents will arrive in your Evernote default notebook in a few
minutes. You can then use evernote search to find whatever you need
at a later date even if you don't bother to file these new notes.</li>
</ul>
</div>
</body></html>