09 Aug 2016

Switched Static Site Generator From Pelican To Hugo

Out with the old

I’ve been using the static site generator Pelican for a very long time to generate my blog. The problem I’ve had for a while with Pelican is that I had to also use virtualenv to isolate the dependencies, having to manually run make html to generate the html output. I was never able to get the development server setup with Pelican.

The deployment process was also a little painful but not really Pelican’s fault. I would first push to github then pull down the changes on my server and again run make html.

I’m sure this process could have been automated.

In with the new

Hugo is written in Go which means all I have to install is just a single binary and on OS X with homebrew it was as simple as;

brew install hugo

After installing, I just followed the quickstart guide to get started then I picked a theme - purehugo - that resembled my old blog. I tinkered the theme until I got it exactly the way I wanted it. This required me to edit and create some template files based on Go’s built in templating.

The hardest part of switching to Hugo was just updating the Front Matter that was already in place.

Now to deploy all I have to do is just rsync the output directory from Hugo to my server.

03 Aug 2013

Using Twython To Connect To The Twitter Streaming API via OAuth

Before you can connect to the Streaming API you need create a Twitter application. This will give you the necessary OAuth credentials. To do this, go to dev.twitter.com/apps, login to your Twitter account then click the Create a new application button and follow the instructions.

To connect to the Streaming using Twython, you need create a subclass of TwythonStreamer

from twython import TwythonStreamer

class TweetStreamer(TwythonStreamer):
    def on_success(self, data):
        if 'text' in data:
            print data['text'].encode('utf-8')

    def on_error(self, status_code, data):
        print status_code
        self.disconnect()

Now we will instanstiate the TweetStreamer class and pass in the oauth details


# replace these with the details from your Twitter Application
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''

streamer = TweetStreamer(consumer_key, consumer_secret,
                         access_token, access_token_secret)

streamer.statuses.filter(track = 'python')

The method on_success on the class TweetStreamer will get called for each tweet we receive from the streaming api. The statuses.filter call, will find tweets that contain the word python. Running this script will start printing tweets to the console.

10 May 2011

Export Test Cases From Quality Center Using Python

Here is a Python script that will export out test cases in a folder from Quality Center into a CSV file.

The following script will not handle Attachments. Will work on that later when I have time.

18 Mar 2010

Building a Twitter Filter With CherryPy, Redis, and tweetstream

Background

all the code is available at https://github.com/bulkan/queshuns

Since reading this post by Simon Willison I’ve been interested in Redis and have been following its development. After having a quick play around with Redis I’ve been looking for a project to work on that uses Redis as a data store. I then came across this blog post by Mirko Froehlich, in which he shows the steps and code to create a Twitter filter using Redis as the datastore and Sinatra as the web app. This blog post will explain how I created queshuns.com in Python and the various listed tools below.

Tools

  • tweetstream - provides the interface to the Twitter Streaming API
  • CherryPy - used for handling the web app side, no need for an ORM
  • Jinja2 - HTML templating
  • jQuery - for doing the AJAXy stuff and visual effects
  • redis-py - Python client for Redis
  • Redis - the “database”, look here for the documenation on how to install it


Retrieving tweets

The first thing we need to is retrieve tweets from the Twitter Streaming API. Thankfully there is already a Python module that provides a nice interface called tweetstream. For more information about tweetstream look at the Cheeseshop page for its usage guide.

Here is the code for the filter_daemon.py, which when executed as a script from the command-line will start streaming tweets from Twitter that contain the words “why”, “how”, “when”, “lol”, “feeling” and the tweet must end in a question mark.

In this script I define a class, FilterRedis which I use to abstract some methods that will be used by both filter_daemon.py and later by the web app itself.

The important part of this class is the push method, which will push data onto the tail of a Redis list. It also keeps a count of items and when it goes over the threshold of 100 items, it will trim starting from the head and the first 20th elements (or the oldest tweets).

The schema for the tweet data that gets pushed into the Redis list is a dictionary of values that gets jsonified (we can probably use then new Redis hash type);

{ ‘id’:“the tweet id”, ‘text’:“text of the tweet”, ‘username’:“, ‘userid’:“userid”, ‘name’: “name of the twitter user”, ‘profile_image_url’: “url to profile image”, ‘received_at’:time.time() }

‘received_at’ is important because we will be using that to find new tweets to display in the web app.

Web App

I picked CherryPy to write the web application, because I wanted to learn it for the future when I need to write a small web frontends that dont need an ORM. Also, CherryPy has a built-in HTTP server that is sufficient for websites with small loads, which I initially used to run queshuns.com it is now being run with mod_python. For templating, I used Jinja2 because its similair in syntax to the Django templating language that I am familiar with.

The following is the code for questions_app.py which is the CherryPy application.

The index (method) of the web app will get the all the tweets from Redis. The other exposed
function is latest which accepts an argument since which is used to get tweets that are newer (since is the latest tweets received_at value). nt is used to create a different URL each time so that IE doesn’t cache it. This method returns JSON at.

The templates are located in a directory called templates :)

Here is the template for the root/index of the site; index.jinja

This template will be used to render a list of tweets and also assign the first tweets recieved_at value to a variable on the window object. This is used by the refreshTweets function which will pass it on to /latest in a GET parameter. refreshTweets will try to get new tweets and prepend it to the content div and then slide the latest tweets. This is the template used to render the HTML for the latest tweets;

I explicitly set the the latest div to “display: none” so that I can animate it.

Now we should be able to run questions_daemon.py to start retrieving tweets then start questions_app.py to look at the web app. On your browser go to http://localhost:8080/ and if everything went correctly you should see a list of tweets that update every 10 seconds.

Thats it. Hope this was helpful.

18 Sep 2009

Running QTP tests using Python

QTP provides an interface called the automation object model. This model is essentially a COM interface providing a bunch of objects that can be used to automate QTP. The full object list is available in the QuickTest Professional Automation documentation.

Running QTP tests from the command line is useful for doing scheduled automatic testing. If you use a continuous integration system to do automatic builds of your software, you can run your QTP tests on the latest build.

The following is a Python script that is able to run a test and print out Passed or Failed. It is a direct port of example code in the documentation written in VBScript

15 Jan 2008

(Ugly) Python type checking

I like Python because of the explicitness of the syntax

def add(a,b):
return a + b


explicitness is good as it really leads to code that is understandable at one glance, but quick, tell me two types that the above function will work on ?

int’s and strings




>>> add(1,2)
3
>>> add(‘hello ‘,‘world’)
hello world



the above function works on both integers and strings only because both provide the special method add which gets called for the + operator.

So does this lead to implicitness ? Not really, because you should know (from programming) that you can add integers together and concatenate strings together, Python just makes this general across types.

If you wanted to say, restrict the types of the arguments to our add function above, you could do something like the following


def add(a,b):
if type(a)==type(str) and type(b)==type(str):
return a + b


Type checking is kind of ambiguous to me in a dynamic language. If i want to restrict the the ability of a function to only work with certain types or i don’t know the types of the object im passing to a function then i have design issues (or no design at all).

You could rewrite the above function to do the type checking using a decorator.

EDIT: i didn’t know if the following decorator was written by the original creators or not, but as i was pointed out it wasn’t here is the original link

Python Cookbook Recipe

def require(arg_name, allowed_types):
def make_wrapper(f):
if hasattr<
(f, "wrapped_args"):
wrapped_args = getattr(f, "wrapped_args")
else:
code = f.func_code
wrapped_args = list(code.co_varnames
[:code.co_argcount])

try:
arg_index = wrapped_args.index(arg_name)
except ValueError:
raise NameError, arg_name

def wrapper(
args, kwargs):
if len(args) > arg_index:
arg = args[arg_index]
else:
arg = kwargs[arg_name]

if not isinstance(arg, allowed_types):
type_list = " or ".join("'"
+ str(allowed_type.name) +
"'" for allowed_type in allowed_types)
raise TypeError, "Expected argument '%s' </span>
to be of type %s but it was of type '%s'." <br /> % (arg_name, type_list,
arg.class.name)

return f(*args,
kwargs)

wrapper.wrapped_args = wrapped_args
return wrapper

return make_wrapper

@require('a',str)
@require('b',str)
def add(a,b):
return a+b




>>>add(‘hello ‘,‘world’)
hello world
>>>add(‘hello’,1)
Traceback (most recent call last):
File “snippet3.py”, line 38, in
print add(‘hello’,2)
File “snippet3.py”, line 24, in wrapper
return f(*args, **kwargs)
File “snippet3.py”, line 22, in wrapper
raise TypeError, “Expected argument ‘%s’ to be of type %s
but it was of type ‘%s’.” % (arg_name,
type_list, arg.class.name)
TypeError: Expected argument ‘b’ to be of type
‘str’ but it was of type ‘int’.



above is code that does type checking on input arguments. I may be wrong and there may be use cases where you need to check the type of an object but the point is you should design your program so that you know all the involved types or use a language that has compile time type-checking.

Note: The above type checking is in a Django application that is live and it has users and i didn’t write that decorator.



13 Dec 2007

Never store passwords as clear text

Never store passwords as clear text, that is the general rule with any application that has a database backend that is used for authentication into the system. Why?

Basic authentication with a database usually works by comparing username and password combination that the user entered to values in the database table containing user details such as login name, password etc…It might be possible for a user with correct credentials to be able to inject SQL queries to the application, something like;

SELECT * FROM USERS;

assuming the user can guess or knows the table containing user data.

If passwords are in clear text then lo and behold the users now has access to all other users login name and passwords. Anyway this post is not about security of database backed applications but a post about how i overcame different versions of Python and py-bcrypt’s support.

As i’ve posted before i’ve been working at a web development company as a Python programmer. It’s a Zope ‘shop’ in the sense that there main application, developed in house a shopping cart system is developed using Zope. Anyway i just developed ‘External Scripts’ to do specialized stuff for customers. But recently i’ve been working on a ‘time/job tracking’ web application using web.py. I’ve mentioned this in a previous post. So here is the versions of Python,web.py and py-bcrypt that i used to develop the tracker;


  • Python 2.5.1

  • py-bcrypt-0.1 (Python 2.4 or higher)

  • MySQL-5



Anyway, this past week the application was put on a live production server. That has;


  • Python2.3

  • Postgres



I’ve also used decorators to ‘decorate’ functions to restrict access to certain URL paths. Take a guess in which version decorators was introduced into Python? You guessed right Python 2.4 got blessed with decorators. Guess what the lead/senior developer did ? He re-wrote most of the code to just use plain function calls instead of decorators…he re-wrote…instead of the simpler solution of installing Python >=2.4. (Converted the decorator to a plain function which is called in all other functions that was decorated with it).

I mentioned above that py-bcrypt requires Python >= 2.4 because it needs the function os.urandom which was introduced into Python 2.4. Oh slap! i’ve got Python 2.3 so we can’t generate hashes…the solution that was suggested to me was….“write/copy urandom” i think that was one of the moments in my not so long professional career that i thought that someone more senior and with more was wrong. I would have instead installed Python 2.5 on the server, which seemed to be the ‘path of least resistance’.

It seemed a daunting task. The first step i took was looking at py-bcrypt module. It contains two files;


  • init.py

  • _bcrytp.so



You can’t edit _bcrytp.so file as it a library file. So i looked at init.py, which imports the os module and defines the gensalt function. From my the Python 2.5 installation on my Mac i copied the urandom implementation into init.py just above the line where import os . urandom is not that complex , it just tries to open /dev/urandom and reads in n number of bytes and return it. So here is what init.py looks like after the changes, its a hack.

25 from os import O_RDONLY,read
26
27 def urandom(n):
28     """urandom(n) -> str
29
30     Return a string of n random bytes suitable for cryptographic use.
31
32     """
33”     try:
34         _urandomfd = open("/dev/urandom",r )
35     except (OSError, IOError):
36         raise Exception("/dev/urandom (or equivalent) not found")
37     bytes = ""
38
39     while len(bytes) < n:
40         bytes += _urandomfd.read(n-len(bytes))
41         #bytes += read(_urandomfd, n - len(bytes))
42
43     _urandomfd.close()

44     return bytes

45
46 import os

47 os.urandom = urandom
48 from _bcrypt import 


Now py-bcrypt works and passwords are hashable.

The thing that troubles me and is a question on my mind, is it worth the risk to install Python >= 2.4 on a server that contains ‘live shops’ ? The risk being totally blowing up the default Python installation (2.3) and bringing down the shops ? I would have probably installed a new version of Python and sandboxed it. The irony is that the senior developer was the one who chose py-crypt and told me to come up with a decorator for methods which need to be password protected. I would have thought that with his experience he would have guessed that the request for the app to go online would have come. Also if you are scared to blow the default installation of Python on the production server, WHY PUT AN IN HOUSE APPLICATION THERE?

25 Nov 2007

Exporting a csv file with web.py

This is how you export a csv file and get the browser to recognize that its a csv file and popups the download window with web.py. Lets say we have a database with a table called users and you want to create a csv file that contains all the users with their names and id’s here is how you do it.


1 class export:
2     def GET(self):
3         i = web.input()
4
5
6         users = web.select(’users’, vars=locals())
7
8         csv = []
9         csv.append(”id,name\n”)
10        for user in users:
11             row = []
12             row.append(user.id)
13             row.append(user.name)
14
15            csv.append(”,”.join(row))
16         #writer.writerow(row)
17
18         #f.close()
19
20     web.header(’Content-Type’,’text/csv’)
21     web.header(’Content-disposition’, ‘attachment; filename=export.csv’)
22     print “”.join(csv)
23     return




I export the csv file in a GET method of a class called export which i map in the urls list to ‘/export’,‘export’

A quick breakdown, do a database query and iterate over the IterBetter object create a row and appending a comma seperated string to the csv list. Then at the end you send the appropirate HTTP headers , the first telling the type of the file and the second setting the filename and extension.

Anyway you can download this code from http://bulkanix.pastebin.com/f1f567ea0

05 Nov 2007

embryo.py and py2app awesomeness

At work i created this script that changes permissions on our application BizarShop so that it works with the new Dashboard Widget to control the starting and stopping of of the Zope server. The permissions need to change because the controller that comes with BizarShop starts the Zope server as root, which creates the lock file (Data.fs file if it doesn’t exist) with root ownership. The widget on the other hand tries to control the server via normal user permissions, but the server wont start because all the files belong to root and cannot be overwritten. For example the file Z2.pid needs to be writable, so you need to change the ownership to that of the user.

So i created a python script that recursively goes through all directories under /Applications/Bizar Shop and changes all of the file/folder ownership to that of the owner current user. As you probably know to run a python script you either need to run explicitly via;

python scriptname.py

or by including a hash-bang at the start of the file to tell where python is located and then make the script executable. I thought that the user could just double click on an executable python script to run it but i was wrong. I didn’t want the user to open Terminal.app and execute it manually, this is just not user friendly. I then remembered py2app. From the README file of py2app

py2app is a Python setuptools command which will allow
you to make standalone Mac OS X application bundles
and plugins from Python scripts.

py2app is similar in purpose and design to py2exe for
Windows.

So using py2app i created this installer, that also includes the widget and embryo.py, oh and embryo.py is also a nice little module, from its Google Code description;

embryo is a tiny Mac/Windows/Linux GUI toolkit for Python. It can be used to “boot-strap” the user into downloading a larger GUI toolkit such as PyGTK, PyGame, pyglet, PyOpenGL, etc.

What i used it for is, my script checks if the folder /Applications/Bizar Shop exists and if it doesn’t then it assumes that BizarShop is not installed and then shows a message box saying BizarShop is not installed do you want to download. But if it does find it displays a message box letting the user now know that the Widget is about to be installed and Dashboard opens up a the install widget dialog box.

What is so cool is that py2app is very easy to use and it works! Combining this with embryo you can easily create a quick installation program for literally anything.

Oh, did i mention that embryo is created by Alex Holkner the same guy who is working on pyglet? Well now i did. Here are the links to these modules.

http://code.google.com/p/pyglet/
http://code.google.com/p/pyembryo/
http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html

04 Nov 2007

Redirecting stdout to StringIO object

So how do you redirect everything written to stdout and store it somewhere but also print it out to stdout? This was asked on #python and i answered it.

To access the stdout file object you need to import the sys module. Redirecting stdout to a StringIO object works because all functions that write to stdout expect the write() method of a file-like object, which StringIO has (along with read, seek etc). So here is the code;



So here is a quick breakdown line by line:


  • Lines 1 and 2 are used to import the required modules.

  • Then we subclass StringIO and create an attribute to hold the reference to stdout.

  • In Line 9 we overwrite the write method of the StringIO baseclass which does only one additional thing of writing back out to the original stdout, then it calls the baseclasses write method to store the string again.

  • Then also overwrite the read method that does one additional thing of seeking to the start of the StringIO object and then writing it all out back to stdout.




14 Sep 2007

Experimenting with Python frameworks and modules

I’ve been very busy since last semester got a new job as a Python (Zope) Developer. Then quitting from my previous job which i was still considered an intern. Now im in my second and last semester of university and I graduate at then end of this year!

Even though i haven’t had free time to actually write a post i’ve had some of those times were you need to do work but cbf so you end up doing totally random stuff? Well i’ve had those times a plenty in which i experimented with couple of Python frameworks and modules.

First one i tried was Turbogears. Remember that major project i have were we are using Rails well that’s what spurned me on to actually try a Python web application development framework. Oh and also the Python411 Podcast. Well installing Turbogears on a Mac is very easy by following this guide

  • I had Python2.5 already installed and the default version to run when python is run, so i skipped that step.
  • I had to install the easy_install using ez_setup.py
  • Then i downloaded tgsetup.py and ran it which downloads and install everything for you, except for the database wrapper.
  • The database i have installed on my Mac is MySQL 5.0 and the driver for that is MySQLdb. This was the tricky part of the whole installation process of Turbogears. So i downloaded MySQLdb and tried running sudo python setup.py install, but it failed. The solution to this failure was tricky but after a bit of Googling the problem was that setup_posix.py had the wrong path to mysql_config file which is located at /usr/local/mysql/bin/mysql_config on my Mac. Changing the path and running setup.py again worked. Then i tested it by trying to import MySQLdb from Python again and that seemed to work aswell (no Exceptions).
  • Then to actually test to see if Turbogears was correctly installed i ran this command tg-admin quickstart and entered a name for the project and package name (in this case it was wiki, creating wiki’s with web application development frameworks seems to be the Hello World program of languages). Then started the built-in webserver by running the script start-wiki.py and accessed it via http://localhost:8080/ and i got the “Welcome to TurboGears” page.
  • I haven’t done anything else with Turbogears as i was interested in the installation process.