It’s the beginning of a new year so I’m cleaning out some files in my drafts directory. This post was started on August 18, 2014.

In an effort to blog more regularly and to post content that is more mine than transcriptions of others, I’ll be writing a regular round-up of the events I attend. To start with, here are a few from the last three weeks.

Australian OpenStack User Group

I’ve recently started working at Anchor Systems and sit next to some of the people working on the new Anchor OpenStack cloud, so I’ve been hearing (and overhearing) quite a bit about OpenStack and OpenStack deployment recently. One of the things that sounded quite interesting is “OpenStack On OpenStack” (or “Triple O”) – which uses an existing OpenStack cloud to bootstrap and manage a new OpenStack cloud. The recent Australian OpenStack User Group at the Anchor Systems offices in Sydney had several talks about Triple O.

Sydney PostgreSQL User Group

I’ve always quite liked PostgreSQL but my current project is my first chance to work with it in a while. The Sydney PostgreSQL User Group meeting last week had a presentation by Venkata B Nagothi from Fujitsu Australia about the changes in the forthcoming PostgreSQL 9.4 release.

Port80 Sydney

I was a regular attendee at Port80 in Perth for years, but haven’t been to Post80 Sydney since I moved here 18 months ago. The meeting last week was hosted at Anchor, so I stuck around after work to eat some pizza and listen to the talk about user experience design.

Also: these are notes I’m typing during the sessions. They are probably errors and such. I’ll come back and tidy them up later. Maybe.

DjangoCon Australia 2013 t-shirt

Alex Gaynor on the Divided Web and the Role of Frameworks

Alex Gaynor on the Divided Web and the Role of Frameworks (slides). First person to have spoken at DjangoCons on three continents (for a hour or two).

History of the web: the big bang, ENIAC, Apple II, T.B.L. and Mosaic.

Email and FTP are more applications, but the web technologies (HTML, HTTP, etc.) are, effectively, platforms: they don’t have any specific thing you do with them, you do what you like.

1. Static content. In vim or something.

2. Dynamic content.

3. DHTML - nothing good came of this.

4. Microsoft invented AJAX (with ActiveX controls).

5. “Web 2.0” is what we think “web applicaiton” means these days.

Frameworks are being release (RoR & Django; see the Snakes and Rubies). These made lots of stuff that was a pain (“database driven applications”) more convenient.

Most people land on one side or the other, but current practice seems to have landed on “both”: backend and frontend are both getting more and more complex.

Technologies are living longer: some of this stuff is likely to be the COBOL of the future.

All these new tools and technologies don’t create new capabilities, they make existing capabilities more convenient to manage and to use. Rails and Django to let you do something new.

Django was designed to allow small groups of people to build complex “web applications” quickly. It still solves the same problem in 2013 as it did in 2005.

In 2009 there were a few more things that were though required: monitoring, message quues, APIs, workers, search indexing. Django has nothing to say about any of these (though many of them are addressed by third-party apps and external packages).

In 2013 you need deployment, configuration management, to think about SOA, real-time stuff (Websockets, etc.) and the crazy, crazy world of front-end Javascript.

We’re building more and more platforms for specific use cases.

In spite of all these new frameworks and tools addressing parts of a puzzle, nothing has really encroached on Django and Rails yet. Either Django will start addressing some of these cases or it’ll be made obsolete by something better.

Everything is getting more complex and bigger.

Deployment has always sucked but Alex has reason to believe that there is a bright future.

• Applications in Docker containers.

• Servers configured with Chef & Heat (orchestration as a service; part of OpenStack?) which run your containers.

Most full-service cloud platforms include some sort of orchestration tool (Heat, CloudFormation).

The “divide” of the title comes in a number of axes:

• small and big teams
• contractors and in-house
• “sites” and “apps”

The increase in complexity seems to be favouring one side of each divide. “The ink is never dry”.

Q&A

Where does Django fit in a world which almost forces us to work in (or, at least, interoperate with JS)?

DHH’s comment on Rails’ JS helpers: “Yes, JS is horrible.” Google is trying something interesting with Dart. It’s probably not particularly feasible to implement Python on top of JS (like the Dart compat).

“Best viewed in Chrome” must never come back.

Jacob Kaplan-Moss on Porting Django apps to Python 3

Hey y’all

Django 1.5 supports Python 3, in Django 1.6 it’s considered stable. So let’s all use it.

First question is: whether we should use Python 3? The answer is yes. Python 3 is Python with “30% fewer warts”. Biggest improvement is that Unicode really works.

Second question is: can we use Python 3? This is essentially a question of your application’s dependencies and whether they’ve been ported yet.

Problems:

• First big problem is choice ot DB. If you use Postgres you’re fine. SQLite works. MySQL has a connector (but there are licensing concerns, so the foundation can’t recommend it). You should be using Postgres.

• Second big problem is your server/deployment software: modwsgi, uWSGI and synchronous gunicorn are all fine. gunicorn async is problematic (due to gevent). You may need to switch deployment and operation strategy.

• South, Celery, Raven (sentry), django-extensions, all work. But not django-compressor, django-tagging (last update 2009), django-social-auth, django-debug-toolbar, Haystack. But there are suitable alternatives to these (django-pipeline, django-auth, django-taggit).

Every time working with Python 3 Jacob has had to play this game evaluating apps and libraries for Python 3 support.

Third question is: should I use Python 3? This can be complicated, with the additional costs of learning the new stuff, re-/evaluating apps and libraries for Python 3 support. Perhaps the best advice is to use Python 3 if you can and Python 2 if you must; Python 3 will be improved, get faster, etc. There’s no future for Python 2 (possibly decades, but still; RedHat have committed to supporting it for 2030 or something).

How do I use Python 3?

If you can, support only Python 3. It’s not too hard to add support for Python 2 later if it becomes necessary during the development process.

Python 3 introduced a tool called 2to3 which converts Python 2 code for Python 3 (by doing AST transformations and generating a patch). The Python 3 code being generated, makes it very hard to fix bugs and makes community contribution very hard.

The “single source” approach is much better: write Python 3 code and include shims to paper over the deficits when run under Python 2. If you have to support Python 2 you should be using this technique.

3. Supporting both Python 2 and Python 3 requires automated testing. You can’t hope to maintain it without continuous integration.

4. Fix all the unicode handling issues that your test suite identifies. There will be lots.

5. Iterate on test failures.

Case Study: python.org

Decided to support only Python 3.3+ (if possible). Left single-source open if it became necessary later.

Dependencies:

• Light CMS requirements. None of the existing CMS packages supported Python 3 yet, decided to build it as the requirements were so simple.

• Quite complex and specific user, permissions and authentication requirements, so none of the apps would have been applicable anyway.

• Heavy integration with social media services. None of the social login tools (which they evaluated) had been ported at the time, so they moved on to other features and bugged the maintainers to port (and it worked).

• Moderate traffic requirements with extreme spikes related to releases, etc.

• Testing: django-discover-runner works fine on Python 3 (and is, essentially, included in Django 1.6).

• The hardest part of the process was “retraining our unicode three muscles to do things the Python 3 way”.

  The biggest thing you can do is eliminate the word “string” from your vocabulary. Bytes are just bytes, text is text.

  django.utils.encoding does this.

It worked and it’ll be launching in about a month (hopefully).

They estimate the project cost about 20% more to develop, mostly in terms of increased development time. In large part, most of this time is probably just changing maintenance costs into upfront costs. And the number is probably smaller now.

Case study: django-sitetree

Django application for managing a hierarchical navigation tree. Ported by Jeff Triplett in a single commit.

Shared source approach with: Python 2.6+, Python 3.3+; Django 1.4+. This is a reasonable set of minimum requirements.

There aren’t any dependencies. If your app has dependencies, you may need to port them too!

Using Tox to run test suite in multiple environments. You’ll pick up a bunch of syntax issues with the changes in Python 3.

Django adds the @python_2_unicode_compat decorator to change __unicode__ into __str__ if running on Python 2.

Six (included in django.utils.six) helps to paper over some of the differences (metaclasses, type names of byte and text types, etc.)

There’s a guide for porting to Python 3 with detailed documentation.

Python 3 is ready, the WSGI containers are ported, the databases are ported, the frameworks are ported. Now it’s up to the community to port the libraries and applications.

Q&A

Is from future import unicode a good idea?

Unicode literals aren’t a thing in Python 3, but using stuff from future is probably a good idea.

Greg Turner on using FeinCMS

Greg is the filling in a core-contributor sandwich.

The web and web-content are becoming more complex: there are more technologies, more devices, users expect richer experiences. See NYT Snowfall for an example. Teams are getting smaller and smaller (economics, etc.) We need to build content which is future-proof, cross platform, clean and semantic.

These are the problems that FeinCMS attempts to solve. A document (page) combines a services of lumps of content (content types, not the same thing as Django content types).

See the [MCA site][].

FeinCMS provides a model and admin to use as a base in your own app. This allows site admins to combine and embed different types of content in their documents (instances of your FeinCMS models).

Each model has specific set of content types which are supported. This is kind of ugly, FeinCMStools makes it nicer. He’s made a demonstration (see the tags).

1. Define the regions in your content type. https://github.com/ixc/feincmstools-demo/blob/master/djangosite/meetups/models.py#L17

2. Define content types which can go in your regions. https://github.com/ixc/feincmstools-demo/blob/master/djangosite/meetups/models.py#L23

3. There are lots of tables, so make sure you’re careful with migrations, etc.

Content types don’t have to be content. Example of a content type, which offers the site admin a choice of layouts which have been designed and built ready for them to use.

Q&A

How does FeinCMS handle structured content?

They are just normal Django models, so they can be as structured and semantic as you like.

FeinCMS uses Django admin which some users dislike. Any tips on making it nicer?

Not really. But they use django-admintools. Would like to see a more heterogenous approach to admin, which many other CMS support and which better maps to user expectations.

Russell Keith-Magee on Secrets of the Testing Masters

CTO of TradesCloud. Hiring soon if you want to move to Perth.

Testing is good and we should all be doing it. As we all know. First contribution to django was adding app testing. It has lots of support for testing contrib, but not everything. Introducing two testing tools which are helpful and good and such.

Problem: You need test data

Django’s testing framework supports fixtures (declared in test case class). Fixtures are easy to create, but hard to read, hard to update and include no documentation. Also include a lot of rubbish which isn’t relevant to the test (1 line out of 18 when testing “admin can X”). You need to know that user 23 is the “staff user with no existing content”.

Solution is factory_boy: an EDSL for creating test objects in your tests, programmatically.

You create factory classes for your models. Provide place holder values, use helpers to generate values (sequences for unique email addresses, lazy values which are calculated late in the process).

Subfactories all you to create child objects. E.g. BlogPost factory has User sub-factory for the post author. Can still be overridden on specific cases when tests require. This overriding can traverse joins just like Django model stuff: PostF(author__firstname='thomas')

Lets you access and override various bits of instance data and the creation process.

Factory-boy also supports: SQLAlchemy and Mogo.

There’s also a django-factory-boy package which includes pre-defined factories for Django modules.

Problem: You’re dependant on an external service

Any external services that you use – payment processing, social networks, etc. – will affect your ability to test. You might also have trouble testing your failure cases.

Any internal services might be slow, unreliable or difficult to configure. Things like deliberately slow password hashing, database failure, etc.

You can just ignore this and not test those bits.

You can use mock (Python 3.3 stdlib unittest.mock, installable on Python 2.7+); essentially organised monkeypatching.

• Mock object which provides pre-defined return values.

• @patch decorators to inject mock objects into an existing API.

Mock instances track how they are used, allowing you to assert various properties about the way your code is used.

obj = SomeObject()
obj.method = Mock(return_value=3)
obj.method.assert_...

@patch allows you to use a function (the decorated function) to replace an existing function.

There are other similar packages, but Russell likes sticking with the standard library unless really required.

PS: There will be an announcement during Russell’s PyCon talk.

Simon Meers on Django Unstrained

Simon (DrMeers online) is part of the Django core team, having discovered Django in 2008. Lots of clients with deadlines.

Django (like all real Open Source projects) is our framework, it’s up to us to improve it. We probably want to avoid adding more functionality to Django core, but helping it to evolve to be an [even more] flexible platform for third-party contribution. Rich third-party eco-system.

• django-allauth

• django-braces

• django-generic containing snippets and patterns which you use over and over again. Simon’s snippets package is on his BitBucket account.

• django-import-export for CSV, etc.

• Nested inline forms has been discussed in tickets.

• Floppy forms and Crispy forms improve

• Pass through manager just passes things through to the underlying queryset? This is something I could have used before.

Simon’s Github account will contain the slides at some point.

Tom Eastman on the coolest parts of backend development with Django

Catalyst IT (in Wellington) have a growing Django team. Come talk to Tom if you might be interested in moving.

Large, reasonably high profile project for the Department of Education.

• AngularJS, D3, Compass/SASS for frontend.
• Django, gunicorn, tastypie, celery and south.

Working in a large team means that back end developers can avoid front-end stuff like browser quirks, etc. The job then has only three parts:

• Make it correct
• Make it fast
• Make it secure

Correct

Code without tests is broken by design.

And

The converse of test-driven development is faith-based development.

Your tests should be run regularly. Using a tool based on things like inotify allow you to see test results as you work.

You must see your tests fail before they pass. Without this, you can’t tell that your code makes the test pass. Either the code is wrong or the tests are wrong.

Testing errors is more important than testing the happy case. It’s unusual for people to test error cases (only usually when the errors occur during development). Write test cases which cover error cases.

Django gives you all the tools you need to test the things you need to test, and do it properly.

Continous integration and rigours code review are two of the best inventions ever. Tools like Jenkins and Gerrit make these reasonably easy to setup.

Took three tries to get integration testing to work:

1. Using the Selenium recorder. It generated hideous, unmaintainable code and gave up.

2. PhantomJS is headless webkit controlable with JS. It shows promise but was rather unstable.

3. Went back to Selenium having learned how to split tests from page controllers (see Effective Selenium testing). This makes it easy to maintain tests when pages change, etc.

Fast

Varnish and CDNs are all fun, but can be difficult on sites that contain non-public information, freshness requirements, etc. How do you keep your web site fast without public caching?

Using etag and last-modified headers in Django. Use auto-updating timestamps in models; with appropriate indexes in the database, it can be very fast to check last-modified headers. Takes more DB queries to check freshness, but is almost always a speedup; an example view takes 800ms to generate the full response vs 50ms to check modification.

1. Check authorisation.
2. If the etag matches, yay!
3. With no etag, serve from cache.
4. Generate (and cache) the page. (Invalidate cache on saves).

Secure

If you, as a web developer, haven’t read the OWASP Top 10 then you are a liability to every site you work on.

Things that seem innocuous – processing XML from clients – are often unsafe. Disable functionality that you don’t need: XML, YAML, etc.

If your site requires logins then your site requires SSL. See Firesheep if you haven’t seen it.

Django forms are for validating import; not just generating HTML forms. Use them for everything! Query strings, CSV records, etc. Even if you don’t (yet) have any rules to enforce, create a form so you’ll have somewhere to put it.

Django can’t protect you if you don’t let it. If you bypass the ORM, then it’s your job to handle SQL injection. If you bypass the template language, then it’s your job to validate.

Annoy your pentester so they really want to find your bugs. Current project has sent 18,000 emails about 500 errors.

Q&A

Adding tests to a codebase without them?

You need to be able to put your database into a known state for testing purposes. Doing smoke test (is it all 200 OK?) are quick and easy. Unit tests can be easy to start.

What are the most common issues found in code review?

Silly mistakes. Often, the review comment is “why wasn’t this tested?” Often it helps share knowledge about better ways, etc. throughout the whole team, to notice additional cases which need testing, etc.

If nothing else, it helps to make sure that the whole team knows about the code base.

Curtis Maloney on A state of REST

Curtis is responsible for:

• django-nap
• django-rated

Everyone who doesn’t hangout on irc://freenode.net#django is missing out.

Django doesn’t really have any REST frameworks; the API frameworks we have are pretty useful and kinda REST-y, though.

• TastyPie

• Django REST Framework

• django-nap tries to make a faster, smaller TastyPie.

There are a bunch of features for an API frameworks. Let’s look at a few of them.

Auth

Authentication – identifying the user – and authorisation – deciding whether the user can perform the operation.

TastyPie and DRF both include support for a bunch of authentication approaches. NAP makes/allows you to do it yourself using existing tools.

See the “Fuck OAuth 2 video”.

Both TastyPie and DRF include their own functionality; NAP allows you to provide a callback.

Data Formats

TastyPie and DRF both support many frameworks.

NAP supports a few, but only one at a time?

Serialisation

Shapes

TastyPie lets you define per-field visibility (list vs details)

NAP (and DRF?) you can specify it per-view.

Pagination

TP: Custom class, offset and limit

DRF: django native pagination. page and size.

NAT: django native pagination. Page with size or offset with limit.

Pagination has security implications, unlimited page sizes can allow denial of service.

Rate limiting

Both TastyPie and DRF have their own rate limiting implementations.

NAT doesn’t, because it’s a separate concern (in django-rated).

Filtering

Django ORM filtering is important, but a whitelist is important (don’t let people tranverse links).

Versioning

TastyPie: ?

DRF: Viewset (collection of views for responding to HTTP verbs for a collection of resources) with routing.

NAP: collect a bunch of Publishers into a version.

Generic Views

Can be clumsy to reuse TastyPie resources in your own views.

DRF is built on the generic views approach. It’s the first step in the tutorial.

Using HTTP responses which are also exceptions. So you can produce

Overview

TastyPie is a very quick way to expose a model as an API, but going further means learning a lot about it’s APIs and internals.

In compiling the talk, started to doubt continuing NAP at all because DRF is pretty awesome.

Core Developer Panel

Russell, Chris, JKM, Alex & Simon; moderated by Greg.

What’s the best and worst things about it?

Russell: The fact that it happens at all is pretty wonderful; but there’s so much more we could/should be doing.

Jacob: Watching people get into Python because of Django; the expectation was that it’d see a little takeup from newspaper people and a few who already know Python, seeing half a room say they learned the language because of Django is amazing. The idea that some people consider themselves Django developers (and not Python developers) is somewhat disheartening.

Are their situations in which you don’t use Django?

Alex: Doesn’t use Django much at all in his current role.

Chris: Not much, mostly when required.

Streamline vs bloat

Simon: Core should really, really be streamlined. But we need some more clarity about the one true tagging app, menu app, etc.

Russell: Flask does the apps thing well, by having a registry of packages and blesses specific implementations. Everyone should be able to know that pipeline is better than django-compressor.

Jacob: It doesn’t have to be the core team which does this; they’d much rather support a community effort.

Alex: It’s not going to happen (talked about it at three panels so far); also risks stagnation, etc. (Audience member says bollocks, repectfully)

Django 2.0

Jacob: there will never be a massive change destroying backward compatability as long as I have anything to do with it; 2.0 will just be 1.9++

Chris: deprecation cycles, etc. are good and breaking stuff is bad.

What would you add/remove from Django?

Russell: probably the stuff around apps/settings/startup. Settings have caused as many problems as it’s made. App labels (vs just module) are pretty awful. But still being Django would be difficult.

Alex: the admin; there’s been some reskinning but no real challenges to what django admin should be.

Jacob: Danny Greenfeld is trying Django Admin 2.

The Django Software Foundation

Core committers are all invited to be members, but not all choose to be.

Funding comes from a couple of places:

• Profits from DjangoCon US.

• Membership fees from corporate members.

It’s been very difficult getting companies to join, in part because it’s a US-based NFP. It’s also difficult to raise money without having something to spend it on.

Kickstarter campaigns, etc.

Alex: This is a good thing. Twisted have created a position to someone to review patches, etc.

Jacob: The Django community navigates the fraught relationship between OSS and money better than most. The community is pretty good at coming up with cash where it’s required to make things happen.

What will the replacement for Django be?

Alex: werkzeug?

Jacob: Have you watched the Meteor screencast? The RoR video in 2004 was a “holy shit” moment because “we have this” (hence open sourcing Django). This is a “holy shit” moment because we have nothing.

What should people sprint on?

Russell: 1.6 is coming so people should sprint on triage if they can help; Porting apps to Python 3; think and talk about the big issues.

Jacob: most of the low-hanging fruit have been picked and everything left is pretty hard. Pick a specific bug that’s addressing you, or work on a third-party package.

Dylan Jay on Pyramid, Django and Plone

There’s a Djangocon tradition of asking people from other Python web frameworks to speak so we can learn from them. This is that talk. Dylan is a core Plone contributor and has never written a single line of code for Django; first web app in 1995 with ISAPI.

Big explosion of web frameworks around 2004 of web frameworks, Rails, Django, etc. These are all descended from the CGI lineage.

Seperate lineage of Zope then ZTK and Plone. Zope is older than almost everything in the CGI lineage.

Third lineage of static site generates.

In 1996:

• Webservers serve files at paths.

• CGI = run a script

• OO was big

• CGI was shit, so dude wrote Zope based on objects with traversal and method invocation based on URL paths.

ZODB

ZTK:

• Turns out OO isn’t really all that good; use adapters (adaptor :: Interface i, j => i -> j)

• Traversal turns into a multiadaptor of Contect, Request and method name to a browser view.

Plone:

• 300+ core contributors
• 400+ plugins (supporting current version)
• 1000+ commits/month
• Sprints, a foundation, etc.

Plone services like ploud and pretagov.

Plone is good at ticking the all boxes you see in government tenders (all by itself).

The most important thing to consider when evaluating CMS platforms is roles: reviewers, content editors, site administrators, integrators, frontend developers, backend developers (in kind of increasing order of technical ability).

Many of these roles will be present in a large content-oriented site. Being able to separate and serve the needs of these different user groups.

Frameworks tend to have a straight-line effort/customisation trend, CMS tend to have non-linear relationship. Once you’ve exceded some limit of customisation, you’re often better off choosing a framework rather than a CMS.

A framework allows you to start with a pretty blank slate whereas a CMS let’s you build-up from an already-function-filled starting point.

Pyramid is pretty much the opposite of pyramids:

• Not ancient
• Extensible and flexible (not unchanging for centuries)
• Small to big (not big to small)

Lightning Talks

Tim Ansell talking about Tim’s Finance

Tim’s Finance is an interface to do various financial things.

Jacob Haslehurst on Permission Migrations

Writers, editors, administrators, developers have different permissions.

django-permission-migrations is a small package they’ve pulled out of their code base. Put permissions in the Meta of the model classes, create a permissions app for your project and add/remove permissions in the migrations, define groups in settings, then add migrations which control the permissions.

Chris on Navigation Menus

Lot’s of Django navigation menus have classes and models and database stuff. Chris has a simple template tag for doing doing navigation menus. Basically a helper to write code like:

{% if current == 'apple' %} class="active" {% endif %}

http://djangosnippets.org/snippets/1729

PANDAS loves Ponies

PANDAS is an R-like stats/data processing package for Python.

pandas-loves-ponies is a library which helps to interface Django with PANDAS (carry model field names into data frames and similar things).

Humphrey on form rendering

Rendering forms isn’t very DRY. Simple app with a template filter to render a form with a specific tempate.

https://github.com/humphrey/django_formrenderer

Danielle Madeley on Crisper keeps Crispy Forms crisper longer

Currently working on a form heavy app. django-crispy-forms is a django app to make forms a little DRYer. Crisper is an additional - related - template tag to help detect potential problems with forms, etc. Looks pretty cool.

Thanks

Thanks to all the speakers, sponsors, organisers, volunteers, AV team (the videos will be at pyvideo.org at some point).

On Saturday there’re two keynote sessions and five slots with competing sessions.

1. The Exportables session by Simon Hobbs will no doubt help in my quest to develop a reliable and efficient workflow at work.

2. Allie Micka’s keynote Running a Drupal business and contributing to Drupal sustainably is a keynote and sounds interesting enough, if not quite my cup of tea.

3. Dave Hall’s Building Distributions with Drupal 7 will, again, help in my quest for standardisation and automation to simplify workflows.

4. Josh Waihi’s keynote Goodbye Centralization, Hello Distribution: An Introduction to Git, is an introduction to git (which I already use and am relatively familiar with).

5. Given the number of times I’ve been asked about options for paid membership, the Paid Membership Sites session by Chris Hood will scratch an itch of mine (which other people have been tickling).

6. As I’ve implemented two or three custom CCK fields (and others on my team have implemented another few), Ivan Zugec’s session on Fields in Core: How to create a custom field is sure to be interesting.

7. Dries’ keynote session with details to come is by Dries. And a keynote. And by Dries.

My choices on Sunday are less clear cut in spite of having a choice in only four slots:

1. I expect John Zornig session about Building a document management system in Drupal to be inspiring when combined with my recent interest in Solr-based search.

2. Both Turbo charge your Drupal site by Kim Pepper and Fancy Pants WYSIWYG Editing by Boris Gordon sound interesting and useful.

3. Session by Dries.

4. I’ve got a bunch of projects coming up at work that need payment solutions of various types so The Payment api and its framework approach to module development by Allie Micka will obviously be useful, but I’m personally interested in systems administration and performance so Horizontally Scaling Drupal by Dave Hall sounds interesting. Simple page layouts and customisations using Views, CCK and CSS by Dale Baldwin sounds a bit entry level.

5. Both Case-Study: ABC Dig Music by David Peterson and Migrating to a Drupal Web Environment by Fiona Bush sound like they’ll have interesting insights. I’ll probably go Fiona’s session, if only because I went to ANU for a while.

6. Josh Koenig’s keynote session details to come.

7. Lightning talks are alway good (or, at least, over quickly).

Now that I’ve thought about it these plans will be making their way into my calendar.

Today I had to look at a client’s web-site to resolve a performance problem that I’d heard of a few times but had never been able to reproduce. The reports looked something like this:

Can you look take a look at our site again? It’s very slow to load the page but all the other sites I try are fast. It works find on [manager’s] Mac but my PC and the rest are all slow.

First things first, I fired up a Windows install in Virtual Box and found that, yes, it was indeed slow on IE (but not on any other browser I tried).

Given that there’s a bit of fancy schmancy JavaScript going that was my next port of call. After a while trying to find the option to disable JavaScript (Internet Explorer has one of the worst configuration interfaces of pretty much any software I’ve used) I remembered that this install had the developer toolbar installed. A quick navigate through three or four layers of menu (Seriously Microsoft: try giving some consideration to user experience one day. You might find you like it!) and a visit to Disable -> Scripts and I’d confirmed that yes, the problem was only seen with scripting enabled.

There are various bits of jQuery used on the site including a light-box popup thing, a slideshow type arrangement and a few custom bits. I cloned the site into my development environment and prepared to set to with my trusty /* */. Horror of horrors, the problem went away. A few goes back and forth over hosts and it was clear – the problem happened “live” (with JavaScript) but didn’t happen at all locally. Crap.

As a last resort before breaking a live site to try to diagnose the issue I appealed to Twitter. A few responses from some helpful sorts didn’t point out any obvious things to check (and pretty much discounted my own theory of blaming: my favourite of Microsoft’s misfeatures, filters). In a last ditch effort to find the problem without disabling bits of the live site I appealed again to Twitter for someone to jog my memory. I remembered a site that I’d seen before (in mid-October, it turns out) that provided performance data and screenshots from real browsers. Nick Cowie replied with the linked that saved me hours: I was looking for webpagetest.org!

WebPagetest is a free web-page performance service. Give it a URL (and maybe set some options), wait a little while, and you get back report about your site. Like most such things you get a screen shot, but more usefully WebPagetest provides a timeline of fetching and rendering the page. Here’s an edited screenshot of that first report:

Notice, if you will, all that red and yellow and the fact that the initial request took 53 seconds and subsequent requests took 31 seconds! A closer look at the waterfall chart shows that around a third of the time to load and render the page comes from those red requests – about a dozen requests that result in 404s – so I set about fixing them. It was evident that they were images that form the border of the Fancybox “popup” and a quick grep saw that they were referenced from fancybox.css with a relative path. I was wrong when I though that filters weren’t to blame: Microsoft’s filter hack interprets paths to images relative to the document being displayed not the stylesheet it is defined in (unlike every other reference to an external resource in a stylesheet). One such error might not be a problem: a silly, preventable mistake, but not critical (beyond the resource not being loaded and used). A dozen means that IE stalls for 12 URLs / 2 connections * load time which can be ages if your slowish CMS is serving the 404 pages.

I fixed the missing files (by duplicating them to the expected location, I’m afraid) and the load time dropped by some 20 seconds. Job done.

All that yellow looked a bit dodgy though, so I kept going. Most of that is indicating requests that could have been cached on the client but weren’t because of configuration issues. I added some ExpiresDefault server directives and pretty much all that yellow went away:

Resolving a few 404 errors that slipped through the obviously gaping cracks in our QA procedure and correcting some server configuration oversights took this site from load and render times of 53 seconds and 31 seconds (initial and subsequent, respectively) down to 23 seconds and 2.6 seconds!

There are three things that have come out of this experience:

1. The client’s problem has finally been resolved.

2. It’s pretty clear that I’ve got a lot of work to do improving our quality assurance procedures.

3. It’s also clear that developing a server configuration with reasonable defaults will result in some dramatic performance improvements.

Now it’s time to fill these gaps so that we catch this type of issue and pick these low-hanging fruit in our current and future work.

I pared it back from Mark’s more complex use case and reworked the templates and stylesheets a bit to make them more HTML5-y, but there’s still a way to go. Eventually, I’d like to use the new semantic tags throughout and make the design my own.

I tweaked the 404 page to (naïvely) extract the words from bad URLs and pre-fill the Google search box. Still on the search engines theme I also added an XML Sitemap template – which lists the index, archive, and every post – and a robots.txt which references it.

Following Benjamin Thomas’ guide I now use a custom jekyll command which loads extension code when processing the site (without having to fork Jekyll itself). At the moment, this is just loading Jack Moffitt’s html_truncatewords filter.

The new filter is used in the index, archives, and Atom feed templates and the <meta> tags to generate a description when a post does not have an excerpt. I plan on extending the set of meta-data included in each page to include as much of the Dublin Core Element Set as makes sense. Some of it will come from the site and post YAML data, but others will need custom Liquid tags to determine the correct values automatically.

In all, it’s been a busy couple of evenings but Jekyll is refreshingly simply and delightfully easy to start hacking with.

I’m currently planning to attend the keynote presentations:

• Anil Dash – Next Year’s Web
• Alex Payne – The Importance of Language: How The Technologies We Use Shape the Web; and
• Derek Powazek – Wisdom of Community. And the panel thingo.

And I’ve decided on the following sessions:

• Lachlan Hardy – Building Better, Cheaper Sites on the Open Web
• Dmitry Baranovskiy – Writing Better JavaScript Libraries
• Kevin Yank – CSS Frameworks: Make the Right Choice
• Darcy Laycock & Matt Didcoe – Making your website aware/geoaware
• Justin Mclean – A practical guide to connecting hardware to the web

That leaves only the Wednesday afternoon session to decided. I’m not much for lawyers or the law, so I’m going to have to choose between:

• Nick Cowie – Progressive Enhancement with CSS: Or how I stopped worrying about IE6 and started loving CSS3; and
• Myles Eftos – Stuff They Never Taught You at Website School.

On Friday I’m going to two of the workshops:

• Twitter Platform In-Depth – Alex Payne; and
• JavaScript: beyond the DOM and AJAX – Dmitry Baranovskiy.

In addition to the conference, I’m going to a few of the fringe events:

• the inaugural UX Bookclub
• the Perth Port80
• Webjam 11

A busy week.

Installing the software

There are two important pieces of software to install before we can build our application: Apache with mod_fastcgi, and the Haskell FastCGI library. You should install Apache as you normally do. This probably involves one of the following commands:

    sudo pacman -S apache # On Arch Linux
    sudo aptitude install apache2 # On Debian, Ubuntu, etc.
    sudo port install apache2 # On Mac OS X with Macports

Installing the FastCGI module

Next, you will need to install mod_fastcgi. This may be packaged for your system, in which case you can use pacman, aptitude, port, etc. as appropriate. If it is not, you’ll need to build it. Happily, this is dead easy:

Download and unpack the mod_fastcgi source from http://www.fastcgi.com/dist/. Edit Makefile.AP2 to point to your Apache root (the directory which contains build/). In my case (on Arch Linux), this was /usr/lib/httpd/. Compile and install it with make -f Makefile.AP2 && make -f Makefile.AP2 install.

Next, you’ll need to enable the module. The “correct” way to do this varies wildly from distribution to distribution. In particular, Debian-based systems like to spread this out over some three or four files.

Edit your Apache configuration (mine was in /etc/httpd/conf/httpd.conf) to load the module:

    LoadModule fastcgi_module modules/mod_fastcgi.so

There are a number of ways to configure the module, but the easiest is to add it as a handler for .fcgi files. Add the following snippet to httpd.conf:

    <IfModule fastcgi_module>
        AddHandle fastcgi-script .fcgi
    </IfModule>

You’ll still need to add the ExecCGI option to the directories which contain the .fcgi files to be run. I’m planning to use my ~/public_html/ directory, so I amended the <Directory> section in /etc/httpd/conf/extras/httpd-userdir.conf:

    Options +ExecCGI

Restart Apache and you’re good to go. If anything goes wrong, try looking at the mod_fastcgi documentation.

Installing the Haskell libraries

Installing the Haskell libraries should be a little more straightforward: just use Cabal.

1. cabal update
2. cabal install --global cgi
3. cabal install --global fastcgi

If your cabal does not know how to become root, you may need to prefix those cabal install commands with sudo (or, better yet, edit ~/.cabal/config to specify root-cmd: sudo).

Your environment is now ready to build and run FastCGI applications in Haskell.

Building a test application

A simple Haskell FastCGI application looks like this (largely cribbed from Paul Brown’s blog):

    module Main ( main ) where

    import Control.Concurrent
    import Network.FastCGI

    action :: CGI CGIResult
    action = do
        setHeader "Content-type" "text/plain"
        tid <- liftIO myThreadId
        output $ unlines 
            [ "I am a FastCGI process!"
            , "Hear me roar!"
            , ""
            , show tid
            ]

    main :: IO ()
    main = runFastCGIConcurrent' forkIO 10 action

Copy and paste that code into a file (I’ll call it Main.hs) and compile it:

    ghc -threaded --make -o ~/test.fcgi Main.hs

You’ll now have a file ~/public_html/test.fcgi which you can call using curl. The first request might be a little slow, but subsequent requests should be much faster:

    curl --include http://localhost/~$USER/test.fcgi

Congratulations! You now have a working Haskell FastCGI application hosted under Apache. If you have any suggestions, comments, or questions, please leave a comment.

The only way, as far as I (and the replies on django-users@), is to write a template filter to extract a such a value for you. Thankfully, this it pretty easy:

    @register.filter
    def field_type(field):
        """
        Get the name of the field class.
        """
        if hasattr(field, 'field'):
            field = field.field
        s = str(type(field.widget).__name__)
        s = s.rpartition('Input')[0]
        s = s.lower()
        return s

I can use the above filter to add a class to my form markup:

            <div class="form-field form-{{ field|field_type }}">
                {{ field.label_tag}}
                {{ field.errors }}
                {{ field }}
            </div>

Which I can then use in my CSS to style particular widgets without having to do them one-by-one using ID’s:

.form-field label {
    display: block;
    width: 15%;
    float: left;
}

div.form-checkbox label {
    width: 85%;
    float: right;
    clear: right;
}
div.form-checkbox input {
    float: left;
    margin-left: 12%;
}