It’s the beginning of a new year so I’m cleaning out some files in my drafts directory. This post was started on August 18, 2014.
In an effort to blog more regularly and to post content that is more mine than transcriptions of others, I’ll be writing a regular round-up of the events I attend. To start with, here are a few from the last three weeks.
I’ve recently started working at Anchor Systems and sit next to some of the people working on the new Anchor OpenStack cloud, so I’ve been hearing (and overhearing) quite a bit about OpenStack and OpenStack deployment recently. One of the things that sounded quite interesting is “OpenStack On OpenStack” (or “Triple O”) – which uses an existing OpenStack cloud to bootstrap and manage a new OpenStack cloud. The recent Australian OpenStack User Group at the Anchor Systems offices in Sydney had several talks about Triple O.
I’ve always quite liked PostgreSQL but my current project is my first chance to work with it in a while. The Sydney PostgreSQL User Group meeting last week had a presentation by Venkata B Nagothi from Fujitsu Australia about the changes in the forthcoming PostgreSQL 9.4 release.
I was a regular attendee at Port80 in Perth for years, but haven’t been to Post80 Sydney since I moved here 18 months ago. The meeting last week was hosted at Anchor, so I stuck around after work to eat some pizza and listen to the talk about user experience design.
]]>Sharif Olorin from Anchor systems spoke about monitoring Elasticsearch clusters; and
Clinton Gormley, a core developer, gave an overview of the changes in the impending 1.0 release.
Sharif is a developer and system administrator at Anchor Systems and has been working with Elasticsearch for about a year.
He highlighted a number of key points to be considered by anyone who is monitoring an Elasticsearch cluster (in no particular order):
You should monitor every metric that you can get your hands on and keep as much data for as long as you can, just in case. Sharif described several cases where having data available made debugging problems observed in production much easier.
While you should monitor everything, you should only alter on metrics people care about. Being woken up at 3AM is pretty bad, but it’s worse when the cause is not really a problem! Loss of redundancy, for example, probably isn’t worth getting out of bed for; provisioning a new node can wait until morning.
You should monitor and alert from your entire cluster, not just from some node’s individual opinion about the whole cluster. There are a number of problem conditions that can be difficult to accurately detect without having a “whole cluster” view. Whole-cluster monitoring, though, doesn’t play nicely with most host-based monitoring tools; you’ll probably need to define your own custom checks which know how to interrogate the whole cluster.
Given that they’ll be checking every node in a cluster, these checks will need to be highly concurrent and very fast. Sharif showed us a split brain check he wrote in Golang.
You should automate recovery from as many alert conditions as you can. Where it’s not possible to automatically recover from an error condition, you should aim to respond sensibly. Sharif described an example, in which an alert trigger by a split-brain in the cluster might automatically switch all nodes into read-only mode to prevent divergence.
Use the statistics API as a source of many useful metrics about your nodes and their opinions about the cluster. It also exposes a bunch of generic stuff about the JVM and things.
Finally, Sharif gave a few tips about tuning Elasticsearch clusters. The core of his advice boiled down to taking a principled approach: tuning individual parameters, reproducing each test as closely as possible (same time of day, etc.), consider actual numbers (not just graphs), etc. Essentially: use science!
Sharif’s slides are available on Speaker Deck; I’ve probably got a bunch of stuff wrong here, so you should probably go and review them for yourself!
Clinton works for Elasticsearch where he develops the Perl client libraries, does training and evangelising, “keeps Elasticsearch honest” and some other stuff. He gave us a run down on the new features and other improvements in the forthcoming Elasticsearch 1.0 release.
Amongst the many things mentioned, these few stuck out to me:
You cannot currently use different versions of Elasticsearch in the same cluster; upgrades involve tearing down your entire cluster and bringing up a new one (possibly not in that order). 1.0 will allow rolling upgrades of your nodes without having to do the whole cluster in one fell swoop.
You can backup the data in current Elasticsearch clusters, but it’s very much a do-it-yourself process: disable flushing, find all primary shard locations, copy their files, enable flushing. Version 1.0 will provide API methods to trigger a snapshot which will be written to a configured repository (S3, HDFS, etc.) Comparable changes have been made to the process of restoring a snapshot: the current manual process will be replaced with a few API calls.
The percolator functionality – which allows applications to do things like reverse search, alerts, and updatable result sets – is now implemented in a way which lets it scale as well as any other index in the cluster. It also supports multiple indices, aliases, and has a bunch of other improvements.
The new cat API provides direct access to a range of metrics in human-friendly formats (i.e. not large JSON documents). This includes a bunch of things that humans and monitoring systems are often curious about: sizes, counts, statuses, etc.
The existing support for facets has been drastically improved with a new feature called aggregations. These allow you to express a bunch of things which aren’t really expressible with traditional facets. These looked very powerful and very cool!
Clinton’s slides (originally prepared by Igor Motov) are available on Speaker Deck; go read them!
The questions prompted a few interesting details: Elasticsearch have just hired a few machine learning people to work on the product; they aren’t yet sure what, specifically, they’ll be working on, but we can expect some learning-type things in Elasticsearch soon.
]]>The International Conference on Functional Programming (ICFP for short) is a three-day core conference and is collocated with a number of related events. ICFP 2013 was in Boston and a number of FP-Syd regulars presented and/or attended.
Erik was a long time LCA attendee but ICFP has supplanted it as his “must go” conference. I hope to make the same switch in 2014!
The Haskell Implementers Workshop covers the internals of Haskell
implementations which, these days, means GHC to a very large extent. Covers a
lot of interesting techniques, with a particular focus on compilers. Erik
mentioned work on the non-safety of generalised newtype
deriving; using
Hermit (a dynamic/guided optimisation framework) to optimise
scrap-your-boilerplate code; and Habit (a strict Haskell dialect for OS
programming).
The Commercial Users of Functional Programming was, reportedly, a bit boring, but I’ve liked the few CUFP 2013 YouTube videos I’ve watched so far. YMMV.
The Haskell Symposium was a main draw (for Erik). Highlights which Erik found worth mentioning and I found worth noting down include:
Oleg asked difficult questions of a lot of speakers. I wonder what would happen if he asked an easy one?
Effects seemed something of a hot topic.
Demonstrations of Liquid Haskell (which sounds pretty great), and a Javascript backend for GHC.
Intel are developing a research compiler which uses GHC’s front-end to compile to Core and then uses their own backend. Does loop vectorisation, currently only better performance on a few benchmarks.
The third iteration of the I/O manager for GHC. Multithreaded, influenced by Kazu Yamamoto’s work on Warp and mighttpd. Benchmarks against Nginx seem very good; Warp with multiple cores sees extremely good speedups (contra Nginx).
The main event – ICFP – is an academic conference and a lot of the content will fly straight over the head of many a “working programmer”. Some of the highlights included:
A few talks on vectorisation (w/ SIMD from Intel, stream fusion, etc.) and optimisation (for GPUs, etc.)
A few talks on dependent types.
Tactics in Coq are untyped; one talk discussed an approach to typed tactic programming in Coq. Sounds especially interesting now that there is a “Coq fight” in the FP-Syd calendar for next year!
People who didn’t attend are encourage to watch the video of the “fun with semi-rings” talk. I haven’t been able to find it, though.
One talk described a useful-sounding approach to parsing context free grammars with a divide-and-conquer approach, allowing partial and parallel parsing.
Simon Peyton-Jones discussed the new curriculum for secondary computer science education in the United Kingdom.
An extension or two to System F: System Fc (explicitly kind equality) and System Fi (type indices). Everyone who can understand System F shouldn’t have a problem reading the System Fi paper.)
Constrained monad problem (which, apparently, Oleg said was crap?). Paper on
solving a problem which occurs when using Monad
but they should have used
Applicative
. Seems as though they mostly wanted the do
syntactic sugar;
see also idiom brackets and the attempt to generalise the Monad sugar.
“Querying ordered graphs.” Three words which sound interesting, but I’ve no idea why I wrote them down.
Also: experience reports! Someone took a Scheme compiler from 4-5 to 25 passes (“nanoparsing”?) and, at the same time, also added a good colouring register allocator. Apparently one of these changes made it better.
Other events:
Talking about a benchmark/framework to compare approaches to generic programming at the Workshop on Generic Programming.
Brent Yorgey doing animations with diagrams
at the Workshop on Functional
Art, Music, Modeling and Design.
Chordify is a system (written in Haskell) to analyse recordings and generate chord transcripts. It’s not perfect but gives pretty good approximations.
Ben Lippmeier – an FP-Syd regular – presented a paper at ICFP and reprised that presentation back in Sydney for those of use who weren’t in Boston. He described an approach using data flow to guide the compilation of programs using stream fusion.
Wants to process a list of points, adding 1 to each, filtering those about 0 and also finding the maximum.
Doing stream fusion
map f = unstream . mapsS f . stream
filter f = unstream . filterS f . stream
-- RULE to remove (stream . unstream)
Example computes (vec3, n)
can’t float vec3
because it’s being used in the
result and in the computation of n
. So we get two loops.
**1** -> 2 -> **3** -> 4
| |
( , )
zipWithX
tends to use X+1 loop counters for stream fusion. There’re only 8
registeres to use on some platforms.
Split filter
into two combinators flag
– which contains True
or False
for each member – and pack
– which does the filtering.
This code generates the data flow graph.
fun vec1 (\s1 ->
let s2 = map (+ 1) s1
flags = map (> 0) s2
in mkSel flags (\sel ->
let s3 = pack sel s2
vec3 = create s3
n = fold max 0 s3
in (vec3, n)))
vec1 :: Vector Int
s1 :: Series k1 Int
s2 :: Series k1 Int
flags :: Sel k1 k2
s3 :: Series k2 Int
Series has a phantom type variable which helps keep track of the code which can be fused into a single loop.
We learn that k1 >= k2
With the flow graph (annotated with operations, etc.), throw away the source.
Abstract loop nest:
loop k1 {
start: ....
body: ....
inner: ...
end: ...
} yields ...
Start at the front of the data flow graph and add elements of the graph to the nested abstract loop.
Operations go into different places in the nested abstract loop. A fold
, for
example, allocates and accumulator in start
, increments somewhere within
body
and reads it in end
.
Translate the various bits and pieces of the abstract loop nest data structure into different Haskell combinators.
GHC plugin which grabs Core, does data flow compilation and generates Core to give back to GHC.
Some issues in current implementation where LLVM doesn’t realise that writing to the output doesn’t need to reload the start and length numbers.
If your program is first order (argument functions take scalars, not series), non-recursive, synchronous, finite data flow program using out combinators.
Then by construction your program will be compiled correctly by this system.
Liam O’Connor works for NICTA. Instead of talking about something he recently learned, he’s talking about work: CDSL - a restricted functional language for file system verification.
Trying to establish a formal proof of the correctness of a file system driver in an operating system.
Already have an architecture for this sort of problem (from seL4):
Abstract spec - high-level, nondeterministic (followed by an “interesting” proof of relation to ~ 15%)
Low level spec - purely functional (followed by a “largely boring” proof of relation to ~ 30%)
C implementation - efficient.
~ 55% is showing that the other proofs don’t do something stupid; proving invariants all hold.
Ignoring the kernel proper, architecture support, and drivers (another NICTA
project), the largest part of the Linux kernel is the fs/
directory; 31
different file systems were supported by the kernel running on some random
NICTA server.
There are lots of file systems with, one assumes, quite a lot of common functionality and infrastructure. The goal of the project is not to make a cathedral of a single verified file system, more a factory for churning out numerous file systems. The approach is to use a DSL to generate the low-level spec, proof and implementation. High-level spec and proof are done by hand, so generated outputs need to be readable.
Should
establish key verification properties
compete with efficient C code (imperative, destructive updates, etc.)
be expressive enough to write a file system
But:
Simple-typed lambda calculus is strongly normalising (you can’t write general recursion, e.g. the Y combinator).
First-order language: lambdas go away, use let
binding and restrict to
defining top-level functions. Added structural rules for mixing, weakening, ?
Need to do memory management which is safe, expressive (no pass by value, we need the heap), no GC (you’d have to verify it, introduce latency, etc.)
Automatic member management (GC) is too big a burden. Many static automatic memory management is inefficient or unsafe.
What about manual memory management?
let x = allocateData ()
= updateData x
x' = free x
_ in x'
But this is terrible! Unsafe, inefficient, etc.
So have a linear type system, throwing away weakening, etc. Forces use of things exactly matching (can’t alloc and not use, doesn’t discharge the new fact). The typing rules require that introduction and elemination be paired.
Linear types means that the elimination operations (e.g. updateDate
) are the
last to access terms, so they can do destructive updates.
Two interpresations of these semantics:
value semantics: pass by value, no heap, immutability, reasoning.
update semantics: heap, updates, deallocates, implementation.
Linear types allow for both.
But sometimes you want non-linear, pass-by-value (arithmetic operations, etc.):
Allow structural rules (dereliction and contraction) for certain types only. So
now we have T_{.}
and T_{#}
(unboxed and value types).
make : () -> .Buf
free : .Buf -> ()
length : .Buf -> (#U32, .Buf)
serialise : (.Obj, .Buf) -> (.Obj, .Buf)
deserialise : .Buf -> (.Obj, .Buf)
Non-linear “look but don’t touch” references with *
:
make : () -> .Buf
free : .Buf -> ()
length : *Buf -> #U32
serialise : (*Obj, .Buf) -> .Buf
deserialise : *Buf -> .Obj
Use let!
construct which is like let
but we mark specific variables as
read-only within the let
clauses and back to linear in the in
.
But this is unsafe (read-only can escape the let). Could use regions, but choose not to unless it’s required.
Linear typing breaks some control flow:
let x = alloc ()
in if cond
then update(x)
else x
Hardest, most annoying part of the formalisation of the language.
Built-in loop combinators, map, fold, with, for.
let sum = for (x,y) in fold(arr) with 0
do (x + y)
let arr', sum = for (x,y) in map(arr) with 0
do (x * 2, x + y)
Alas, this is unsafe. Double free, etc. But you can restrict linear types in the loop expression. Then have to make any required linear types into accumulator parms.
The return-code convention using in languages like C is pretty bad. Instead, separate statements and expressions.
Statements have three types:
Type of if then else
is T_{t} \leastupperbound T_{e}
. Lattice join,
subtype, etc.
Make let
and let!
only handle success cases. Force sub-expressions to
handle potential errors. Type system forces you to handle your errors and the
linear type system forces you to free your resources.
Product and sum types (implemented as structs and tagged unions).
Accessing members of linear records is problematic as you use the record multiple times:
let sum = operation(x.field1, x.field2)
Instead use an open/close structure.
]]>Tran gave an experience report using array computation libraries in Haskell (Repa and Accelerate) to implement cellular automata. “Falling sand” game: simulate gravity and “alchemical” interactions between elements.
First step is in simulating gravity. Dealing with falling blocks, randomising
block. Using “block CA”, with blocks defined by grids (2x2 cells) which
alternate between time steps (red grid, then blue grid); this allows you to
implement gravity using a single rule [: ] -> [..]
.
Repa and Accelerate both have concept of stencil convolutions. Use them to implement CA rules. Stencil has a shape (the neighbourhood) and a fold-ish function to process each cell.
Phrase the problem in terms of array computation.
Repa: slap Repa functions onto standard Haskell code.
Accelerate: EDSL means you can’t do lots of Haskell stuff (little things like pattern matching).
Repa has Gloss integration. Hmm.
Code is on GitHub. Called falling-turnip?
Andrew Frederick Cowie
twitter.com/afcowie
AfC on #haskell
Cloudy stuff means there’s never only one of things these days; everything is distributed (or will, hopefully, need to be distributed).
Streaming I/O: iteratee, conduits, io-streams, pipes all provide abstractions for processing data in a single thread in a single process.
Pipes explicitly talks about clients and servers (ends of the pipelines) but this is all just structuring computations within a single thread. This doesn’t really help us do anything interesting to build distributed systems.
AWS regions and availability zones; no SLA unless your app spans availability zones.
CAP theorem: consistency, availability and resilience to network partition.
Two generals: attack succeeds iff both attack at the same time. No solution can guarantee coordination in the face of unreliable messaging.
Jepsen blog posts about testing distributed databases.
Computers are slow.
Author mention “CRDTs” a few times (difficult to Google; did you mean “credit”?); original meant something like Convergent and Commutative Replicated Data Types, now Conflict-free Replicated Data Types. Paper in 2007 from INRIA. Is it possible to arrange things so that distributed states will always converge? Yes, joint semi-lattices.
Paper proposes a protocol and implementations of various abstractions on top of it: counters, registers, sets (grow only - G-Set, add & remove - 2P-Set, observer-remove - OR-Set), graphs.
Global invariants can’t be enforced over the whole system without synchronisation; eventual consistency allows you to accept changes which, after merge, break the invariant.
Paxos algorithm (Lamport. 1998. The Part-Time Parliament), peer-to-peer consensus algorithm; most people don’t bother trying to implement a full peer-to-peer consensus system.
ZooKeeper elects a leader.
Raft. There is just one leader and the point of the algorithm is to ensure that the leader’s log is the most correct. Leaders have terms? Consensus is something like “3 nodes agree”.
Ceph - large distributed file system. Lots of nodes, three of them are special “monitors” which maintain the cluster map. It’s pretty hard to build a file system if your objects (disk blocks) change underneath you; Ceph have focussed on the consistency needed to build a file system.
Rather than maintain an index (which doesn’t scale), Ceph places blocks
according to a layout algorithm. CRUSH()
. Paxos to elect monitors.
All this stuff needs very fast interconnect, i.e. a single data centre.
Amazon’s SQS:
Reliable delivery of all messages.
Delivery is not ordered. (Not much of a “queue” is it?)
Delivery is guaranteed at least once.
We’re now back where we started: workers that recieve messages need to be able to figure out whether the message needs processing (at least once), etc.
Queues like this are good for signalling, but solutions like CRDTs will help manage the state we still need to track.
Idempotence is everything, see FP for salvation.
Make state an Albion group, commutative monoid and this stuff comes largely for free.
Cloud Haskell duplicating Erlang model, but has tight coupling in the wrong places, needs same ABI (version) on all nodes; are we on our way back to a single data centre?
NoSQL vs RDBMS. Now settling down to small transactional world and larger non-transactional world.
Taking functional approaches (git’s similarity persistent data structures, log journaled databases, etc.)
]]>Database access library for SQL databases. Focusses on the database side of things (rather than domain model). Just a nice simple SQL database access layer. Supports lots of (SQL) databases and several Python runtimes.
While Django focusses on the database access needs for web applications, SQLAlchemy is aimed on the broad range of database applications. This makes SQLA harder to learn and far more flexible, exposing far more of the relation model.
Layered architecture:
Core provides and abstraction of Python’s DBAPI. Generates SQL in Python code (EDSL?). Bulk operations, complex queries, etc.
Schema handles DDL-type metadata.
Expressions handled DML-type operations. Dialects, pooling, etc.
ORM
Installing is easy:
$ pip install sqlalchemy
query = vehicals_table.select()
print query
> SELECT vehicals.name
> FROM vehicls
Query objects are mutable.
query.where(vehicals_table.c.odometer < 10000)
Yet another testing framework.
Framework for “aceptance testing and acceptance testing driven development”. Developed by Nokia Siemens Networks, contributions by Google? Pretty old.
Yet another attempt to make things easy for non programmers by deleting all the syntax (except the number of spaces between things, of course that’s meaningful; how could anyone not see that?)
Keywords (just sequences of keywords which are defined elsewhere); libraries of keywords.
Gherkin syntax for BDD (for people who think BDD and cucumber are related).
Generates reports in HTML format (plus XML). Includes screenshots when using the Selenium library.
Library to add annotations to screenshots for use in documentation, etc.
YAML, jinja and Python; all with significant white space.
Configuration management solutions should (in Daniel’s option) have: visibility, repeatability, iterability.
]]>Variance is essentially about substitution or sub-typing. A thing that produces
a Super
may be replaced with a thing that takes Sub
. A think that that
takes Sub
may be replaces by a thing that takes `Super.
Rat -> Real <: Int -> Complex
Reference to
Be liberal in what you accept and conservative in what you produce.
Use site variance in Java
final function Option<a>
In Scala:
List[+A]
Function[-T1, +R]
Kleisli[M[+_], -A, +B]
Eg:
class GPar
class Par extends GPar
class Child extends Par
class Box[+A]
def foo(x:Bor[Par]):
foo(Box[Child])
foo(Box[GPar]) // error
f
trait Box[+A] {
def get: A
def take(a: A) // Error
}
trait Box[-A] {
def get: A
def take(a: A)
}
Functor
trait Functor[F[_]] {
def map[A,B](a: F[A])(f: A => B): F[B]
}
This is actually a contra-variant functor.
trait Contravariant[F[_]] {
def contramap[A,B](r: R[A])(f: B=>A): F[B]
}
Variance in mutable values:
Two type variables. One goes up (things coming out?), one goes down (things going in?).
http://biosimilarity.blogspot.com.au/2011/05/of-monads-and-games.html
class Monoid a where
mempty :: a
mappend :: a -> a -> a
Obeys laws
mappend a mempty == a
mappend mempty a == a
mappend a (mappend b c) == mappend (mappend a b) c
Given types may have multiple monoids: numbers have (+, 0) and (*, 1).
Define monoids to do min, max, count on the pattern of Sum
and Product
.
Have a smart constructor for each
The foldable type-class makes fold polymorphic.
class Foldable t where
foldMap :: Monoid m => (a -> m) -> t a -> m
Use the monoids we defined earlier:
foldMap sum as
foldMap count as
foldMap max as
The mappend
operation distributes through structures like tuple types too.
This lets apply multiple monoids with one traversal.
a2 :: Applicative a => a b -> a c -> a (b, c)
a2 b c = (,) <$> b <*> c
> foldMap (a2 min max) as
(Min 1, Max 456)
This doesn’t let us do Mean very nicely. We need to unpack the pair at the end and calculate the actual
Have an additional Aggregation
class (superclass of Monoid
) which has an
associated type for the result which is extracted from the aggregation.
So we can make a version of foldMap which can finish and unwrap the aggregation:
afoldMap :: (Foldable t, Aggregation a) => (v -> a) -> t v -> AggResult a
afoldMap f vs = AggResult (foldMap f vs)
Combine the aggregation and monoid to apply a filter during the single traversal.
Two monoids for maps:
Replace the values
Insist the values are monoids too and append them.
Compose accessors with the “smart” constructors and apply them to calculate statistics over a stream of records.
Live at UNSW and looking at some optimisation problems. SpecConstr (constructor specialisation) is part of GHC (phase? optimisation?); has some problems, particularly with stream fusion, DPH, etc.
Dot product (pairwise multiple two vectors and sum the results) is a motivating example. We want to write that as:
dotp as bs = fold (+) 0 $ zipWith (*) as bs
But we want to actually execute:
dotp as bs = go 0 0
where
go i acc
| i > V.length as
= acc
| otherwise
= go (i+1) (acc + (as!i * bs!i))
(Assuming that the Ints aren’t boxed, etc.)
]]>David Lyon gave an introduction to Python on the Raspberry Pi. He mentioned wiringpi2 for general purpose I/O in the same vein as Wiring. I thought he spent gave too much credence to the suitability of general purpose uses of the Raspberry Pi when it’s the ability to hook it up to hardware that’s truly interesting.
Announcements: some startup wants a new co-founder; a new site based on Zope has launched: swimmingpoolregister.nsw.gov.au.
Eugene Van den Bulke gave a talk about using the pandas library for statistical analysis and, in the process, demonstrated the awesomeness of iPython Notebook (see the notebook at goo.gl/c84WQ). Particularly interesting were the I/O libraries for dealing with data in Google Analytics. Alas, he also used “data scientist” without scare quotes.
Someone else, whose name I missed, gave a brief introduction to R as a lightening talk so that we’d all have a basis for comparison. He used RStudio which looked pretty interesting. He also described RPy2, a Python library which provides an interface to R. After the break someone else (was it Jiří Baum?) mentioned gnumpy, which is like Numpy but ships stuff to the GPU for processing.
Tim Ansell spoke about playing with gstreamer pipelines in Python. He gave
a quick rundown on gstreamer pipelines, the gst-launch
command and the gst
Python module. I’ve been meaning to play with gstreamer since an LCA talk
a few years ago; perhaps it’s time?
The approach involves using an EDSL to write CUDA code in a Haskell program. This is generates code to be compiled and shipped to the GPU at run time. Being Haskell, there’s a bunch of nice types involved: matrices have shapes, matrix and scalar programs/fragments are of different types, and there are a few of typeclasses.
Trevor demonstrated a brute-force MD5 collision finder written with Accelerate which was very impressive.
Mark Hibberd spoke about interpreters and little languages with the Free
package. Essentially, this boiled down to using Free with its functor and
monad instances to replace an API full of IO a
typed operations, each of
them possibly unsafe, with a nice monadic EDSL and a single IO
interpreter
function.
The various operations in the API no longer “do” the operation but build up a data structure describing the operations to be performed as a little data structure (essentially an AST) which the client code passes to an evaluation function.
The more advanced theory categories and groups and left adjoint functors and such left me behind, but his running example of a password manager library (to generate, store and retrieve passwords) very illuminated the practical side of using this technique. I’m quite looking forward to Jed Wesley-Smith’s YOW! Lambda Jam talk about a similar topic: Connection Management, FP Style: A Case Study.
]]>Chris Harrop from Acquia introduced their devcloud platform and some of the other tools and services they offer.
Murray Woodman presented some performance measurements of node loading when using Entitycache and other techniques.
Justin Randall gave a pretty thorough introduction to using XHProf to detect and identify performance problems and regressions.
Chris’ talk about Acquia’s offerings was very short and quite high level. I was interested to see their “checklist-style” Drupal evaluation and testing tool as I’d was looking at a similar sort of run-the-tests-in-this-checklist tool yesterday.
Also: I got a code for a free dev environment on their cloud service.
The bulk of Murray’s talk was to present benchmarks of several approaches to
improving entity_load
performance in Drupal sites.
He compared:
Setting innodb_buffer_pool_size
appropriately to your machine and
workload.
Enabling the MySQL query cache.
Installing the Entitycache module.
Using APC to store cached objects.
Several combinations of the above.
The biggest and quickest win was installing entitycache (which resulted in an “80%” performance improvement), with the other techniques yeilding smaller improvements.
Justin began his presentation with a quote:
Don’t bring logic to a data fight.
Analysing system performance and identifying performance problems can be complex and real data on specific cases is more useful and more reliable than logic. “Don’t trust your gut, use science.”
After a bit of good advice about performance – it’s a process not a goal; test, and fix, client and network issues first – Justin dug into using XHProf to record and analyse performance profiles of Drupal pages.
The XHProf extension has so small an impact on performance that you can run it on production servers ready and waiting for a performance problem to profile (which degrades performance by around “<10%”).
Along with the XHProf extension are several tools to process and analyse the recorded profiles:
The moderately awful web interface which comes with the extension.
The XHProf Drupal module.
Mark Sonnabaum’s XHProf CLI tool to aggregate multiple profiles so that runs can be compared more reliably (when, e.g., checking for performance regressions in a patch).
A good tip, which hadn’t occured to me, was to use the auto_prepend_file
feature in PHP to enable XHProf. This is application agnostic, does not require
changes to the application code, and ensures that your profiles include the
whole application.
The idea of building XHProf into my production environments, with a prepend file configured and waiting to enable profiling (or automatically profiling, say, 1% of requests) is intriguing.
So this was a pretty good introduction to the Drupal community in Sydney, and it was nice to see how other groups run meetups (though I gather there have been some recent changes here).
]]>I’d just installed the Arduino, so I got that up on the projector for those who hadn’t seen it before and talked a little bit about the goals of the Arduino project and how that has affected the implementation. Talking about the libraries and the “shield” approach helped solidify it in my mind a little and it’s an interesting approach to this sort of development that seems new to some of the others (who’re familiar with real embedded development). I also explained a few things about Haskell and the approach (as I understand it; i.e. “poorly”) used in the atom library. I’m planning to have some more to say (show!) about using Haskell to implement hierarchical state-machines next fortnight.
Harry spoke about his numbatcam project (which he also talked about at Perth Barcamp 2009) in a bit more details and showed us some of the code. He mentioned the idea of reducing the duplication of states (one of the reasons we use hierarchical state-machines is to reduce the state explosion inherent in other state-machine formalisms) involved in his handling of setting the current time and the alarm time (each is an arrangement of a few states and transitions which are identical, other than their differing parent states).
I suggested that it would be possible, in a framework which supports orthogonal
state-machines (multiple machines which run concurrently and independently of
one another), to implement a “time chooser” state-machine and replace the
current duplication with waitingToSetTime
and waitingToSetAlarm
states
which have an entry action (posting a SET_TIME(curr)
event) and a single
transition on TIME_SET(new)
events. Given that the UI state-machine is can be
in either the waitingToSetTime
or the waitingToSetAlarm
states, then the
“time chooser” state-machine can handle the entire choose-a-time interaction
and then post the TIME_SET(new)
event to the UI state-machine and whichever
state it’s currently in can set the time or the alarm as it wishes. Alas, the
framework used in the numbatcam doesn’t support orthogonal state-machines
(easily).
Russell spoke about the doorbell system he’d been working on as a learning exercise. Rather than just play a sound every time the button is pressed, Russell’s doorbell incorporates a “polite pause” a light and a buzzer (to deal with those annoy people who feel the need to press the button repeatedly). Though just a learning exercise, it was certainly an interesting one. Even more interesting was his hierarchical state-machine-based hierarchical state-machine generator! Based on a description of a hierarchical state-machine description in emacs outline format (plus some support code), his project will generate the C code implementing the state-machine. Even more impressive, it’s implemented in exactly the same framework as the programs it generates. The only way he can make it more awesome would be to treat the current version as a bootstrap, and generate the next version from a state-machine description (to “eat his own dog food” so to speak).
In any case, there’s lots of interesting stuff going on in the group. I’m looking forward to the next meeting.
]]>