Richard Searle's Blog

Thoughts about software

Moved to github

Posted by eggsearle on January 26, 2014

Moved to github 


Posted in Uncategorized | Leave a Comment »

Working around Scala parser memory leak.

Posted by eggsearle on January 26, 2014

This ticket covers an ongoing issue with the parser combinators. The last fix replaced the thread safety issue with a memory leak.That was unfortunate since it fairly easy to deal with the thread safety limitation. Resolving the memory leak requires more heroic efforts, illustrated by this code.

This issue has forced me to remain on Scala 2.9.x, which is becoming less tenable as time passes. The ticket implies that 2.11 will contain a fix, but there is no evidence of any change in the current code.

Reworking the code to use the phrase combinator side-steps the memory leak and allows an upgrade to Scala 2.10.x

def rawParse(reader: Reader[Char]) = parse(rootParser, reader)


def rawParse(reader: Reader[Char]) = {
 parse(phrase(capture(rootParser) ~ dropDead), reader) match {
 case Success(wrapped, _) =>
 wrapped match {
 case result ~ in => result match {
 case ns : NoSuccess => ns
 case _ => Success(result, in)
 case ns: NoSuccess => ns //should not be seen

private val dropDead = Parser { in => Success(in, new CharSequenceReader("")) }

private def capture[T](p: => Parser[T]) = Parser { in =>
 p(in) match {
 case ns : NoSuccess => Success(ns,in)
 case _ @ s => s

Posted in Uncategorized | Leave a Comment »

annotate Datomic transaction to record side effect

Posted by eggsearle on October 13, 2013

Extends previous post to indicate how the transaction can be annotated to indicate the email was generated.

In the general form, this can capture any metadata required to describe process(es) that have been executed, decisions made, etc.



Posted in datomic, datomisca, Scala | Leave a Comment »

Datomic txReport and tracking changes via Datomisca queries

Posted by eggsearle on October 13, 2013

A common web site use case is notifying a user of a change of address, with notifications to both old and new addresses.
This is a simple mechanism that helps to address fraud.

Generally this code would form part of the http post action, perhaps including some indirection via a queue.
Arguably that logic is orthogonal to the primary function of the web site.

Datomic records all changes to entities and provides a notification mechanism

The following gist illustrates how this might be implemented.

Line 39 creates a user with the name “john” and email “”.
Line 44 updates that user, changing the email to “”

Line 46 defines a query that will retrieve the name and previous email, given a delta to the user entity

Line 59 will print empty result since qe is the change that created the user (and there is thus no previous value)
Line 62 creates the result:(“john”, “”,””)
Which provides the name, before and after emails needed to send the warning email.

Posted in datomic, datomisca, Scala | Leave a Comment »

Unexpected attribute handling when querying datomic txReport

Posted by eggsearle on October 12, 2013

Perform queries with results indicated 
Peer.q(“[:find ?value :in $ :where [_ :db/doc ?value]]”,tx.get(Connection.TX_DATA)); //returns []
Peer.q(“[:find ?value  :in $ :where [_ :db/doc ?value]]”,db); //returns values
Peer.q(“[:find ?value  :in $ :where [_ 61 ?value]]”,tx.get(Connection.TX_DATA)); //returns values
Peer.q(“[:find ?value  :in $ :where [_ 61 ?value]]”,db); //returns values
So :db/doc is mapped to 61 for a query against the database but not when referencing the txReport.
This initially looks rather strange but is a direct consequence of how Datomic is implemented. 
:db/doc is merely an entity stored in the database, just like any other entity. It is not a special constant, baked into the implementation.
The txReport value does not contain the entity and thus cannot perform the mapping to attribute id. 
Set up the test
uri = “datomic:mem://hello”;
datom = Util.list(“db/add”,
                  “hello world”);     
queue = conn.txReportQueue();
db = conn.db();

Posted in Uncategorized | Leave a Comment »

datomic history

Posted by eggsearle on October 7, 2013

The history API returns a Database that contains asserts and retractions.

This seems to duplicate the Log API, so further study is required.

Posted in Uncategorized | Leave a Comment »

Query datoms via transaction id fails due to full scan error

Posted by eggsearle on October 6, 2013

The recent versions of datomic provide a  log interface that provides access to the transactions that have been processed.
That interface is unfortunately not implemented for memory databases, which complicates testing.

The datoms stored by Datomic are quads, which include the identity of the transaction.
It is thus possible to locate datoms asserted by a specific transaction

[:find ?e ?name  :in $ ?t  :where [ ?e :person/name ?name ?t]]

will locate all datoms that asserted a person’s name for a transaction id:t

One might then expect this would retrieve all the datoms asserted for a transaction

[:find ?e ?a ?v  :in $ ?t  :where [ ?e ?a ?v ?t]]

That query fails with “Insufficient binding of db clause: [?e ?a ?v ?t] would cause full scan”.

Which would make sense if datomic does not provide an index over transaction id.
However, it is not clear how the log interface can provide reasonable performance w/o that index.




Posted in Uncategorized | Leave a Comment »

Some concerns with akka.js

Posted by eggsearle on August 17, 2013

The akka.js  project provides a direct linkage from angular.js to Akka and was referenced by Jonas Boner.

However, there are some concerns with architecture and the implementation. The latter would be irrelevant if this project is merely a POC.

  1. The WebSocketClientStore object contains a mutable Map shared between two actors.
  2. ActorRefs are added to the Map, but never removed.
  3. The akka module contains a replyTo  map, to which a promise is added, but never removed.
  4. The actor protocol is defined in terms of JsonObject, which are essentially untyped.
  5. The Akka actors are directly exposed to untrustworthy client code, without an mechanism for authentication, authorization, etc.


Posted in Uncategorized | Leave a Comment »

Type providing from SQL provides excellent justification for Scala macros

Posted by eggsearle on August 17, 2013

sqltyped processes SQL statements to extract the DDL details and then uses macros to generate type safe code.

This provides a DRY implementation where the database schema is the master and errors in the referencing code are detected at compile time.


Posted in Scala | Leave a Comment »

Play as an ESB

Posted by eggsearle on July 31, 2013

Back in 2007 Mark Nottingham suggested squid could serve as an ESB , which was probably considered a rather radical idea.  Certainly I never came across any further references to the concept.  This recent Play blog post  triggered my recollection of that half decade old idea.

It has many functions that one might expect from an ESB:

  1. Caching
  2. Rate limiting (via timeouts and return codes)
  3. Proxying (via REST call to external service)
  4. Data conversion (http response to json)

The sample is compact, easily understand and easily extended via composition. (characteristics that are rarely encountered using the big name ESBs)





Posted in play, Scala | Leave a Comment »