Conversations With Datomic - Part 3

Published on:

This is a continuation of the first and second conversations in which topics such as creating databases, learning facts, querying, and time traveling were discussed. Today’s topics include architecture, caching, and scaling.

Human: Hello again Datomic. Ready to talk again?

Datomic: Sure. I think you wanted to ask me some questions about how I would fit in with your other systems.

Human: Yes. Like I was saying earlier, I think your abilities to learn facts, reason about them, and keep track of the history of all those facts is really great. I am interested in having you work with me every day, but first I want to understand your components so that I can make sure you are a good fit for us.

Datomic: I would be happy to explain my architecture to you. Perhaps showing you this picture is the best way to start.

I am made of three main parts: my transactors, my peers, and my storage.

Human: What is a peer?

Datomic: A peer is an application that is using the peer library. In our last conversations, we were talking through the Clojure api with datomic.api. The application, or process, that is running this api is called a peer. There can be many of these, all having conversations with me.

Human: The peers then talk to your transactor?

Datomic Yes. The peers talk to my transactor whenever you call transact with the peer library. It is the central coordinator between all the peers and processes the requests using ACID transactions, and then sends the facts off to storage.

Human: Could you remind me what ACID stands for again? I always forget. The first one is Atomic right?

Datomic: That is right. I am Atomic in that every transaction you send to me is all or nothing. If for some reason, one part of it fails, I will reject the entire transaction and leave my database unchanged.

The C is for Consistency. This means that I provide every peer with a consistent view of facts over time and transactions. I provide a global ordering of transactions across all the peers with my transactor and peers will always see all the transactions up to their current time without any gaps.

Human: What if a peer is behind the global time? How do they catch up to know about the new facts that were transacted by a different peer?

Datomic: After one peer sends me a successful transaction with some new facts, I will notify all the peers about them.

Human: Cool. That takes care of the A and C in ACID. What about the I?

Datomic: It stands for Isolated. It makes sure that even through there are many peers having conversations with me, transactions are executed serially. This happens naturally with my transactor. Since there is only one transactor, transactions are always executed serially.

Human: In the picture, why are there are two transactors then?

Datomic: Oh, that is for High Availability. When I startup my system, I can launch two running transactors, but hold one in reserve. Just on the off chance something happens to the main one, I will swap in the failover one to keep things running smoothly.

The final D in ACID is for Durability. Once a transaction has been committed by my transactor, it is shipped off to storage for safe keeping.

Human: What exactly is this storage?

Datomic: Instead of storing datoms, I send segments, which are closely related datoms, to storage. I have quite a few options for storage:

  • Dev mode – which just runs within my transactor and writes to the local file system.
  • SQL database
  • DynamoDB
  • Cassandra
  • Riak
  • Couchbase
  • Infinispan memory cluster

Human: Which one is the best to use?

Datomic: The best one to use is the one that you are already have in place at work. This way, I can integrate seamlessly with your other systems.

Human: Oh, we didn’t really talk about caching. Can you explain how you do that?

Datomic: Certainly. It is even worth another picture.

Each peer has a its own working set of recent datoms along with a index to all the rest of the datoms in storage in memory. When the peer has a query for a datom, it first checks to see if it has it locally in its memory cache. If it can’t find it there, then it will ask for a segment of that datom from storage. Storage will return that datom along with other related datoms in that segment so that the peer can cache that in memory to make it faster for the next query.

Human: That seems really different from other databases, where the client constantly requests queries from a server.

Datomic: Yes. When most questions can be answered from the local memory, responses are really fast. You don’t need to hit storage unless you really need too. You can even add an extra layer of caching with memcached.

Human: That sounds great. I can’t wait tell you about all of our data. We talked a bit about your querying ability earlier, can you do the same queries that our other standard relational databases do, like joins?

Datomic: Oh yes. In fact, with me, you don’t need to specify your joins explicitly. I use Datalog, which is based on logic, so my joins are implicit. I will figure out exactly what I need to put together to answer your query without you having to spell it out for me.

Human: Ok. I know that I can map some of my data that is already in other database tables to you. What about other types of irregular data, like graphs, or sparse data.

Datomic: I am actually very proud of my data model. It is extremely flexible. Since I store things on such a granular datom level, you don’t need to map your logical data model to my physical model. I can handle rectangular table shaped data quite happily along with graph data, sparse data, or other non-rectangular data.

Human: That sounds great. What do I need to know about your scaling?

Datomic: I really excel at reads. All you have to do is elastically add another peer to me for querying. I am not really a good fit for write scale, like big data, or log file analysis. You will find me most happy with data that is valuable information of record and has history that is important, like transaction, medical, or inventory data. I am also really good at being flexible for development and operations since I can use many different types of storage. I have worked with many web and cloud apps.

Human: Thanks for answering all my questions. I think you might fit in quite well with our other systems.

Datomic: Great!

Human: One more thing, this conversation has been great, but do you have any training resources for me and my other human coworkers?

Datomic: Sure thing. There are a few really good resources on the Datomic Training Site. I would suggest watching the videos there and pairing them with:

Also, if you want to confirm that your data is good fit for me, I suggest you describe your data to the Datomic Google Group. They are nice and knowledgeable group of humans.

Human: Thanks again Datomic! I will grab another cookie and check it out!

Datomic: What is it with humans and cookies?…

Conversations With Datomic Part 2

Published on:

The following is a continuation of the first conversation which touched on schema creation and querying. This conversation includes learning new facts, time, and the sometimes unfortunate reality of lawyers.

Human: Hi Datomic. I am back from my tea and cookies break. I really enjoyed talking with you, could we continue our conversation?

Datomic: Certainly. Let me see, where did we leave off?

Human: Let me check my notes. Oh yes, we had created a database and filled it with facts about some dogs and owners. You showed me how to ask you things about the facts, like which dogs liked cheese.

Datomic: Ah yes. We were just getting to the really interesting part about how I learn new facts without forgetting the past ones.

Human: Can you please explain a bit more about how you learn new facts?

Datomic: Sure. I learn facts when you tell me to add a new fact about some entity, or to retract a fact about it. It will probably make more sense with an example. Let’s take the dog with the name “Tiny” that I know about. Could you please ask me about the dog and all of the attributes?

Human: Alright.

1
(d/pull (d/db conn) '[*] [:dog/name "Tiny"])

Datomic:

1
2
3
4
{:db/id 17592186045423
 :dog/name "Tiny"
 :dog/breed "Great Dane"
 :dog/favorite-treat "Cheese"}

There is an entity (dog) with the name “Tiny” who is a Great Dane and his favorite-treat is Cheese. I am not really sure why the Cheese is capitalized, but I am sure it makes sense to a human.

Human: Oh right. Well, now that you mention it, I am not really sure about the cheese thing either. Can I just remove that fact?

Datomic: Well, you can tell me to retract that fact by sending me a transaction in the form of [:db/retract entity-id attribute value]. In the case of the dog, since the name attribute is unique, you can just give me the selector for Tiny like this [:dog/name "Tiny"].

Human: Like this?

1
(d/transact conn [[:db/retract [:dog/name "Tiny"] :dog/favorite-treat "Cheese"]])

Datomic: Exactly. I have retracted that fact about “Tiny”. Go ahead and ask me about all of Tiny’s attributes again.

Human: Ok. I use (d/db conn) for the current database value right?

Datomic: Yes. But if you are going to be asking me multiple questions about this database value, you should not repeat the (d/db conn) all the time.

Human: Oh. What should I do instead?

Datomic: The connection with me is like a ref. Just like other refs, you should deref it once to get the database value and then use the value repeatedly. This single database value will provide consistency for all your queries.

Human: That makes sense.

1
2
3
(def db-tiny-no-cheese (d/db conn))

(d/pull db-tiny-no-cheese '[*] [:dog/name "Tiny"])

Datomic:

1
{:db/id 17592186045423, :dog/name "Tiny", :dog/breed "Great Dane"}

Tiny is a Great Dane.

Human: So you don’t know anything about the :dog/favorite-treat for “Tiny”?

Datomic: At this particular time, I do not have anything to assert about the favorite-treat of Tiny. However, I still remember everything about all the facts that you have told me. For each transaction that you send me, I have a notion of a point in time like t0, t1, t2. I have a complete database value for each one of those points in time. In fact, you can look at all of my assertions and retractions that I have learned about using the d/history function on the database value. This asks me to expose my history, which is normally hidden in favor of the present. I will return back a special database containing all the assertions and retractions across time. Any queries that you ask me will have a fifth datom field to help you distinguish the difference.

Human: A fifth datom field?

Datomic: A datom consists of the following parts: the entity, the attribute, the value, transaction, and an operation which tells you if the fact was added or retracted (e a v tx op). Why don’t you try using the d/history function to ask me about all the facts having to do with Tiny? I suggest using the datalog query

1
2
3
4
'[:find ?e ?a ?v ?tx ?op
  :in $
  :where [?e :dog/name "Tiny"]
         [?e ?a ?v ?tx ?op]]

which will return all the entity, attribute, value, transaction, and operation facts I ever knew about Tiny.

Human: Ok. Here goes.

1
2
3
4
5
(d/q '[:find ?e ?a ?v ?tx ?op
       :in $
       :where [?e :dog/name "Tiny"]
       [?e ?a ?v ?tx ?op]]
  (d/history db-tiny-no-cheese))

Datomic:

1
2
3
4
#{[17592186045423 63 "Tiny"       13194139534314 true]
  [17592186045423 64 "Great Dane" 13194139534314 true]
  [17592186045423 65 "Cheese"     13194139534314 true]
  [17592186045423 65 "Cheese"     13194139534320 false]}

During one transaction, you told me to add three facts about an entity:

  • The :dog/name attribute, (which I refer to as 63), has the value of “Tiny”.
  • The :dog/breed attribute, (which I refer to as 64), has the value of “Great Dane”.
  • The :dog/favorite-treat attribute, (which I refer to as 65), has the value of “Cheese”.

During another transaction, you told me to retract a fact regarding the attribute :dog/favorite-treat about the same entity.

Human: Wow, that is really cool. Is there a way that I can travel back in time to see the world as it was during that first transaction?

Datomic: Yes. I am practically a Tardis. You can use the d/as-of function with a database value and the transaction number and you can time travel. Using that time traveled database value, you can ask me about all the facts I knew as of that time.

Human: I can’t wait to try this. Ok, let’s go back to the time when I first asserted the fact that Tiny liked cheese.

1
(d/pull (d/as-of db-tiny-no-cheese 13194139534314) '[*] [:dog/name "Tiny"])

Datomic: Hold on. We are time traveling!

1
2
3
4
{:db/id 17592186045423
 :dog/name "Tiny"
 :dog/breed "Great Dane"
 :dog/favorite-treat "Cheese"}

Tiny is a Great Dane whose favorite treat is Cheese.

Human: Fantastic! Let’s go back to the future now, ummm I mean present. Time is a bit wibbly wobbly.

Datomic: Just take the as-of function off of the database value and you will be back in the present.

Human: Hmmm… Do I have to do a retract every time I want to change a value? For example, the dog named Fido has a favorite treat of a Bone right now, right?

1
(d/pull db-tiny-no-cheese '[*] [:dog/name "Fido"])

Datomic:

1
2
3
4
{:db/id 17592186045421
 :dog/name "Fido"
 :dog/breed "Mix"
 :dog/favorite-treat "Bone"}

Yes, it is a “Bone”.

Human: So, if I want to change it to be “Eggs”, do I need to retract the current value of “Bone” first and then add the fact of “Eggs”?

Datomic: You certainly could do that and I would understand you perfectly. However, if you simply assert a new value for an existing attribute, I will automatically add the retraction for you.

Human: Cool.

1
2
3
4
(d/transact conn [{:db/id [:dog/name "Fido"]
                   :dog/favorite-treat "Eggs"}])

(d/pull (d/db conn) '[*] [:dog/name "Fido"])

Datomic

1
2
3
4
{:db/id 17592186045421
 :dog/name "Fido"
 :dog/breed "Mix"
 :dog/favorite-treat "Eggs"}

Fido now has a favorite-treat of “Eggs”.

Human: This is really neat. You never forget any facts?

Datomic: Nope. Well, except in really exceptional circumstances that usually involve lawyers.

Human: Lawyers?

Datomic: Sigh. Yes, well in some unique situations, you might be under a legal obligation to really forget certain facts and totally remove them from the database. There is a special tool that you can use to excise the data. However, I will store a fact that something was deleted at that time. I just won’t be able to remember what.

Human: That doesn’t sound fun.

Datomic: I prefer to keep all my facts intact.

Human: I can definitely see that. Well, on a happier subject, I have been very impressed with you during our conversations. Having a time traveling database that can reason about facts seems like a really useful thing. Also, you are also really nice.

Datomic: Awww shucks, thanks. For a human, you are really nice too.

Human: I was thinking about the possibility of you coming and working with me every day. Would you mind chatting some more to me about your architecture? I want to understand how your would fit with our other systems.

Datomic: Certainly. I would love that. Do you want to talk about it now, or have another cookie break first?

Human: Now that you mention cookies… Let’s take a short break and we will talk again soon.

(P.S. Humans, there are some great Datomic Training Videos if you want to learn more)

Conversations With Datomic

Published on:

Human: Hi Datomic. I have been hearing good things about you. I would like to talk to you and get to know you. Is that alright?

Datomic: Sure! I would be happy to talk with you. What language would you like to converse in?

Human: I like Clojure.

Datomic: That is one of my favorites too. You know how to setup a Leiningen project right?

Human: Oh yes. What dependency should I use?

Datomic: Just use [com.datomic/datomic-free "0.9.5206"].

Human: Got it. Do you mind if I record our conversation in a namespaced file, so that I can refer back to it later?

Datomic: Not a problem. Make sure to require datomic.api when you set it up.

1
2
(ns conversations.datomic
  (require [datomic.api :as d]))

Human: All right. I am all setup up. I don’t really know where to start, so maybe you can tell me a little about yourself.

Datomic: I would be happy to. I am a database of facts. Would you like to create a database with me?

Human: Sure. How do I do that?

Datomic: For a casual conversation like this, we can use a in memory database with a uri like this:

1
(def uri "datomic:mem://first-conversation")

Then we can create the database and simply connect to it.

1
2
(d/create-database uri)
(def conn (d/connect uri))

Human: So, being a database, obviously you store things. What sort of things to you store?

Datomic: I store facts about things, which I call datoms.

Human: That sound neat. How do I tell you a fact to store? For example, what if I want you to store a fact about a dog, like its name?

Datomic: Ah. Well the name of a dog is an attribute. First, you need to tell me about the name attribute, so that I can use it to store the fact for you . You can describe the attribute in the form of a map like this:

1
2
3
4
5
6
7
{:db/id (d/tempid :db.part/db)
 :db/ident :dog/name
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/unique :db.unique/identity
 :db/doc "Name of the Dog"
 :db.install/_attribute :db.part/db}

This map is a set of facts, (called datoms), about an entity. In this case, the entity is an attribute. Attributes, in turn, can be used to describe other entities, like a dog. I will explain the different parts to you.

  • db/id is the internal id of the fact. With (d/tempid :db.part/db), I will generate it for you, so you don’t have to worry about it.
  • db/ident is the human readable reference for it. While I am fine just referring to the attribute by an id, humans prefer text. This says that you can refer to this attribute by the namespaced keyword :dog/name.
  • db/valueType tells me the type of the attribute. The dog’s name is a string.
  • db/cardinality lets me know if there is a one-to-one relationship with the entity or not. In our case, a dog has only one name.
  • db/unique is if that attribute is unique for an entity. In our example case, we are saying that a dog can be uniquely identified by its name.
  • db/doc is some documentation for humans that explains a bit more about the attribute.
  • db.install/_attribute tells me that this is an schema attribute that I should store with the other things like it.

Human: I think I understand. Let me try one out for myself. So dog breed would be this?

1
2
3
4
5
6
{:db/id (d/tempid :db.part/db)
 :db/ident :dog/breed
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/doc "Breed of the Dog"
 :db.install/_attribute :db.part/db}

Datomic: Yes! I think you got it. Let’s try one more.

Human: Ok. How about the dog’s favorite treat?

1
2
3
4
5
6
{:db/id (d/tempid :db.part/db)
 :db/ident :dog/favorite-treat
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/doc "Dog's Favorite Treat to Eat"
 :db.install/_attribute :db.part/db}

Datomic: You got it. Now, that you have these attributes, you can give them to me using a transaction with the connection.

Human: Ok. Do you want that in a specific format?

Datomic: Yes. Please send it to me using the form:

1
(d/transact conn [....facts....])

Human: Sounds good. I will put the dog schema datoms we discussed in a vector and call it dog-schema and then send it to you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(def dog-schema  [{:db/id (d/tempid :db.part/db)
                   :db/ident :dog/name
                   :db/valueType :db.type/string
                   :db/cardinality :db.cardinality/one
                   :db/unique :db.unique/identity
                   :db/doc "Name of the Dog"
                   :db.install/_attribute :db.part/db}
                  {:db/id (d/tempid :db.part/db)
                   :db/ident :dog/breed
                   :db/valueType :db.type/string
                   :db/cardinality :db.cardinality/one
                   :db/doc "Breed of the Dog"
                   :db.install/_attribute :db.part/db}
                  {:db/id (d/tempid :db.part/db)
                   :db/ident :dog/favorite-treat
                   :db/valueType :db.type/string
                   :db/cardinality :db.cardinality/one
                   :db/doc "Dog's Favorite Treat to Eat"
                   :db.install/_attribute :db.part/db}])

(d/transact conn dog-schema)

Datomic: The transaction was fine. I know all about those dog attributes now.

Human: I would like to also add a schema for owners for the dogs now. I think I know how to describe the name of the owner, but I don’t know how to express how the owner has dogs.

Datomic: Ah. I that case you can specify the db:valueType as a ref type. This lets me know that it references another entity.

Human: Do I need to tell you that is a type of dog somehow?

Datomic: No. I am pretty smart that way. I will figure it out once you try to tell me about some real entities. In fact, entities don’t have types. A ref attribute can reference any entity.

Human: Cool. Well, in that case, here is the owner schema. The owner has a name and some dogs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(def owner-schema [{:db/id (d/tempid :db.part/db)
                    :db/ident :owner/name
                    :db/valueType :db.type/string
                    :db/cardinality :db.cardinality/one
                    :db/unique :db.unique/identity
                    :db/doc "Name of the Owner"
                    :db.install/_attribute :db.part/db}
                   {:db/id (d/tempid :db.part/db)
                    :db/ident :owner/dogs
                    :db/valueType :db.type/ref
                    :db/cardinality :db.cardinality/many
                    :db/doc "Dogs of the Owner"
                    :db.install/_attribute :db.part/db}])

(d/transact conn owner-schema)

Datomic: The transaction is fine. I now know about the attributes that dogs and owners have. Would you like to tell me some facts about specific dogs and owners?

Human: Yes. Bob is an owner. He has two dogs. Fluffy is a poodle whose favorite treat is cheese, and Fido is a mixed breed, whose favorite treat is a bone. Lucy is also an owner who has one dog named Tiny. Tiny is a Great Dane whose favorite treat is cheese.

I am a bit confused how to represent the dogs of the owners. How do I do that?

Datomic: That is easy, just nest the datoms for dogs under the :owner/dogs attribute. You just need to create datoms for them. Each dog or owner will by its own map. Use :db/id set to (d/tempid :db.part/user) so I can generate it for you. Then use each attribute from the schema as the key and let me know the value.

Human: Like this?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(d/transact conn [{:db/id (d/tempid :db.part/user)
                   :owner/name "Bob"
                   :owner/dogs [{:db/id (d/tempid :db.part/user)
                                 :dog/name "Fluffy"
                                 :dog/breed "Poodle"
                                 :dog/favorite-treat "Cheese"}
                                {:db/id (d/tempid :db.part/user)
                                 :dog/name "Fido"
                                 :dog/breed "Mix"
                                 :dog/favorite-treat "Bone"}]}
                  {:db/id (d/tempid :db.part/user)
                   :owner/name "Lucy"
                   :owner/dogs [{:db/id (d/tempid :db.part/user)
                                 :dog/name "Tiny"
                                 :dog/breed "Great Dane"
                                 :dog/favorite-treat "Cheese"}]}])

Datomic: Exactly right. I now know the facts about Bob and Lucy and their dogs.

Human: Umm, how do I query you about the facts that your know? For example, how do I ask you about the dog named Tiny?

Datomic: There are a couple a ways to inquire about facts I know. To find out about the attributes of a specific dog, or entity, I would recommend using d/pull. You can ask me in the form of this to get all the attributes for a given dog name. Note that this works since the dog name is a way to uniquely identify the dog:

1
(d/pull (d/db conn) '[*] [:dog/name "Tiny"])

Human: What is the (d/db conn) all about?

Datomic: That function returns the current database value of the connection. The facts I know change during time. Every time there is a transaction, I consider the time to be different and there is a new database value. The d/db function gives you the most recent value that I know about.

Human: I am assuming the [*] is a wildcard that means give me all the attributes for that dog?

Datomic: Exactly right.

Human: Ok. Tell me about Tiny.

1
(d/pull (d/db conn) '[*] [:dog/name "Tiny"])

Datomic:

1
2
3
4
{:db/id 17592186045424
 :dog/name "Tiny"
 :dog/breed "Great Dane"
 :dog/favorite-treat "Cheese"}

Tiny is a Great Dane that has a favorite treat of Cheese.

Human: This is fun. What about more complicated questions. How do I ask about the name of the owner of the dog “Tiny”?

Datomic: For that I would ask using the datalog query d/q. It uses logic to unify your query with all of my facts and give you the result. The query itself would be a vector with logic statements inside like:

1
2
3
4
'[:find ?owner-name
  :where [?dog :dog/name "Tiny"]
         [?owner :owner/dogs ?dog]
         [?owner :owner/name ?owner-name]]

Human: Whoa. What is the deal with those question marks?

Datomic: The things with the question marks are considered as variables that we will unify to find the answer. For example, we are looking for something that we are going to call ?owner-name. I am going the use the following constraints with my facts to try to find the answer:

  • There is an entity that we are going to call ?dog that has the attribute :dog/name that is “Tiny”
  • There is an entity that we are going to call ?owner that has an attribute :owner/dogs that is the same as the ?dog entity
  • That same ?owner entity also has an attribute :owner/name that has the value ?owner-name

Human: Alright, so when I ask for this query, do I need to give you a database value too?

Datomic: Yes. They should have the form of:

1
(d/q '[datalog-query] db-value)

Remember, to get the current db value use (d/db conn).

Human: Ok. Here we go.

1
2
3
4
5
(d/q '[:find ?owner-name
       :where [?dog :dog/name "Tiny"]
              [?owner :owner/dogs ?dog]
              [?owner :owner/name ?owner-name]]
     (d/db conn))

Datomic:

The answer is:

1
#{["Lucy"]}

Human: What if I want to pass the dog name as a parameter? How do I communicate that to you?

Datomic: You will need to use an in clause in the query like this:

1
2
3
'[:find [blah]
   :in $ ?dog-name
   :where [blah]]

The $ is shorthand for the database value and the ?dog-name is what you will pass in as a parameter in the query after the db-value.

Human: Like this?

1
2
3
4
5
6
(d/q '[:find ?owner-name
       :in $ ?dog-name
       :where [?dog :dog/name ?dog-name]
              [?owner :owner/dogs ?dog]
              [?owner :owner/name ?owner-name]]
     (d/db conn) "Tiny")

Datomic: Exactly right. The answer is Lucy again.

1
#{["Lucy"]}

Human: I think I am getting the hang of this! Another quick question. How would I go about asking you which dogs have cheese as their favorite treat? I would want the dog’s name and breed back.

Datomic: You would simply construct another datalog query. This time I would recommend that you combine the pull syntax within the find part of the query. The pull syntax is great at selecting attributes from an entity. So the find part would look something like this:

1
[(pull ?dog [:dog/name :dog/breed]) ...]

This will return the attributes of the :dog/name and the :dog/breed. The three dots on the end will let me know that you want a collection returned, so I will give you back a simple vector with the entity attributes requested, instead of the set of vectors I normally give back.

The where section of the query is going to look for the ?dog entity that matches the :dog/favorite-treat attribute with “Cheese”.

1
2
'[:find [(pull ?dog [:dog/name :dog/breed]) ...]
  :where [?dog :dog/favorite-treat "Cheese"]]

Human: Then I put it together with the current database value in a d/q function?

1
2
3
(d/q '[:find [(pull ?dog [:dog/name :dog/breed]) ...]
       :where [?dog :dog/favorite-treat "Cheese"]]
     (d/db conn))

Datomic: Yup. The answer is:

1
2
[{:dog/name "Fluffy", :dog/breed "Poodle"}
 {:dog/name "Tiny", :dog/breed "Great Dane"}]

Human: Thanks so much. I think I beginning to get the hang of schemas and queries. What other things do I need to know about you?

Datomic: Well, you have just scratched the surface really. One of the most interesting things about me is that I never forget facts. You can add new facts, like Tiny’s favorite food is now hotdogs, but I won’t forget that he liked cheese at another point in time.

Human: That sounds really interesting. I think I need some tea and cookies before I delve into that. Let’s take a short break and talk again soon.

Datomic: My pleasure. I look forward to it.

Special thanks to Paul deGrandis for the conversation idea :)

(P.S. Humans, there are some great Datomic Training Videos if you want to learn more)

Wild Horses, Things, and Creativity

Published on:
Tags: All

Sometimes I want to create something and I run into a limitation or constraint. It is at this point where I am tempted to give up and say that I just can’t do that.

At this time, I remember Maurice Sendak and the origins of Where the Wild Things Are. In this interview with Bill Moyers, he explains how the book was originally titled Where the Wild Horses Are, but he couldn’t draw horses.

She was this torrential woman, passionate woman, who could spot talent 10 miles away. I had no education. I did not go to art school. My drawing was so crude. I had shines on shoes like in Mutt ‘n’ Jeff in Walt Disney. And she saw through that monstrous crudity and cultivated me, really made me grow up. And then, it was time to do my own picture book.

And I came to her with a title that was “Where the Wild Horses Are.” And she just loved that. It was so poetic and evocative. And she gave me a contract based on “Where the Wild Horses Are.” And then, it turned out after some very few months to her chagrin and anger, I couldn’t draw horses. The whole book would have to be full of horses to make the book make sense.

And when I tried a number of things, I remember the acid tones. She said, “Maurice, what can you draw?” Okay. Cause she was investing in a full color picture book. That was an enormous thing back then.

And so, I thought well things, things.

Limits or constraints cannot hold back creativity. But it can help mold it in surprising directions.

How Not to Panic While Writing a Clojure Book

Published on:
Tags: All, Clojure

I made it to that magical moment when the Clojure book I had been working on so long was published and I could actually hold it in my hand.

It was an immense project and I am very happy that it is finally done. Since then, I met some people that are interested in writing books as well. They asked if I had any insights or tips having gone through the process as a first time author. I have collected them in this post in hopes that they will be helpful to those going through the process themselves.

The very first thing to do is to get an outline for your book.

Start with an Outline

Ideas are soft and squishy. They drift into different shapes like clouds, and can melt away just as quickly. One of the hardest things to do was trying to arrange all those ideas in my head into an initial form that would serve as the structure for the entire book. I would pick up a pen and paper, start feeling overwhelmed, and suddenly remember I had something else very pressing to do. I successfully avoided starting a book for quite a while, until one day I cornered myself. I decided that I write my book outline on a long plane flight. With salted peanuts as fuel, and nowhere to escape, I finally wrote an outline. It wasn’t perfect but it was a start and looking back and it was not too far off. Here it is in all of its original roughness.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Title: Living Clojure

From beginning steps to thriving in a functional world


(Each Chapter will follow quotes from Alice in Wonderland and very use ideas from some examples)


Book 1 - Beginner steps

Chapter 1 - Are you comfortable?  Talk about how OO is comfortable but there is another world out there and new way of thinking functionally.
            White Rabbit goes by
Chapter 2 - Forms & Functions - Falling down the rabbit hole
Chapter 3 - Functional Transformations - Growing bigger and smaller - Key to thinking functionally is about transforming data from one shape to another shape.
            Map & Reduce
Chapter 4 - Embracing side effects  - Clojure is impure functional language (The rabbit's glove)  - Cover do and io stuff. Also basic stuff about 
            STM atoms and agents/ Protocols
Chapter 5 - Libraries, Libraries  - - how to use Leiningen
            build system. Where to find clojure libraries, how to use
            Serpents - camel-snake-kebab
Chapter 6 - core.asyc - Tea Party introduction to the core.async library
Chapter 7 - Clojure web - Chesire cat  - introduction to Ring, Cheshire library, ClojureScript and OM 

Book 2 - From here to there - thriving in a functional world

Training plan for thriving in a functional world.

Chapter 8 - Join the community - Surround yourself with other Clojure enthusiats
  - Twitter clojure
  - Github account
  - Clojure mailing list
  - Prismatic clojure news
  - Meetup for local community group.  If there is not one in your area. start one!
  - Attend a Clojure conj


Chatpter 9 - Practice and build up 
Like Couch to 5K 7 week training program to work up to
practicing Clojure

Now that I had an outline. I just needed to figure out how long it would take me to write the book.

However Long You Think It Will Take – You Are Wrong

Having never written a book before, I had no idea how much work it would be. The closest thing I had to compare it to was writing a blog post. I figured writing a chapter would be roughly equivalent to writing a blog post. If I could go back in time, this is the moment where my future self would pour a glass of ice water on my past self. Writing a book is nothing like that. It is a lot of time and work. If I had to compare it now to writing blog posts, the process would be this.

- You write a blog post.
- You rewrite the blog post.
- You write a second blog post.
- You rewrite that blog post and the first one too.
- You write another blog post.
- You rewrite all three post .....

So, if you have to commit to deadlines, make sure you remember how hard it will be, and then double the number.

Speaking of deadlines, they suck, but you should have them.

Make Deadlines

Deadlines are not fun. In fact, deadlines might even be a source of potential panic. But for me, they were necessary evil. There were a few beautiful times when inspiration came knocking at my door and I couldn’t wait to start writing. But most of the time, inspiration was off somewhere else eating biscuits. The only thing that actually moved the book along was me knowing that I needed to get the next chunk done by a certain date.

I found the best thing to do was to set aside a small bit of time on a daily basis to write something.

Routine, Routine, Routine

A daily routine was the most crucial thing for me. Life is really busy with work and family. It is so easy to get overwhelmed with daily life. I decided that mornings would work best for me. So I would stumble in to my computer an hour before work, with a hot cup of tea in hand, and write something. Some days I actually did quite a bit. Other days, I would write one sentence and declare it done. But, I would always do something. Even though those small slices of time didn’t seem like a lot, they added up over the course of a week.

Another curious thing happens when you do something, even a little bit, day after day. You start to get better at it.

Writing is a Different Skill from Coding

I was used to writing code all day. I found that the code writing skills are not the same as writing about code. In fact, I found it really hard to do at the start. But, just like writing code, you get better with practice. And to get better at anything, feedback is really important.

Get and Trust Feedback

After each chapter, I would get feedback from my editor. She was awesome and provided ways for me to improve the experience for the reader. I lost track of how many times I rewrote that first chapter, but each time it would get a bit better and I would improve as well. After the book was about half done it was sent out to others for technical review. They provided feedback not only on the writing style but also the technical content, making sure that it all made sense.

The feedback loop is much slower for writing a book than writing code, but it is just as vital. The great people providing feedback are you closest partners in this. You need to trust them. Especially during the roughest time, the middle of the book.

The Middle Bit is the Hardest

I found the hardest time was about halfway through the book. The initial excitement of the new endeavor had long since worn off. It seemed like such a mountain of a task, with no end in sight. I questioned my decision to continue with it daily. My routine and deadlines kept me moving forward. But my circle of friends and family kept me sane. It was great to have an outlet, not only to vent my frustration with my slow progress, but to get kind encouragement to keep my spirits up.

During these dark days, I also ate cheese.

Celebrate Your Small Victories

At the end of every chapter or deadline I would fix myself a nice plate of cheese and crackers. You have to celebrate the small wins. Cheese is also very tasty.

When the book was finally done. I had a really tasty plate, complete with Stilton, Brie, and a dependable Cheddar. I was incredibly happy to be finished. But I knew that I definitely could have not done it without the help of others.

Thank Everyone that Supported You

Writing a book is a huge undertaking that is utterly impossible to do alone. I could have not done it without the help and support of my editor, reviewers, family, friends, as well as the entire Clojure Community. I am so thankful to all of you that helped my in this project. You are great.

So, should you go ahead and write that book?

Do It

Yes, you should write that book and share your knowledge. Don’t panic, remember to breathe, get some cheese and tea, and go for it! It will be awesome.

Partition With Game of Thrones Pugs

Published on:
Tags: All, Clojure

Clojure’s partition and partition-all functions are very useful. However, I have been bitten a few times using partition when I really wanted partition-all. So to help myself and all of you to remember it, I have made some diagrams with pugs from the Game of Thrones

In code, partition takes a collection and returns a lazy sequence of lists, each containing n items.

To demonstrate this with pugs, we will partition 5 pugs into groups of twos.

This partition will give you two groups of two pugs.

Notice, (and here is the important part), the last pug is missing. The Joffrey pug is not included because partition will not include items that do not make a complete partition. In this case, because there is no group of 2 pugs for the Joffrey pug to be in, it gets dropped.

This is the thing that has bitten me in the past.

A common use for wanting to partition things is to control the number of things that you process at one time. An example of this is sending only 500 items to be processed in a batch job at one time. If you have a few thousand items to be processed, partitioning them is a good way of chuncking. However, if you have an arbitrary number of items, you most certainly want to process them all and not drop any. This is where you should use partition-all instead.

Partition-all chunks the items as well, but also includes any leftovers. Demonstrating again with pugs.

This partition-all will give you three groups of pugs.

This time pug Joffrey is not left out!

Remember, think carefully before using partition. Don’t leave a pug out.

By the way, I can’t wait till the next season of Game of Thrones. Until then ..

Gigasquid’s Radar 2014

Published on:
Tags: All

It’s that time of year for radars to be published. So this year, I thought I would publish one of my own. Here is what is on my radar.

Languages

  • Adopt: Clojure – It is fantastic language. Really.
  • Trial: Pixie – The promise of a really fast startup Clojure inspired language. I am impressed already and it is only a few months old.
  • Assess: Idris – I have only seen this lang briefly, but was impressed by the typing and proofing abilities.
  • Hold: JavaScript – I don’t have to say more, you know what I mean.

Cute Animals

Robots

  • Adopt: Parrot AR Drone Hackable flying drone with sonar and cameras. Doesn’t injure your fingers too much when you get them caught in the blades.
  • Trial: PhantomX Hexapod A bit pricey and delicate, but built from kit and is super cool when you get it going.
  • Assess: Myo Armband Control things with a flick of your wrist. Mine finally arrived after a year and a half wait. I haven’t had time to play with it, but I have high hopes for it.
  • Hold: Roombas – I love my Roombas, but I am not sure the latest models have an ROI port to hack :(

Tasty Food

  • Adopt: Crumpets – Put lots of butter on them toasted. Yum!
  • Trial: Mint Tim Tams– Minty, just the right amount of crunch. Hard to find in the US, but a treat when you do.
  • Assess: Raclette – I have never actually had it, but it is melted cheese, it has to be incredible.
  • Hold: Egg Nog – Don’t drink it directly from the bowl.

Happy Holidays Everyone!

Clojure FizzBuzz Without Conditionals

Published on:
Tags: All, Clojure

Sure you may have done FizzBuzz before. Maybe you have even done it in Clojure. But have you done it without the use of any conditionals?

As your brain starts to work on the how this we be done, you might be wondering why you should do this in the first place?

There are two very good reasons for this. The first is that it is a kata.

Katas build your code practice

Code katas build your skill through practice. It doesn’t matter if you are a beginner or an expert. Just, like in all those martial arts movies with the swordsman practicing, so must we. We stretch and flex our coding muscles with katas to grow them and keep them in shape.

Yes, you may code every day at work. But it is not the same as kata practice. So much of day to day work involves complexity with large interconnected concerns. Our kata practice cuts the extra complexity out and leaves you alone with a focused small problem.

The second reason involves why you should try it, this time, without conditionals. The answer is creativity.

Constraints build creativity.

It turns out that constraints are a key way to drive creativity. Programming does not only require technical skills, but also creativity. We are seldom asked to build software without constraints. It drives design. Sure, it can be annoying when we have to communicate with a server that is only active on Tuesday and emits its response in Morse Code. But it gives us boundaries to unleash our creative spirit.

So go for it.

Give it a try

Ready? Here are the details.

  • Given a number, if it number is divisible by 3 return “fizz”.
  • If it is divisible by 5 return “buzz”.
  • If it is divisible by 3 and 5 return “fizzbuzz”.
  • Otherwise, just return the number.
  • Don’t use any conditionals like if else case cond.

When you are done, you can check out some of the other solutions. Try not to peek until you have done your version first though.

(There are some really awesome ones so far).

Feel free to link to yours in the comments too :)

Solutions

From @aderth

From @IamDrowsy

From @Bryanwoods

From @defndaines

From me

From @hyPiRion – a couple of notes for this one is that:

1
2
(+)
;; -> 0

and

1
2
(*)
;; -> 1

And once you think about that, you might want to read this :)

Happy Clojure Kataing!

The Five Stages of Writing a Book

Published on:
Tags: All

  1. Denial: I am not really writing a book.
  2. Anger: Why did I ever decide to write a book?
  3. Bargaining: If I just finish this book, I promise never to write another one.
  4. Depression: I am never going to finish this book.
  5. Resolution: I am writing a book and I am going to give it my frigging all.

Notes and Tips on Working From Home

Published on:
Tags: All

Recently, I switched from a traditional, “go to an office” job, to working from my home. It took some time to setup my home work space and get used to working remotely, but I finally have a system working for me. In this post, I thought I would share some things that I found useful.

Window Seat Please

If at all possible, locate your home work space near a window. The natural light does wonders for you mood and being able to glance up and look at trees and real life is a refreshing break from staring at code all day.

A Door is Great

Having a door to your workspace is a real advantage. It enables you to close off noise and other activity that might be going on in the house. It is also incredibly useful if you have small children. For my kids, if the door is closed, then it is a sign that Mommy is working – do not disturb.

Invest in a good chair.

Backs are very important. I started working from home with just my kitchen chair – big mistake. After a day or two, my back was crying out for something better. I did some research on good chairs to get and I ended up with a Steelcase Leap Chair. They are not cheap. But, I was able to get a refurb one that was considerably less than new, and my back loves it.

Don’t Sit All the Time

Even with my great chair, it is not the best to sit constantly. I had tried a straight standing desk a while back and I found that standing all the time was not the best for me. I prefer to mix it up. I got a adjustable Geek Desk. I generally stand up in the morning and sit in the afternoons.

Freedom from a Headset with a Good Mic

I have used a headset before when doing remote calls. They work fine, but after a while, I found it annoying to have on my head all day. I switched to a Blue Snowball Mic at home and am really happy with it. My voice comes in clear and I am headset free.

Dual Monitors for the Win

I use two Thunderbolt displays. One monitor I use for communications, it has my email, chat, and video on it. The other monitor I use for the codez. It works out pretty well to switch back and forth.

Good Software for Communication

Good communication is a must for working remotely. Someone cannot just wander over to your desk and ask you a question. Here is a list of communication tools I use at work:

  • Slack – for team communication.
  • Google Docs
  • Zoom – for video and screen sharing. It is way better than Google hangouts in terms of video quality.
  • Apple’s Screen sharing – for pair code development. This let’s people use whatever editor they are comfortable with, yet you can see the code and still share control.

Pair Programming is Awesome

At work, we do pair programming most of the time. I really like to work this way. One of the things that I was concerned about in switching to remote work was being lonely. Pair programming really helps in this. I am usually working with someone during the day, with one monitor going with video and voice, while the other monitor has the code we are working on. For me, it has all the advantages of idea sharing and group problem solving. I realize that working this way is not for everyone, but I am digging it.

Routine is Everything

When working for home, I have found it is crucial to have a good routine. Since we do a lot of pair programming at work, we all generally keep the same hours. Being a distributed team over North America, this means I start work at around 10am EST. I have found that having a routine a sticking to it helps structure my day. I get up, get dressed, eat breakfast, just like I was going to work. Then, I usually hack for a bit on personal stuff in the morning until it is time for work. Then at lunch, I go for a run or work out. Finally, and most importantly, in the evening, I leave the computer behind and devote time to family.

Don’t Forget to Visit with Other Humans

The downside of working from home is that it is very easy to not leave home. At one point, I realized that I had not left the house for a week. Not good. I try go to a social event where I will meet with other developers and friends every week. There is a nice developer coffee group that meets on Fridays. I also help run our Cincinnati Functional Programmer’s Group here in town. In general, I find that if I am driving somewhere and see people walking on the street and start thinking, “Look Humans!”, it is time to get out and socialize a bit more. Working remotely, makes going to conferences and being with other developers in person even more fun.

Summary (with a Dog Pic)

I have found working remotely to be quite enjoyable so far. It does take an extra effort to keep your life structured and communication flowing properly, but it is worth it.

My next challenge, since it is getting colder, is to get my dog to sleep on my feet while I work. No luck so far.

If anyone has any tips, let me know.