qryq
Stop RESTing, start using query queues
/ˈkwərik/Brendan Graetz
In one sentence
qryq is a NodeJs library that allows one to express a series of queries and define dependencies between them either in parallel, in sequence, or in a directed acyclic graph.
A directed graph with no directed cycles.
That is, it is formed by a collection of vertices and directed edges, each edge connecting one vertex to another, such that there is no way to start at some vertex v and follow a sequence of edges that eventually loops back to v again
Implementation
Inspiration
- Neil Jenkin's talk Tips, Tricks and Hacks in the Pursuit of Speed
POST /api/
[
[ 'deleteMessages', {
idList: [ 'msg1' ]
}],
[ 'getMailboxMessageList', {
mailboxName: 'Inbox',
position: 0,
limit: 30,
sort: 'date descending'
}]
]
Inspiration
- Play framework's
- Linkedin talk by Yevgeniy Brikman
- See slides 85 through 88
The Itch
- NodeJs callback spaghetti
- Fix this using promises
- While better, if the code is sufficiently complex, you can still end up with:
- Promise spaghetti
Light Bulb
The Query Queue
[
{id: "q1", api: "add", qry:{a:1, b:9}},
{id: "q2", api: "add", qry:{a:99, b:1}},
{id: "q3", api: "multiply", qry:{a: "#{q1}", b: "#{q2}"}},
{id: "q4", api: "multiply", qry:{a: "#{q3}", b:5}}
]
`q2` --> add(99, 1) --> 100
`q1` --> add(1, 9) --> 10
`q3` --> multiply(`q1`, `q2`) --> multiply(10, 100) --> 1000
`q4` --> multiply(`q3`, 5) --> multiply(1000, 5) --> 5000
q1andq2may execute in any order,q3may only execute after bothq1andq2,- and
q4executes last. - Wiring is done automatically by
qryq
What about async?
- Dev productivity in accomplishing the same thing:
- Sequential: ~Same
- Parallel: ~Same
- Dependent: A lot easier
Code
Let's look at some!
What is it like to write the same thing using async vs. using qryq?
(also callbacks, and promises)
Key comparisons
- assembly && execution
- declarative || imperative
- developer productivity
- what it is and what it isn't
Benefits
- Developer productivity
- Bandwidth && latency savings
Benefits - Dev Productivity
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together.
-- Doug McIlroy, invented pipes in UNIX
Benefits - Dev Productivity
- Less need to write dedicated API endpoints
- instead write small API endpoints, and chain them together
- readable && composable
- declarative query rather than imperative
- avoid callback spaghetti && promise spaghetti
Benefits - 'Net Traffic
- Concatenation of
- Multiple requests
- Multiple responses
- Protocol overhead minimised
Benefits - Dev Productivity
- Less duplication of biz logic required
- Client/ server
- Groups several queries together as an atomic unit
Limitations
- non-RESTful
- Testability?
- Expression engine
Limitations - REST
- Do you need REST when you are not doing CRUD?
- What if you typically chain more than one CRUD operation together?
Limitations - Testing
- Harder because clients may compose API calls in novel ways
- Need to write more robust code
Limitations - Expressions
`#{previousQry}.flights.length`
- A
qryreferences results of anotherqryin the sameqryq - Limited: Can only "drill down" through properties
Where to use: Client side
- Instead of making multiple XHR requests to each RESTful API endpoints
- and assemble them together by hand
- Invoke qryq, and it will assemble the multiple requests together itself
- still makes multiple XHR requests
- Great if goal is not so much bandwidth savings
- instead dev productivity
- (or if you're not making any XHR requests)
Where to use: Server side
- Instead of making multiple XHR requests to each RESTful API endpoints
- and assemble them together by hand on the client
- Invoke qryq API endpoint, and it assembles and executes the multiple requests
- Single XHR request-response pair
- Dev productivity + Bandwidth savings
- Only if your server is in JS ∗
How do I use qryq?
- Instructions in the main README.md
Horizon
- Rewrite the
Qspaghetti in walkre- Demonstrate how declaratively defining dependent queries can make code more comprehensible
- Feature to reference results of dependent queries inline in query data
- Kinda like this
- Separate qryq into its own library - out of walkre
- Unit tests -
nodeunit - Pick a licence for this library - GPL v3
- Infer
dependsif not provided using two passes when parsing eachqry - Comparison with async, use diamond shaped query graph
Farther Horizon
- Cyclic graph detection in dependent query queue validation
- Allow client to create promises and pass in to
qryq - Benchmarking for performance
- Load testing/ stress testing
- Start including high latency ops, e.g. disk I/O
- Create a NodeJs/ ExpressJs server wrapper for
qryq- For full stack load testing/ stress testing
- Minify
- Restructure to follow CommonJS directory structure
- Publish as a bower project
- Write test cases for series and paralllel
Decisions
- Should the server return all reponses?
- Perhaps allow specifying which queries we are interested in the return value of.
- Port to other languages?
- Which ones?
- Should remove sequential and parallel altogether?
- Just focus on dependent (DAG) instead?
Fin
- Recommendations for load testing a NodeJs server?
- What other libraries are there out that that perform this function? In other languages?
- Submit some patches!
Thank you
qryq
Stop RESTing, start using query queues
/ˈkwərik/
Brendan Graetz
@bguiz
bguiz.com