Sean Glover

How Alpakka Uses Flow Control Optimizations In Apache Kafka 2.4

Sun, 26 Jan 2020 00:00:00 +0000

Original post on Lightbend’s blog, Archived PDF

This blog post begins with details about how the Kafka Consumer and Alpakka Kafka internals are designed to facilitate asynchronous polling and back-pressure. We’ll discuss how the Consumer’s flow control mechanisms were improved in KAFKA-7548 to provide better performance. We conclude with some benchmarks that demonstrate the performance improvements before and after the new Consumer is used.

Monitor Kafka Consumer Group Latency with Kafka Lag Exporter

Tue, 07 May 2019 00:00:00 +0000

Original post on Lightbend’s blog, Archived PDF

A blog post introducing Kafka Lag Exporter, a tool to make it easy to view consumer group metrics using Kubernetes, Prometheus, and Grafana. Kafka Lag Exporter can run anywhere, but it provides features to run easily on Kubernetes clusters against Strimzi Kafka clusters using the Prometheus and Grafana monitoring stack.

Exactly Once Streams - Exactly Once Streaming in Kafka

Sun, 16 Jul 2017 00:00:00 +0000

Link to report on GitHub

An engineering report and a set of PoC’s that demonstrate “Exactly Once” message delivery semantics in Apache Kafka using Kafka base clients, Kafka Streams, and Alpakka Kafka.

Technology and Politics - David Graham

Thu, 22 Oct 2015 09:00:52 +0000

This week I was proud to learn that David de Burge Graham was elected as a Member of Parliament of Quebec Canada’s Laurentides–Labelle riding. As far as I know, David is one of our country’s first real technology enthusiasts in the house of commons.

CBC 2015 Interactive Election Results

I met David online nearly 8 years ago. Like many of my peers in technology, I grew up in the 90’s during the Personal Computing revolution. I’ve met thousands of people online through a text-only chat technology known as Internet Relay Chat (IRC). Some of those people I went on to meet in person, but for the most part my digital and “meatspace” life have remained separate.

In 2007 I became a Network Representative for the Open and Free Technology Community (OFTC). I made a number of new online friends of the OFTC staff. At the time David was recently elected as Ombudsman of the network and when he found out I was a fellow Canadian we began getting to know one another.

When I joined OFTC’s staff I learned a little about David’s history. I’ll leave it for someone else who was closer to David during this time to tell the story in detail, but I can summarize. He was a core member of the OFTC community since its beginnings in the early 2000’s. He was elected as chairman a number of times in OFTC’s yearly staff elections. He’s known as cdlu on OFTC (a Canadian Debian Linux User), and is still involved with the network in an advisory capacity.

David’s a command line jockey. Many of his websites ran on a web framework he created using nothing more than the filesystem and bash scripts! He was employed as an editor for Linux.com for a number of years and he was a regular in the official IRC channel for the Debian Linux distribution. David has held senior positions with the Software in the Public Interest (SPI) organization. SPI was created to help other organizations (such as OFTC and Debian) develop and distribute open hardware and software. In short, David gets Linux and the merits of free software.

If you want to be extra safe, check that there's a big block of jumbled characters at the bottom.

I’ve only met David once in meatspace. At the time he was living in Guelph and needed to take a trip to Toronto. We met at a local pub called the Bistro on Avenue and enjoyed a dish of nachos and wings. During the meeting we took the opportunity to expand our GPG web of trust by signing each others public encryption key. What is GPG you ask? Well, it’s a way to securely communicate with someone over the Internet. Each person generates their own public and private key. When you want to send a message to someone else, that only the recipient can read, you encrypt the message with your private key and the recipients public key.

One problem with this process is that at the end of the day you still don’t know if the recipient of your message is who they say they are. Establishing a “web of trust” is a way for people to validate each others identity. Typically two individuals will meet in person with their printed public keys and proof of identity. Then from that point forward you can trust that the person who’s key you signed will be the only recipient of your message, assuming their private key is never compromised. This may hurt my geek cred slightly, but to this day, David is the only person I’ve properly validated identity with for GPG.

If you’re a technologist like me then you really cringe when you see politicians bungle a technology issue. Whether it’s comparing the internet to a series of tubes or thinking open source will hurt commercial interests, there are plenty of examples of technology issues that are handled poorly by politicians. The root cause of this is of course ignorance of how technology works. With technology ruling nearly every facet our lives, this ignorance can have serious consequences for our society. Many politicians are alarmingly impressionable by biased parties, such as lobbyists paid for by capitalistic entities looking out for their own best interests. I have high hopes that David, already a technology enthusiast, will be able implicitly understand many technology issues and help advise his colleagues accordingly.

David and I aren’t really close, but I consider him a friend. I was really proud of him when he started working in politics and I gave him my support when he ran for office in the 2015 election. You’re not going to find many people like him in office. Not only does he understand and embrace technology, but he’s an advocate for free and open source software. He knows how to use a Linux command line and he has some experience in programming. Now that he’s an elected official I hope that he continues to champion open technology initiatives at the highest levels of government. I encourage fellow Canadian technology enthusiasts to write to David to congratulate him on his victory.. and if you ever get a chance to meet him in person, make sure to bring along some ID and your public key to join his web of trust!

You’ll find David’s public key below

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: SKS 1.1.5
Comment: Hostname: pgp.mit.edu

mQGiBD0lFwURBACn6ecaZY+PcD0o5AJUDwDSjZgZYDR4202VlRa7yWDHzc+VjXVUMitEN8rU
iBGaIBd+TwzKMevhvNAnSZSq5WhwjmDxgOGcGOiSL16H8UStpyS1V8aTkaMfuSUy6uLVYZhD
gaZQKdB6XjkdVx5feUBZafn2dBvvjPlz9afh4gtpZwCg2BJmeSSoHpMsmUwClg80Ty8QCeMD
/RuSOgVswYlLCjkVWv0aWiqhPGlY6z4rPmTRRDNn80tVWsmoO75cQUS+wY90XQtZNkLRMkvy
vSIqFvTv6LoC0oECwxUcPYkciaSY2aIWIVokvS1bOdoOnRYlviwHrHYzdMbS2DqpuTB3yJPA
3FrButVSB7KisUxo+ZXPev35JLniA/0ekKgfOrsiCk5iKlOYePA85GQaH30jHKUYiZSg75Qy
QMmbQSQUmSDcvdKpKbXQugSU/Lgg/CRzKOcl2Rkz1eBBlaDbuDI+fCsQkhaKfzHpQyEcWwb9
EYkPpTvy8DbJVSTtsLqg/hswIrq+LwrteiKW8w+YXv4dmapmsIpfdt6b3rQiRGF2aWQgR3Jh
aGFtIChjZGx1KSA8Y2RsdUBwa2wubmV0PohGBBARAgAGBQI9VF4nAAoJEKVAqqQQ8ec0HBEA
n1MNA2TOunfwWixu5MX3tL859uW8AKCQwylFSKB4FRtnMZUw90RrIYlkW4hGBBARAgAGBQJG
mTZ9AAoJEDzyYD75xckzSIsAnjtaGej9W4liLSCzZ1c4sHgs8rvDAJ91CmVEaY9lBJUNkHgR
4g1PwqwQfohGBBMRAgAGBQI9Wtk4AAoJEDVzMsRagnotu+UAoIKyuSnOAoBo9+G0KhY0ORCm
ecgPAJ4wUDA+EMws2pK9TwkpeoddfChXP4hGBBMRAgAGBQI+FwgRAAoJECYOi6fu5I1/BMAA
oLxT8oFEAp7ibUQt9UOPe3GoMpDOAKC9GIExxQmXaLmvwwqj8X09co9pwIhJBDARAgAJBQJF
QjvjAh0gAAoJEIUaDqnV9FiJTDgAn3ZeV+FzabABIhmJPNdxfanfRhzMAJ43ipag0JSzZWBA
7W15SDuLWPC+QIhZBBMRAgAZBQI9JRcFBAsHAwIDFQIDAxYCAQIeAQIXgAAKCRCFGg6p1fRY
iZVnAJ9tShnxcqZ7Xfe6tOLmDq5pfr53kwCfYrek27b6YZJT3qRTyD8Sk65SuSG0K0Rhdmlk
IEdyYWhhbSAoU1BJIEJvYXJkKSA8Y2RsdUBzcGktaW5jLm9yZz6IRgQQEQIABgUCRnot0AAK
CRD5heNACvx0dmUmAJwMPxwd8XESWYhZ9reI8xpHTorjeACgk0gI6hM2q2Qe+tYNpTRkGZrQ
u5eIRgQQEQIABgUCRno5OAAKCRD3ssHBs0W900jtAJwNPGDebnIr8vHw7bJnAfz8Lby0kQCg
h6K8kfcPGXSYGT4jwJo7DItGgJSIRgQQEQIABgUCRnp+QAAKCRDIkHMPo/njDoVnAJ9cWgeM
K6NVyziqwREjWX/Y+uNk5gCfe3kXzQx1LZagKRc3Ct6f7VE3M4WIRgQQEQIABgUCRn60jQAK
CRALVEeiIA0VlsYDAJ9qi66z5zm04HvBiKwBWtoKRO/9dgCgmQ2yrDWh9xEJI718mIl8ZwyW
WnaIRgQQEQIABgUCRo4AhAAKCRBQctA2rFg1IEttAKDOfb6mfxgT33ZMUnT+PlJIyQCVKQCd
FYVB0+ftIHScK9InKtSo6YGZaaOIRgQQEQIABgUCRpQDRgAKCRCv5SzGOaalPyWQAKDvZMVl
CTXHm+1z3hM7s8nq+lbVNACg6+MwtkA09QRcrBuUOOjCk0ApIWuIRgQQEQIABgUCRpk2gwAK
CRA88mA++cXJM+aAAKCVuQSzw+ftmXR4gBe8i/yH0e4zMACZAfwaz5Aj5z+YykyTNt1HpLgO
F1OIRgQQEQIABgUCRpk3DgAKCRCxf9JBdmKCp1ObAJ9yUn4zazD9zKsomSo1hjq6JjkH5ACf
XnzXK2NQ2TdL5kB839+J6HZBFZuIRgQQEQIABgUCRrERXgAKCRBxXtagfnuKye99AKCXwIOr
QWSQuckrwUY+73EoQ7ST0QCeMQvSfNQa0V6a8kHeVFL3P2Z8986IRgQQEQIABgUCRu+6yAAK
CRAiGMgejnwD/1H5AJ4ytvnWM66TysiEHa+JfYnTZGNVUACdHEsw4aHlkEGhQGkmoQLaScp3
W62IRgQSEQIABgUCRng8lAAKCRBZJzzNVTDsdgbHAJkBA1ZkWE+M5nc/7HFM7uUfOmb2lACg
6a3ffbtG5Eak9+temTtsUrECkXuIRgQSEQgABgUCUEtklwAKCRCJzUshYHVZ5hrIAKCj/UEA
vBSLndHk/4TWHL8kMrTAhwCgutvm0Q/D/RbbPP28tJSkqBv860KIRgQTEQIABgUCRoh1zQAK
CRDeeq9ulMCcf1vKAKCvE1auX2Ex1eeywerJvJhczp38GwCfeyOIXzXqROy1XV+yb1krQRFW
HouIXgQTEQIAHgUCQCD4vwIbAwYLCQgHAwIDFQIDAxYCAQIeAQIXgAAKCRCFGg6p1fRYiel+
AJ9oOqGTmjyxaVzVUwBi4gLPLyDh5gCgnCBYXPEOy+fLWva9YpMlTOS9H1SJAhwEEgEIAAYF
AlBLZKoACgkQnqM0FPWFL06i5BAAwl0VptxZmbtWnfr5qY+y9Dg3MT5WZWf4rW5dSVwW8aN5
mazOQB2z8tSl7XBhh84tdKyN7GQ0aNeoIJHx+z1EjoneKN/R35d0HLk+sas9RP/3IXFJOC9R
1m0EOsuxGmWH469F/RnMYW+NcrB81QLzkxztacyEBRdwaFYRR7TOonwwC8tjrHfAcgVGwnKq
N3oq5hgAUX/dCUsr5yLVP11oJx+IsKYNExZlyrknv1ys8Q8g80i8uqAvRqiTlNsj3hM/WHSQ
xeDEOxthm+UwPTcasjxC7hoPusM/136UMBYdb0837qeX6kF8FgPVJMgR/12U3OdYiiw6/Vzw
cMEcxzMRffNqVZsBrQc5/fBSlOx6TDGbwxb7Hmcj0S9EkMyetFWWSvQJy1T+0B8yQhB9Khbn
t8FzhqFWAd/vaK3TgyVVUekhzSfJ+j8Lnwp1A9Qr2b2Hp62BaniNx1eFGrnk/+4z9f7rwBIR
M0r2VtsPvIPqtBCR/GOp3YfLJ9764q36o2LM6wi55HCYZ6ipSkQO+SQ3YHVy1EjbmICc4QQy
uICPIE/aftSrDPhzlpeg/x9g8pCN/J0eQVNiQ0UUnUXd+KT4QE/R4rwCzUn4p/PrChGbkgJ6
apzz5cDUHMzIsEFHdJt/6UAb0tKGhJMYuIROhviEDX/c2SBkDhc+AvVBW3lNV/a0LURhdmlk
IEdyYWhhbSAoTmV3c0ZvcmdlKSA8Y2RsdUBuZXdzZm9yZ2UuY29tPohGBBARAgAGBQJGei28
AAoJEPmF40AK/HR2iyUAoLfEm/b5z0sXqInRkB+LlRigps+4AJoDIIrBwBDC4LpvVdtRV9KE
KGaewYhGBBARAgAGBQJGejk0AAoJEPeywcGzRb3THx8AoI48mDhZgqKRvjTaQwNqSnsURj0f
AJ9RpmoFeXRfHNwZ/9UtknRzzVTRg4hGBBARAgAGBQJGen46AAoJEMiQcw+j+eMOsfoAnjsU
dTb/KPYhQVMZkgN3NrbC0W/uAJ9D0j0Ripn3aeFzPju/zJbfl4bZU4hGBBARAgAGBQJGjgB/
AAoJEFBy0DasWDUgdtIAoIee73MPsOVwKVnMbo+Atej7zmkYAJ0fZ60NsjiC/G28CIVGH9QB
VqQMMIhGBBARAgAGBQJGlANGAAoJEK/lLMY5pqU/HAEAni6Vy3fMB8eOY7PF/eLBxumtv6KP
AJ9KzCfRzs1kLETOCM+9yZe+2kZunYhGBBARAgAGBQJGmTaDAAoJEDzyYD75xckz/UYAn3sB
kHLzA0dVAf6DFrsjfDcLWYksAJsF+W6wm5GC6VbxPZjD4ilMnJ7JVohGBBARAgAGBQJGmTcO
AAoJELF/0kF2YoKnPFgAn2IctU3hC7j/gbffqBdm7adUm0OJAJ9+NwQ91jKqxyin5hX5pIDc
4wkJqYhGBBARAgAGBQJGp8nvAAoJEMWvd0pYUQtanEsAoJt8s3h68bm6PDpnIcDtVKMzFadk
AKCnEuZxXIOWQwbxcKGlq+uNkNtfWYhGBBARAgAGBQJGsRFeAAoJEHFe1qB+e4rJIeYAoJQV
cAKOfmoGG/MvxtfWVeTFc72tAJ9/7tbLvgP63yDpm9EHeMW9eOFYOYhGBBARAgAGBQJG77rI
AAoJECIYyB6OfAP/BpIAmwRGquZIyveLaSBmcq674tJgxIdSAKCUbbTr8Ia49hyDiO62Mxx4
/1w8S4hGBBIRAgAGBQJGeDyUAAoJEFknPM1VMOx2XjIAoJGZz+swkn30kxAPHRpzzta8bYAl
AJ4rJ07H1JbYzBU+8BPqCW3j+y7YgohGBBIRCAAGBQJQS2SXAAoJEInNSyFgdVnmDgkAn1ia
1gobwdCv+Prj9Ay6VeKWh7T7AKCUcOvF0DEWdvIYsfv0F+rc80cmb4hGBBMRAgAGBQJGiHXN
AAoJEN56r26UwJx/23UAnAtzIQn1KHTfbz/CrMNRnPAb+hjjAKDBmR029Tnvf4CNoTJEFXwv
ui4QzIheBBMRAgAeBQJAIPmbAhsDBgsJCAcDAgMVAgMDFgIBAh4BAheAAAoJEIUaDqnV9FiJ
USUAnihls15N2o2tl/YqleOqxMvf8ryaAJ0fXB8MEtoJnPOJ5HKMEmED0H9dkYicBBABAgAG
BQJGp8n5AAoJELRrkjttir5xKyMEAKc2vQqgVX8QEvX6RHQlZyqfDxhi5I0869RbnonjHkil
jLQI2Zv0qdC3NPhrkSD0XPN3waI+ycf94Gml8z+zxuZjNq/uFvticoanPyfTBeiHPR+C3n3k
2gk6v660ZTBBauc3plCOzEFj4SxhPTEimxYeUdHLO9GV/wfEZfypARZUiQIcBBIBCAAGBQJQ
S2SqAAoJEJ6jNBT1hS9OXZcQAM5PqrfE4SOGhk7bFTu+vMDn/5u6m82Cy7DPTaGyvfJ4yiTR
oL/hVR2e2I3nfuG0EbiFa9fRhyw06A2TmPsUfjAJ+ckC3vJ/QDg1yp/hSIgCQcXXLK2l1XUa
hf6JMZ7fH0KWhS79V5y4QrIwcr1xtUiUR2RYp22X6rX9OArKpEr/dbTj+OyTtMwAT+wLQnUr
5Zsl6VrvcuKSrruEpXbuLi/WD+n+AbWk5lRBvn0P4neGOMRw+Ii7TBGftimONELDEVhLi5ic
rzjZPxVHPkSdO6cMWHGTScpFWfPNcGdziIUkLNntIuQxOQPe2LkvyOZVmPohi9jbmd6BDL8U
3pA4OijQ+tzMJzBCxCsjZleRrszJW7FwfXsr5X/RxJjxYz9EwdHW4QcYQXMAenOK92vIwDUZ
DwOtotvKdJGPpPKXOnJG5nzJjAyey0n4RX40LEBSfz44c0P6YT8lIL5X0s0Fvy5BtoaET6kI
vlzn+lnKhJlQyxeUXbXawDI+dSAr/SX8ImN/u4bBr4SLTDwyX+ujnaTtmup0HrZKGD9xNtVv
TOYGuMU8zlEoU/7pLAvRy+8kSx8aSYS8qXIdYF2KfUGNXIDePWw8M67U0V8JcK+iOoKWIMWQ
iTxVwsNbZJ94IN+7qr1foftdE/ypkMDlOCBBAR8NXRPkHrxDzh8eo8x+NESftDZEYXZpZCBH
cmFoYW0gKFByaW1hcnkgZW1haWwgYWRkcmVzcykgPGNkbHVAcmFpbGZhbi5jYT6IRgQQEQIA
BgUCRnot0AAKCRD5heNACvx0dvZvAJ4hHpRX4ioQql+UJz2Hvmveec2X2ACghGfgQ5IlUhSN
0+N5VUu7ph1GTgOIRgQQEQIABgUCRno5OAAKCRD3ssHBs0W900FMAJ4sICNjkANMHE8KXHr9
1MRJqC+ZLwCfeNBJn47X0u4NCoIYYGV6xTgMbQeIRgQQEQIABgUCRnp+QAAKCRDIkHMPo/nj
DhhWAJ9D0Otd2yXLOWpkjfJZis0kdzBvrACdFcGm4QMygz/bW0NehR+Pf2tmX3qIRgQQEQIA
BgUCRn60jwAKCRALVEeiIA0Vlh0UAJ0QMuKOvw/yrkg/QChZKz+MKa0+LwCgmQrYRwJypFuv
MQNhW/RpzGzHTAKIRgQQEQIABgUCRo4AhAAKCRBQctA2rFg1IHB/AKCQO7ZXqaJJcfMnOMBu
WW5z07vczwCfe5c/nA5WUdymQXacrDLloZpQinOIRgQQEQIABgUCRpQDRAAKCRCv5SzGOaal
P0F2AJ9JcTS/jLZfG7pqIzHRGgckP0dLpwCgkOtBi9SuYSKxMyAYqTPVOOkeDSeIRgQQEQIA
BgUCRpk2fQAKCRA88mA++cXJM0iLAJ47Whno/VuJYi0gs2dXOLB4LPK7wwCfdQplRGmPZQSV
DZB4EeINT8KsEH6IRgQQEQIABgUCRpk3CwAKCRCxf9JBdmKCp/3MAJ41KWuA2JX0/1lohFFf
A27jppMCdQCfQ0a6YopRnFTHU0Ji/Lp0/14/K4OIRgQQEQIABgUCRqfJ6AAKCRDFr3dKWFEL
WvtIAJ0YrmluEB8n8E8k40+9cqH7dBP19ACgxCqsjTSAJ8bkG5eW8LNJG1hDZcSIRgQQEQIA
BgUCRrERWAAKCRBxXtagfnuKyb/bAJ0UgrTmSHCaR91/B96hOiFGKxAhdwCfUrHb9r2QYcC2
IGFeBsWMsR0b6J6IRgQQEQIABgUCRu+6vQAKCRAiGMgejnwD/zKnAKCNurLIDZdKS9NdQxwG
EetZGKV2VACcCnanhg2zmHvv3LbPOWtRJcVIczyIRgQSEQIABgUCRng83wAKCRBZJzzNVTDs
dhLPAJ4ozSIOhavzSwjHv4eDaxWc8YxjsACfWVvmY5iRZf+CsAenDPhHClA01lmIRgQSEQgA
BgUCUEtklwAKCRCJzUshYHVZ5s3QAKD7CaPZ5OYLgGrRUHsHQ+2CP/i3JgCgj7LluIO/biQl
u2NcdCK4fFFILBqIRgQTEQIABgUCRoh1zQAKCRDeeq9ulMCcf4HzAKDchPod3p0ElL2pJNX/
TVbDfXTn0gCgztApQfr+/GUqPZ0zCD5oDM7QCuSIXgQTEQIAHgUCQCD44AIbAwYLCQgHAwID
FQIDAxYCAQIeAQIXgAAKCRCFGg6p1fRYiW5VAJ45MhO9STFM1zc+BtveYevFgVNu6gCfd+Dl
8aUUd05AK+HjcQqdWIlR9jeIYQQTEQIAIQIbAwYLCQgHAwIDFQIDAxYCAQIeAQIXgAUCRpJf
HwIZAQAKCRCFGg6p1fRYiVmnAKCBeEl3Tf1uOVeHTkUSpjINrvsCzwCgnJl1qFhojkeg6MWN
l83AnF/QYkKInAQQAQIABgUCRqfJ9gAKCRC0a5I7bYq+cXo9A/4mYEHPFw/UXWQ4kmr1qPv0
Jmi4/UiPk9duwv7TwHtuXfM5aK16Tn7/rHmyC6M/dpLNpSEF9LTJZmBTEfcYGtSNPRxhJmgW
+Ztt7ZC8oxxQcSLaWm/6BWtpCzyHB2vm4b0br/Ek/Gs6n9oyTcTvGYU/B3TT8JcWJ+8UNHqt
KcrJ24kCHAQSAQgABgUCUEtkqgAKCRCeozQU9YUvTldvD/9Tbq9vS0duAmj01qrmsGsigiv4
8h28PeRl9cGs/QsUE6WqOjGbpoXwGoU1f1NndiqjBdbZZBx/PMEa5L+xkcZmka1IzjTG2Yr5
Ysd9Nivw95h6+HqpxQHY7MmP2EXF5UNiX8zvgAnTmIpL9mOXcgeatbm71JolGI6I+dD41p47
M2Fyz3lbLU1BLfxevP5QLUfxCaUjjHAszlco3LTTpshHQLdLE0DWdwQ0woQ49c6Xj8AFh44S
S1Gc7rqJ4QBDbK5Az5HHCn7YE/ajb9pvDfWtvYhveS8fk3rH1tZygdr8ZbMPxBOpYhHuwNzU
J6Flx6JEAdPbV0U42YhVprBtJX1dc9luVBQXf94cTjHsSxy2qK47oPulWGP7FWbzNIE0eKq6
IKqFCtRYJ473yz8Vl6ZeMP73qFFoUS50AojMaOsvexgDA8noa6sb2pKiLdIk8Lu0BTdTXFtc
/vPgsNK0ylD9mZNw2SLiby3nkz7Bf2TIXB9RDvZTtzH7XRcuiQLd4okmGRM5AxaUj3XwSHDS
APinij19FoeiRdh6gmQAyDPdHscIHJobIJM7iQ9of1kmO5i5Dc1gOx/LC9CHlaa7pSBuNbno
mtYaqOL7Nwie8a0OFU5NQ/NFfwptAB0GXCSaOrBVHRw51Z3IBdsPjesYHg/lDThwci11lLYt
1W66U6H6PbRDRGF2aWQgR3JhaGFtIChHdWVscGggSGlzdG9yaWNhbCBSYWlsd2F5IEFzc29j
aWF0aW9uKSA8Y2RsdUBnaHJhLmNhPohGBBARAgAGBQJGei3QAAoJEPmF40AK/HR2O7IAoM3U
FUjI2OjDr9iG6ZGwFLnBfGAPAJ9fxUxV1PGIoQSznzWeIQ4V3iNY9IhGBBARAgAGBQJGejk4
AAoJEPeywcGzRb3TLnkAoIm7OHiAodulCJycF1G9AlikX/k/AKCPCzvZntXEU5Ylh6X4nIoh
f/NGgIhGBBARAgAGBQJGen5AAAoJEMiQcw+j+eMOGYAAn0FrUzolgD+y1MuvSmqPVSyuyMcU
AJ9VPniEwCdOYa4Zj9qmS0jceJlhb4hGBBARAgAGBQJGjgCEAAoJEFBy0DasWDUgHicAn1Ol
ATaiDFYLBCTinTHqwODKy2shAJ9i8DNVwrXTY/yM5vboi4gRgs5NsohGBBARAgAGBQJGlANG
AAoJEK/lLMY5pqU/j4cAoLu6cQoqYD2f40PAjTQ2ionbqLNFAKCLkNfQJWRHppFb/DjowcGd
UI8JS4hGBBARAgAGBQJGmTaDAAoJEDzyYD75xckzQSkAn15B8rgBt33r10MJmk1r1Kxol7lu
AJ9zGV2+7jDR3N5ULKvG5YqCTl3fNohGBBARAgAGBQJGmTcOAAoJELF/0kF2YoKnruwAn3eg
HbI227YxofxhHbe9+bEgq4GGAJ9EBXp9bOApj18R3tsNjE1B1f4F/4hGBBARAgAGBQJGp8nv
AAoJEMWvd0pYUQta9KAAn3mRqROtj1TbJ4LkM4HqBkKtVV3IAJ0SSRmccyKEPSoyCuRNpmQ4
CGlHYohGBBARAgAGBQJGsRFeAAoJEHFe1qB+e4rJ8V4AoIDyKnPkBoa49h6lE4mW0gVvI8xc
AJ4gbIREXjq3UzFVJpYBxK/dMwy/NohGBBARAgAGBQJG77rIAAoJECIYyB6OfAP/lOAAmwWC
i8fGeydFNU/bVBQUU5S1HeddAKCP4BAtoFN85v3SNKuOPYsyILs8xohGBBIRAgAGBQJGeDzf
AAoJEFknPM1VMOx2DoEAn3XMCrK9M6trtfkf7F9y0yaoAPGdAKCRxCJgeznr1p+j/o+YPd7E
SHMmAohGBBIRCAAGBQJQS2SXAAoJEInNSyFgdVnmou4An3aboC1BC3SfnUPgY49hq3E7/ww6
AKDlKGAY2gxOErtUJmybilrjFuWw/4hGBBMRAgAGBQJGiHXNAAoJEN56r26UwJx/OXoAoIz6
1e+2tl+zFwDPYPtzBdVlWdu4AJ4n4RQ0OOePWwV4jANMsZ57xGWXkoheBBMRAgAeBQJAIPlF
AhsDBgsJCAcDAgMVAgMDFgIBAh4BAheAAAoJEIUaDqnV9FiJeeYAn0l7Ne2Xva3aEpHuH9Mm
WpqEhOFFAJ4z/a706XTicPIJat9+0S+TvIIK9oicBBABAgAGBQJGp8n5AAoJELRrkjttir5x
wNoD/i4/VzVnwTTz1q0jkOx+aaoTEHVPAya+R57ot5U96D1XI9z+M9572I5ugs3oKmK23XmD
4T0JLCMWtcdkT2g45+bIpOzcjMy4TwqXQs/eJqQjECDWyg3zu+SlD4jU/vajeUVrhxbTa6jU
yMm8TkOcgZN+kmaKM+jBcpedjJdViFf9iQIcBBIBCAAGBQJQS2SqAAoJEJ6jNBT1hS9Oli4Q
AMd7Vm0RDP8/R1Fo1JVVG6mR7OHoqBqwz9P5fgZYXrTrhL6jxdMD3mUGM9atLwdIv8gvt4eU
BImw45IXSFgEH49pL+1Kx3AsorADL1t92VndcOUEpeA2Fu8ahNv57lzURyYR5+3ESq9t/FHV
f3I9Dh5Nvmn3NnPXq3QjPb2UklugOT0ZTi3t4s8XVicHh9IMZ3QS+XubNpOhcggRzzosyYmc
6xgVhnoCtVqtLT86KJnxmLekbjLtqt9OkctUAzc99ADKvlcPntm7WJXpIBnR0I1CyBXOwCQh
noWDAtytQVwna3EN1/PnXm5hFJjEEeVS82oAvEo1AcqiJn+89xSy9OqodctDdKeYYfbiel7d
giC4fkob2h87xPvGfWI7rpMLZnIThDKSaHgRrMvM3HhBalfRCyatnw0LHpRt7atU16HfUUqj
LgPoGO8R3rvSuPGtuNNjPYKk4MDEwR/vFTpF1vPCnOk9ou2ECxPm38/ciSZGh7KfmASUfdDb
4CPyPet+p4ldb2RS0LBlMTS+BTvnLGdws8MK+2Av/u89EH0ziNXwT0oiXt5rGRzrEbEafFqm
xLweBRWlE6zEmop8bfA7x3VyI22CHh3og3fgN3ShdE7izKhCCUvZPAqH8lYlP1E7YmPLHmdy
vHm0FpuW0AitDYVCzdxCbd/V5Gdxjz2svsv7uQENBD0lFwkQBACqflYTEcBxcwfdXArUIj4T
/Bb1QBJY/Cv8/bvp19H2ffOSRMVnSZmyEa1uzbrvKz18NrN5aj2g4PqdOGKghlpx1e1h6++G
m3rPkSTQ4YwXKLzGyXT7X2F06Wo+debdkyA99s+T0EaTsm+QBZJA+qVSvQUDJgee2+uczlg5
9DgJxwADBQP/THM3XTw1m4H5F5M4W/YgK1tnosAtlDPKL8DH9DrgeqYrFjnSTTRnq4nEchOq
hbL+LVS8WaVNNTH/X7E/lH1xnxEPT6FiE55eFt7XDJEdjmUrUbtpwOvtVr8w1JnDFot5vhoy
INk5JXP0yyJy4bTEJAp1PHg2t+jPLY9iVvS1+ouIRgQYEQIABgUCPSUXCQAKCRCFGg6p1fRY
iaWKAKDJYbbH5zM6kbKstCktOaid0haV/QCeOf3jyPMXY/6wjOC4bZjoF/q0ywM=
=Mr30
-----END PGP PUBLIC KEY BLOCK-----

New features in connect-prism 0.4.0

Mon, 14 Jul 2014 01:57:16 +0000

Prism has had an abundance of new features since I first released it about a month ago. This post discusses the rationale behind some of those features as well as a peak of what's next.

Prism is a library to record, mock, and proxy HTTP traffic. Check out my previous post to learn more about its core functionality. Also checkout the github repo's for connect-prism, grunt-connect-prism, and prism-sample-project.

New features:

gulp support & core library usage
delay - simulate real world latency for mocks
404 stubbing - create mocks for endpoints that don't exist
URL rewriting
mock & record mode

gulp

Prism started out as a grunt plugin, but shortly after its début I started receiving requests for a gulp equivalent. Not having much experience with gulp I investigated how to develop gulp plugins and eventually discovered their writing a plugin guidelines. It's a really interesting read. The gulp community is a lot more strict about what makes a good plugin than the grunt community. Among various suggestions, requests, and outright demands, the number 1 requirement on their list is:

Your plugin should not do something that can be done easily with an existing node module.

If you're familiar with grunt this should strike a chord as there are thousands of grunt plugins that do nothing but wrap existing functionality of a core library. Whether this was intended from grunt's beginnings or not, I'm not sure, but one positive that has come out of it is that it has opened the door for less nodejs savvy developers to use popular libraries within their grunt configuration.

The gulp community appears to pride itself on its succinctness of a deployment pipeline through their file streaming operations. If your plugin isn't specifically integrating something else into the streaming pipline then it has no business being a plugin in the first place.

This inspired me to factor out the core functionality of grunt-connect-prism into a new simple to use core library called connect-prism. I've updated my prism sample project to include an example of its usage within a gulpfile.js.

To use with gulp you simply add the prism middleware to your connect declaration and then instantiate 1 or more instances of prism.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
var prism = require('connect-prism');
var connect = require('connect');
 
var app = connect()
  .use(require('connect-livereload')({
    port: 35729
  }))
  .use(prism.middleware)
  .use(connect.static('app'))
  .use(connect.static('.tmp'))
  .use(
    '/bower_components',
    connect.static('./bower_components')
  )
  .use(connect.directory('app'));
 
require('http').createServer(app)
  .listen(9000)
  .on('listening', function() {
    console.log('Started connect web server on http://localhost:9000');
  });
 
prism.create({
  name: 'serve',
  mode: 'mock',
  context: '/api',
  host: 'localhost',
  port: 9090,
  delay: 'auto',
  rewrite: {
    '^/api/bookauthors': '/api/authors',
  }
});

delay

The delay option is 1 of 2 features contributed by Miloš Mošovský. In many cases you probably want to use prism to eliminate latency of your API, but in some cases it's useful to see how your app behaves when it exists. The delay option has several different options that let you generate this delay when you mock requests.

You can define a time in milliseconds or choose one of three modes that generates random delays within a range. To learn more about how to configure this feature see the delay option section in the connect-prism README.

In the prism sample project I use the 'auto' mode of delay which will generate random delays between 500 and 1,750 milliseconds. Below is a screenshot of the chrome debugger's network tab showing the 3 calls made to prism.

404 stubbing

The second feature by Milos enables prism to generate stubs in your mocks directory when you're in mock mode (in record mode it would simply record a 404 response from your server). If your API doesn't have an endpoint implemented then prism will create a hashed file like it normally does, but the file will have a .404 extension and will have an empty data property value. If you choose to use the mock then customize the response data, status code, and content type (but NOT the request URL). To use the mock just drop the .404 file extension so it's in the standard requestHash.json filename format.

If you keep an eye on the STDOUT you'll see that prism lets you know when it creates a 404 stub:

Serialized empty 404 response for /api/newendpoint to mocks/serve/d36d19f9d90fa6e1fc1755463d7c3a70616539af.json.404

This lets you know which file to inspect in your mocks directory:

1
2
3
4
5
seglo@bit:~/source/prism-sample-project$ ls mocks/serve/
0b956737b1a9660f2c555f9328db1191f6f2a050.json
2723f866830446c640c9cc9942fed2988e0a2c1a.json
6654ad4d1d494222ce02c656386e6955575c17ed.json
d36d19f9d90fa6e1fc1755463d7c3a70616539af.json.404

Default state of the .404 file.

1
2
3
4
5
6
{
  "requestUrl": "/api/newendpoint",
  "contentType": "application/javascript",
  "statusCode": 200,
  "data": {}
}

URL rewriting

This is your standard, run of the mill URL rewriting functionality. It was copied in wholesale from the grunt-connect-proxy plugin. When a request matches a context that prism is configured for then it will modify the request URL to the matched URL replacement.

This could be useful for rerouting many same requests to one mock. For example, since prism creates and matches mocks given a whole request URL, if you have query parameters they are included as part of the hash. To ignore all query parameters to /api/authors you could create a rewrite rule like the following.

1
2
3
rewrite: {
  '^/api/authors\??.*': '/api/authors'
}

Mock & Record mode

While in record you may not always record every request you intended to. If you miss a use case then while you're in mock mode it will simply return 404's (and create stubs as mentioned above) when there may well be a valid endpoint it could have used. I created a new mode you can run the plugin that combines both mocking and recording.

When prism intercepts a request it will first check if a mock response exists. If it does not, instead of the usual behaviour of stubbing out a response it will attempt to proxy & record the response from your actual server.

This is useful while developing if you've want to move to a new area of development (that triggers new requests) without switching to and from record mode.

It's also helpful to use during automated testing so you don't have to remember to record mocks every time new tests are added to your suite; the next test run will generate them and then subsequent test runs will use the mocks.

This mode can be used with the 'mockrecord' setting for the 'mock' property of the configuration. See the mode setting in the connect-prism README for more details.

What's next?

Runtime configuration override

This is a candidate feature request submitted by Myles Bostwick several days ago. Myles proposed a feature where you could override the options of prism at run time with custom HTTP headers. This would allow you to override options like the prism name to use (i.e. in VCR-speak, use another cassette) in order to test a different response. This would be useful in test suites where you want to test how your app behaves under several different use cases, but not have to restart your test server or prism to do it.

Keep the suggestions coming

Since releasing this library it's been enhanced extensively through pull requests and suggestions from the community. I'm not really sure where it's headed. This is my first forray into open source worth mentioning and I'm thrilled by the feedback I'm receiving.

Record, mock, and proxy HTTP requests with grunt-connect-prism

Mon, 23 Jun 2014 12:15:45 +0000

EDIT 26/6/14: I've had a lot of great feedback from everyone. Gulp support is in experimental mode right now, but if there's anything else you would like to see then I encourage you to post an issue on my repo's issue tracker. Thanks everyone! :)

grunt-connect-prism is a new grunt plugin I've written to help make developing a Single Page Application (SPA) easier. If you're a RoR aficionado then you've probably heard of the project VCR, which is the inspiration for Prism.

The purpose of this plugin is to provide an easy way for you to record HTTP responses returned by your backend API (or some other remote web resource). After recording you can then replay the responses when requested and disconnect from the actual backend. It allows you to work independently of the backend. It's like an HTTP cache that persists cached responses to the disk and is only active when you explicitly turn it on.

I've written a sample project that uses the grunt-connect-prism project. The rest of this blog post will cite examples that are demonstrated in prism-sample-project.

Why would I want to use it?

Speed up development time

Prism is most useful for mocking complex & high latency API calls during development. Setting up a full stack for your web app can be costly and time consuming. The backend introduces a number of different variables to your development practices. In extreme cases, you may not even host it locally or have access to change anything. Having a way to record & mock calls to the backend can eliminate a lot of troubleshooting cycles and complexity from your development of a modern front end Single Page Application.

Speed up client side end to end and integration tests

Instead of taking the time to setup mocking infrastructure on your front end you can simply record 3rd party calls your e2e testsuite makes and then mock them out during future test runs. This allows you to avoid the complexity of testing a fully integrated application. It makes your test suite less brittle and has the potential to speed up test runs tremendously.

How does it work?

Prism works by adding a middleware for the grunt-contrib-connect plugin. It operates in much the same manner as grunt-connect-proxy, in fact by default it works the exact same way and lets you proxy traffic through your development connect server.

While in record mode you perform all the actions that trigger requests to your backend that you would like to record responses for. In record mode prism acts as proxy, but with the added capability of recording the proxied response. Prism will capture the request URL, including query parameters, content type, HTTP response status, and the data itself. If the response type happens to be JSON then it will pretty print it in the response file for easier inspection. For any other content-type it will fallback to a string.

An example response file in prism-sample-project:

./mocks/serve/6654ad4d1d494222ce02c656386e6955575c17ed.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
  "requestUrl": "/api/users",
  "contentType": "application/json; charset=utf-8",
  "statusCode": 200,
  "data": [
    {
      "user": "john",
      "fave_genre": "science fiction",
      "last_book_purchased": "the light of other days"
    },
    {
      "user": "genevieve",
      "fave_genre": "fantasy",
      "last_book_purchased": "game of thrones: a dance with dragons"
    },
    {
      "user": "zach",
      "fave_genre": "non-fiction",
      "last_book_purchased": "freakonomics"
    }
  ]
}

The filename is a SHA1 hash of the request URL in order to [almost] guarantee a unique filename for easy lookup in the future. After recording some responses you can switch to mock prism mode. When you launch your application and begin making the same requests you did before then instead of proxying your request, prism will intercept it, hash the request URL, and attempt to find a matching response in the appropriate directory.

If a response file can't be found then a 404 will be returned.

How do I use it in development?

The project homepage discusses pre-requisites and configuration in detail, but I'll summarize below. You must be using the grunt-contrib-connect plugin. If you use the Yeoman Angular.js generator (generator-angular) then this has already been done for you. Install the latest grunt-connect-prism package from NPM

npm install grunt-connect-prism --save-dev

Add the middleware to the appropriate target in your Gruntfile.js connect configuration section.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
    connect: {
      options: {
        port: 9000,
        // Change this to '0.0.0.0' to access the server from outside.
        hostname: 'localhost',
        livereload: 35729
      },
      livereload: {
        options: {
          open: true,
          middleware: function(connect) {
            return [
              require('grunt-connect-prism/lib/events').handleRequest,
              connect.static('.tmp'),
              connect().use(
                '/bower_components',
                connect.static('./bower_components')
              ),
              connect.static(appConfig.app)
            ];
          }
        }
      },
...
   }

Next create a prism configuration object. You must specify a context of your server that prism will listen for requests on. Ex) http://localhost:9000/api/ would be the context of /api. You must specify the hostname, port, scheme configuration of where this resource is actually located. This constitutes the minimum configuration required to get up and running. Below is taken from the prism-sample-project. For a full example see that projects Gruntfile.js.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
prism: {
  options: {
    mode: 'proxy', /* optional */
    mocksPath: './mocks', /* optional */
    context: '/api',
    host: 'localhost',
    port: 9090, /* optional */
    https: false /* optional */
  },
  serve: {
    options: {}
  },
  e2e: {
    options: {}
  }
}

Inject prism into your serve call. This is a modified version of what the yeoman generator-angular creates for you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
grunt.registerTask('serve', 'Compile then start a connect web server', function(prismMode) {
 
  var prismTask = 'prism:serve';
 
  if (prismMode) {
    prismTask = prismTask + ':' + prismMode;
  }
 
  grunt.task.run([
    'clean:server',
    'wiredep',
    'concurrent:server',
    'autoprefixer',
    'express:api', /* my example backend server running on port 9090 */
    prismTask,
    'connect:livereload',
    'watch'
  ]);
});

The prism task can be called like any other grunt plugin: {pluginName}:{target}. Prism has an extra argument called "mode". This optional argument allows you to specify the prism mode without having to reconfigure the prism configuration in the Gruntfile.js each time. In the case of the above example, when you run `grunt serve` it will run the prism:serve target by default. If you have not configured this prism target's mode then it will default to proxy.

grunt serve

To enter a different mode then simply pass in the appropriate mode value when you call grunt serve.

grunt serve:record
grunt serve:mock

How do I use it with end to end testing?

Using prism for end to end testing involves much the same configuration as for development. You must add the middleware to the connect target you're using in e2e testing (i.e. connect:test) then call the prism task when you execute your e2e tests.

For this example I'm using protractor, grunt-protractor-runner (installed with prism-sample-project), and the standalone selenium server to run my e2e tests for my Angular.js project. I created a custom task to launch my tests with grunt.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
grunt.registerTask('e2e', 'Run end to end tests', function(prismMode) {
  var prismTask = 'prism:e2e';
 
  if (prismMode) {
    prismTask = prismTask + ':' + prismMode;
  } else {
    prismTask = prismTask + ':mock';
  }
 
  // start the backend API if we're in record or proxy mode. this is probably
  // not applicable if you don't launch your backend server with grunt
  if (prismMode === 'record' || prismMode === 'proxy') {
    grunt.task.run(['express:api']);
  }
 
  grunt.task.run([
    'clean:server',
    'wiredep',
    'concurrent:server',
    'autoprefixer',
    prismTask,
    'connect:test',
    'protractor:e2e'
  ]);
});

The e2e task is configured to mock responses when you run `grunt e2e`.

grunt e2e

In order to record responses, or simply just let your e2e tests proxy to a real server then you can pass the prism mode into the e2e task in the same fashion as with the grunt serve task

grunt e2e:record
grunt e2e:proxy

Feedback

I hope that this blog post in conjunction with the README's of the plugin and sample project will provide you with enough information to get started with prism. Please see the grunt-connect-prism README.md for more information on the plugin and the prism-sample-project for a working example.

This is my first released open source and Node.js project. I would love feedback of any kind (pull requests, ideas, bug reports, critisicms, etc). Thanks!

Technology Change: .NET to Scala

Thu, 24 Oct 2013 14:15:11 +0000

In our careers as software developers we're frequently pigeonholed to one particular language & framework. Often times these specializations go even further and before you know it you'll find yourself being the front end guy, the datalayer girl, or in my case: the person who gets saddled with anything that has to do with deployment and devops work in general :). This is a story about how Empathica as an organization transformed itself from being a Microsoft .NET shop to a Scala one. It didn't happen overnight, and this is definitely not a howto guide on how to achieve similar change somewhere else. If you're looking for a side by side comparison of Scala and C#/.NET, you won't find it here. This is just an account of how the change itself unfolded. Since a transformation of this kind is thus far unique in my career I wanted to share it with the community and hopefully provide some wisdom for other teams deciding to take a similar plunge into Scala or some other open source technology.

The stage

Empathica was almost exclusively a Microsoft shop as late as the end of summer 2012. There was some legacy infrastructure in Java, but it had remained largely untouched since the company formed in the early 2000's. For all intents and purposes, Empathica embraced using Microsoft technologies throughout the entire stack. Everything was written in C#, all servers ran on Windows Server, OLTP and OLAP databases were SQL Server and Analysis Services. It wasn't until Simon Palmer joined the company as CTO that anyone in the development department gave the technology stack a hard look. After all, once you're fully invested in Microsoft it can be quite difficult to break away.

As an aside I would like to mention that in recent years Microsoft has made great strides in supporting the OSS community. They've open sourced large parts of the CLR, open sourced big projects like Entity Framework and ASP.NET MVC, and have even provided a portable class library license that can be run on platforms other than Windows. There are lots of new Microsoft OSS initiatives as indicated by their OpenTech division. However, I think it will still be some time (if ever) that we see widespread support and adoption of Microsoft works into the larger open source community.

When Simon came onboard it represented a sea change for Empathica's development department. At a higher level it changed the company from being services orientated to products based. The development process was switched nearly immediately to XP. Consultants were brought in from Berteig Consulting to help with the transformation. Internal political red tape was cut and development teams were empowered to do their own project planning, track their own progress, and self organize with as little direct supervision from the management team as possible. Members of the development team were encouraged to challenge their existing beliefs about how to build software.

EDIT 25/10/13: I want to elaborate a bit about when our new CTO came onboard. At the time, Empathica's products were stagnant, we were beginning to lose big clients, and something needed to be done. Simon was brought on to give the company more effective technical direction because up until then there were a few people wearing CTO-type hats, but not one voice in particular. Simon brought a wealth of software development leadership to the organization. He's incredibly tech savvy, a serial entrepreneur, and has a great deal of experience applying Agile practices and principles to large development teams. One of the first changes Simon made was to bring Agile to Empathica. It's my opinion that that change is what set the stage for developers to try new technologies like Scala.

As usually happens with change, it didn't resonate with everybody. A lot of the existing development staff weren't too keen on what was happening to the department. There was a slow exodus of technical staff who left to seek new opportunities. This gave the management team the opportunity to hire a new team of smart and passionate developers, product managers, and user experience people that were onboard with the new direction. It was around this time that I was fortunate to be offered a developer position at Empathica in July 2011.

The new team was challenged to reimagine our space (Customer Experience Management) by developing a new line of products. XP gave us lots of flexibility to operate as we saw fit. Teams were flat; the most junior developer had as much of a voice during an iteration planning session as a veteran of the craft. I had more input into the development process and product direction at Empathica than I did as a Team Lead and Development Manager in my past positions. I credit this open and collaborative culture as the main reason that we were able to try new technologies like Scala in the first place.

Why Scala?

Now that the stage is set, I'll elaborate on what exactly got the ball rolling to try Scala. Before every development project we have a "technology discussion". We're always open to trying new things, but for the most part new projects chose a base technology stack of C#, ASP.NET MVC, etc. We kicked off a number of new projects around August of 2012. One of these projects turned into a Scala project that I'll affectionately refer to as BeetleJuice for the remainder of this post. The kickoff for BeetleJuice started off as a "Hey, wouldn't be cool if we had something like this to compete with competitor X?" In fact BeetleJuice initial requirements were so scarce we didn't even know if it would be something we would release to our clients. It was an experiment; a grandiose software spike into an area of the business we never fully explored before. This gave us an enormous amount of liberty to try a new language and framework because:

The project wasn't overly complex
As with most software spikes there was an implicit permission to fail and to try new things
At the end of the day it didn't really matter what language or framework we chose for our application server because of its simple nature

It was then I challenged the rest of the team to honestly consider other technologies. "This is our chance to try something new. Let's take it." I said. Personally, I had had some experience with Python and Django, but I was also interested in Ruby and the Rails framework, so I proposed both. Another colleague tossed Node.js into the ring. And finally, one of our newest hires at the time, Steven Skelton proposed Scala.

I didn't know much about Scala. I gave myself the weekend to read up on it and I quickly appreciated its elegance and concise syntax. I recognized the opportunity that since it is JVM based we implicitly gained awesome platform interoperability and native OSS library support. Scala is statically typed, which seemed refreshing to me given the mainstream's infatuation with dynamic languages like Ruby, Python, and JavaScript over the past 10 years. It has a reputation as being a great language to write highly performant concurrent systems in. We were sold and in the next iteration the BeetleJuice team decided to give Scala a try.

The experiment

We created a Play! 2 web project. It was an easy transition from ASP.NET MVC from a framework perspective. We started developing BeetleJuice and it wasn't long before we knew that we could deliver a functioning Scala web application with a minimal hit on productivity.

Once we began demonstrating our progress at weekly iteration demo's it became clear that BeetleJuice was evolving out of its experimentation phase and into something that had real business value. When this became apparent, the management team decided to take a closer look at our technology choice and started asking tough questions like "How do we deploy this in a production environment?" and "Will this framework and language be around next year?" and inevitably "Am I going to be able to find people to support this technology in 1, 3, and 5 years?" I started investigating the maturity of Play! and to my delight found that it had an great list of endorsements from tech companies around the world (LinkedIn, Klout, theguardian, to name a few). The Play! 2 framework was still new when we first began using it, but its version 1 had been available to the community for some time. From Play! we soon learned more about the Typesafe stack and services. We came to the conclusion that due to Typesafe's inclusion of Play! into its stack and the successful history of the project itself, that we had picked a winner.

It was also becoming clear that Scala was being heavily adopted in the industry based on data from job sites like Indeed.com, Typesafe's impressive client list, the number of native Scala OSS projects available, and more anecdotal information like the number of recruiters contacting me on LinkedIn! This chart on Indeed.com shows that in early 2013 employers looking for developers with Scala experience for the first time outpaced the demand for those with Clojure experience or any other JVM language other than Java itself.

EDIT 26/10/13: I felt it important to only compare 2nd generation JVM languages. IMO it's not fair to include the Java language itself because it dwarfs adoption of 2nd gen languages due to its omnipresence in the industry for over 20 years.

Simon and some of our other technical management began dabbling with learning Scala themselves. I remember Simon telling me that the power of the language is obvious, but that it would be easy misuse it and get yourself in trouble. Why of course. With great power comes great responsibility. This reminded me of that old programmer humour post "How to Shoot Yourself in the Foot in Any Programming Language" by Mike Walker.

Scala: you stare at your foot for 3 days without any sleep, you then figrue [sic] out how to shoot yourself in the foot with one line of code… recursively.

Granted, this addition came from a commenter about 7 years after the original post, but I think the author described Scala perfectly within the context of the meme!

Legacy integration

Integration with the rest of our Microsoft platform and products was a concern from the start. It was time to start investing in backend API's. Not only to integrate with our .NET projects, but because of a separate long term goal to abstract away our persistence layer so we could change it in the future.

Our API initiative began with serving only the data needs of BeetleJuice. We started a spike on different API technologies. The first was a tomcat based web service that let you abstract queries based on an ANTLR domain specific language. You could craft HTTP requests that contained projections, filters, and joins (like SQL) and it would return a JSON representation of the data. It worked, but it was quite complicated to use and the queries looked a whole lot like our data access queries themselves.

Our second API implementation was written by Steven and based Apache Thrift and Twitter's Finagle. The API endpoints had methods you could call much in the same way as a traditional RPC service. Thrift/Finagle gave us the capability to generate native C# and Scala code so we didn't need to manually write our own data transfer objects, clients, and services. Finagle has a ton of functionality out of the box such as easy horizontal scaling with ZooKeeper, excellent stats using Ostrich, concurrency made easy with Twitter's library of threadpooling logic, and an easy way to instantiate servers and client connections. Thrift/Finagle is an excellent choice to write API's in. Steven has blogged about Thrift and Finagle extensively and even gave a short talk at a local Toronto Scala Meetup. You can find his slides here.

Once we had a Thrift API ready and in production we were able to get BeetleJuice and our other .NET projects to connect to it and start consuming data.

A moment of doubt

BeetleJuice and our new API were proving to be a success. It wasn't long before a large new project, Grail, was on the table and a new technology discussion began. This discussion was different than those past because we now had some actual Scala experience on the team. We even hired top Scala talent such as the likes of Katrin Shechtman (who runs a cool startup on the side called Becipe that runs on a Scala, Play! Mongo and Heroku stack). I'll admit that I had my reservations about the language and I thought it was my responsibility to contrast some of the Scala evangelization that had been growing on the team and play devil's advocate.

My main argument against Scala was a common complaint about the tooling. The compiler's slow, the IDE's are buggy and lacking in features, the preferred build and dependency management framework, SBT, has a tremendous learning curve. I was seriously questioning whether or not the rest of the developers on our team were going to get onboard with Scala knowing that using it would mean taking a productivity hit at first. I made my arguments, but ultimately it was decided it was worth the risk. I was relieved that management and my developer colleagues were onboard despite the drawbacks. It was at this time that I decided to make it a priority to really embrace Scala and find new ways to cope with the different development practices and toolchain from what I was used to in .NET.

An endorsement of Open Source technology

After Grail had been humming along for a few weeks, Simon made an official announcement to the development department that endorsed the use of Scala, Linux, and other open source tools and technologies. This was a critical moment in our organization because up until then C#, .NET, and the Microsoft stack were our default choices. This was Simon's announcement outlining our long term technical strategy.

To: Empathica Developers
Subject: Technical Strategy

Folks,

I seem to have set the Scala cat among the C# pigeons yesterday, so I wanted to lay out my view of our long-term technical strategy in the hope that it will allay some fears and clear up any ambiguity that I may have caused. And I’d like to apologise if anyone felt disenfranchised, or as though there were important decisions being made without their input.

First, some principles:

We make a point of recruiting the brightest technical people we can find and our expectation is that you will be comfortable working in whatever technology is required for the job at hand. I think you all fit into this category and I hope that you feel the same way, but if you are working for a long time in a single technology a rut can look a lot like a groove.

The job at hand will require different technologies depending on the need, and we will not be bound to any single technology. A technology choice will be made appropriately for the needs as we see them. The choice of technology is one of the hardest because it happens early and is somewhat irreversible. It also has many dimensions, some of which are hard to discern at the start of a project.

We have to balance our agile rapidity against good architecture. It is one of the core criticisms of agile, and particularly XP, that you too easily ignore architecture as you ricochet from one user driven feature to the next. This point is particularly relevant to us now as we are embarking on two significant architectural changes, one being a core data model change, the other inserting an API layer.

There’s never a good time to make a large architectural shift and you will always be balancing tactical commitments against long-term aspirations, as we are right now. Furthermore you never get given the time to do it. A data API is one of the key missing components in our platform. What we have done with the [API] is an excellent example of how to start to insert a formal tier in the architecture, and what we are contemplating for [Grail] and [MultiPass] access is a good natural extension. The success of any future revolution in our data tier will be contingent on us having done the [API] work well.

A profusion of service endpoints has a somewhat bad architectural smell. You should have good reasons to fragment your APIs into many pieces and expedience is rarely a good reason. A single endpoint may not be the answer either, but there should be a good architectural reason to split it apart.

With these in mind here is what I would like to see us move towards in the medium to long term:

Minimal dependence on Microsoft, both O/S and data stores

Increased infrastructure in “the cloud”

Horizontal scalability as a fundamental architectural principle

As a consequence this probably means:

Less C#/.Net, more Scala/Java/Open Source and

Less SQLServer/OLAP Services, more NoSQL/Columnar Storage/Mongo/Vertica

Less Windows, more Linux

Less cabinets/pizza boxes, more AWS/Private cloud

Taking a leaf from the agile manifesto wording, I mean: “while I see value in the things on the left, I see more long-term value in the things on the right”.

So, no hard decision has been made about our future languages or technologies, and there are no rules that all new development must happen in Scala. Instead I want us all to align on the principles and then make the right decisions as we balance our short-term needs against our long-term aspirations.

If you have any further questions please feel free to come and find me. I am stuck at home today because the four snowmen of the snowpocalypse arrived in Oakville this morning, but I’ll gladly take a phone call.

Simon Palmer
Chief Technology Officer
Empathica Inc

- Simon Palmer

Dealing with technology change

Not everybody was pleased with the transition to Scala. Giving up years of experience with one particular language and framework just to be a "beginner" with a new one can be a tough pill for some people to swallow. Most people inherently don't like change. For a programmer, changing technologies can have a profound impact on your productivity and in some cases your mental stability! In our industry you have to fight to overcome these innate reactions.

In the technology world there are no excuses for people who don't adapt. Software development doesn't have a deep body of knowledge like some engineering practices do (I know calling software development an engineering discipline is a slippery slope, but please indulge me). For example, the know-how and best practices for building a bridge have been known for thousands of years. What we consider software engineering may have been around for close to 50 to 80 years, depending on how liberal your definition of what defined the beginning of software engineering is. It could be decades or even centuries before we reach some kind of steady state of technological development. Until that time people in the software industry need to learn to adapt and embrace what's new or be left behind.

A good carpenter finds the right tool for the job. This analogy easily extends to software, especially when the tools and jobs themselves change at such an astonishing pace! Compiled, scripted, imperative, or functional. Pick something the community ascribes to to get the job done. Haven't used it yet? Try it out! Those who don't strive to learn new languages and technologies are in for a rude awakening when the years of experience they have with Blub don't count for anything with employers.

A data renaissance through API's

There became a need to extend our Scala Thrift/Finagle API as Grail was being developed. It was decided that when there were data needs for new projects they would be put behind our API. This began a data renaissance of sorts; not only did we put Grail's needs behind an API, but as we identified overlapping needs of our existing products, we retrofitted them to make use of the API as well. This had a snowball effect of establishing API's for the remainder of our legacy projects that were still under active development.

A significant allocation of our development team was refactoring projects to use Thrift while at the same time establishing data contracts for API's. For a number of months we checked off boxes on our API wishlist. An incredible amount of code was written, refactored, rewritten, or simply not needed any more. Our product middle tiers shrank down to a token of their former sizes. Introducing an API layer forced us to establish clear lines between our apps and their data. This separation of concerns was a huge boon to the distributed architecture of our entire infrastructure and in line with the sentiments Simon made in his technical strategy.

We started thinking a lot about concurrency and distributed systems. We started investigating new data access technologies to standardize across our API layer such as Slick for RDBMS and Couchbase for caching. We had already been using Mongo for parts of our infrastructure, but we also started planning an overhaul of the rest of our RDBMS infrastructure from SQL Server to cheaper and ironically, more scalable solutions like Vertica (a columnar relational database) and "No SQL" databases. All of these decisions were made that much easier because of the introduction of our API layer and the myriad of library choices available in the Scala and Java OSS ecosystem.

Training and education

A lot of Empathica developers were interested in using Scala. Unfortunately, other than those that had been working on BeetleJuice, Grail, and our API, there weren't a ton of people with experience in the language. We entertained the thought of bringing in instructors from Typesafe or their Canadian Scala training partner, tindr.co (more on them later), but in the beginning we looked to Coursera and their growing catalogue of Massive Open Online Course's (MOOC's).

Martin Odersky, the inventor of Scala and a key figure in the Java community, provided video lectures for the Functional Programming Principles in Scala course on Coursera. This course has been wildly popular and is regarded as one of the most successful software programming MOOC's ever.

We had more than 50,000 registered students— an unfathomably large number in the context of traditional teaching. While large, that number doesn’t tell the whole story; as is typical for a MOOC, a statistical majority of those students participate no further beyond watching a couple of videos to find out what the course is about. Of the 50,000, about 21,000 students participated in the interactive in-video quizzes that are part of the lectures, and a remarkable 18,000 unique students attempted at least one programming assignment. A whopping 9,593 students successfully completed the course and earned a certificate of completion— that’s an incredible 20% of students, which blows the average 10% rate of completion for MOOCs out of the water.

- Functional Programming Principles in Scala: Impressions and Statistics By Heather Miller and Martin Odersky

With the blessing of the management team we decided to allocate one day a week for the length of the course to watching video lectures and working on weekly assignments. On Thursday morning we would meet together in a room, discuss last week's assignment and watch the lectures for the week. We often paused and attempted to solve in-lecture exercises and to discuss the concepts being taught.

Some of us had doubts of how useful a functional programming course would be in day-to-day work at Empathica, but with the explosion of LISP dialects in recent years and the need for side-effect free code that can run concurrently there's been a huge demand in the industry to apply these skills to new systems. The course exercises do involve a lot of algorithms and basic data structures, but it's also an excellent introduction to Scala syntax such as pattern matching, for comprehensions, Scala's collection types, and much more. I would strongly recommend this course to anyone interested in learning Scala, functional programming, or both!

As I mentioned earlier, there are many vendors that offer a variety of Scala training options. tindr.co not only runs training courses based on Typesafe curriculum, but they've also introduced a new program called the Scala Developer Factory; a rigorous training course that promises to deliver skilled and productive Scala developers! I've kept a line of communication open with Mike and Eric at tindr.co to discuss possible future on-site training options for the developers at Empathica.

Embracing the Scala community

Toronto has an active Scala community. Several meetups are planned each year and are usually hosted at software shops that have adopted Scala into their organizations. The Toronto Scala Meetup has run meetups for a few years. Chris Dinn has often organized these events. Speakers from the community are invited to present on any Scala related subject they like. Representatives from Typesafe and tindr.co often show up as well.

Katrin proposed the idea that we host a meetup at Empathica's downtown office. At the time, our office was a dingy top floor on Peter St. atop an infamous night club in Toronto's entertainment district called "Time". Plans to host the meetup were shelved until we setup shop in our new digs on Spadina ave. With renewed vigor, Katrin began preparing a schedule. Our RSVP list grew. We ran the event with 3 speakers and representatives from both Typesafe and tindr.co. Empathica provided the space plus refreshments and the folks at tindr.co were kind enough to buy pizza's for a group of roughly 50 people. It was a huge success and the social gathering after the talks allowed for lots of networking and prompted many interesting discussions. I can safely say on behalf of the devs at Empathica that we're psyched to continue to attend, contribute, and host these events in the future.

Reflection

There are now several dev teams at Empathica working on Scala projects. Tool chains are being developed. Coding standards are being established and enforced. Libraries are being standardized across our projects. It's starting to return to a comfortable pace of software development for everyone.

That's it. I'm known for being verbose so I apologize for the lengthy read, but I wanted to get this story out in its entirety if for no other reason then for Empathica's own posterity. It wasn't an easy transition. Looking back, there are things I probably would have done differently, but I think with the information available we did a pretty good job. I hope that for those of you thinking about a technology change that our experiences help you come up with a plan to bring about that change. I'm sure there are also some people reading that have gone through such a change and I welcome your input in the comments whether good or bad.

If I were involved in a significant technology change again I would definitely take a different tack both in its proposition and implementation. Some things I would do differently and some things I would not. I've compiled a summary of what I think are the most important matters to address.

Consensus building. Get the whole development team's input and suggestions. Be as accommodating as you can, but don't expect unanimity. It's important to get input from everyone you work with on a daily basis because they will all be affected by the decision.
Start with something small that has potential to grow it into something big.
Budget for training in your iteration/sprint/project plan. Some people prefer learning on their own by reading a book, some like the classroom environment, and some just want to start writing code. See who falls into what group and try to accommodate them all.
Ask hiring prospects how they would feel if they worked with a different technology than one they have experience with.
Include OSS experience as part of your hiring criteria. Either experience contributing to specific projects or experience working with open source languages and frameworks.
Be patient. Significant technology changes don't happen over night. But with the right people and the right attitude amazing things can happen.

I'll conclude this post with a link to a little bit of internet history. This video made the rounds in 2010 and although it does not specifically endorse Scala it's still a harrowing story about leaving the Microsoft nest and embracing open source technologies. Plus it co-stars the lovely Scala Johansson :)

(NSFW, depending on where you work!)

EDIT 08/09/14: If you would like to know more about our transition on .NET to Scala then check out my follow up post: .NET to Scala developer Q&A with Typesafe.