Eventually Perfect Distributed Systems

This article is based on O'Reilly Velocity 2019 Keynote by Lena Hall.

It's about the impact of our work, the complexity and obstacles we face, and what is important for building better distributed systems, especially when other life-critical areas rely on and build on what we create.


Continue reading!
08.20.2019
|
Keynote O'Reilly Velocity Distributed Systems Distributed Algorithms
Azure Functions and Azure Storage: secure authentication with Managed Identities and without managing keys!

In this article you will learn about using Managed Identities for secure and convenient approach to authenticate to Azure services from Serverless Functions without managing or storing any access keys or credentials!

We will create an Azure Function, obtain an access token from local service identity endpoint, and we will use the access token in the request to a file on Azure storage account.


Continue reading!
09.27.2018
|
azure security access identity azure functions fsharp
How To Securely Access Cloud Resources From Linux VMs On Azure

In this article we are discussing Azure Managed Identities!
Why is it a good idea to use them?
How can they help make access to Azure resources more secure and less error-prone?

They can significantly improve authentication process between cloud resources, reduce operational overhead and vulnerability of managing and storing authentication credentials.


Continue reading!
09.25.2018
|
azure security access identity
Processing streams of data with Apache Kafka and Spark:
ingestion, processing, reaction, examples

We need to be able to build solutions that can receive data from a variety of sources, perform specific computation and analysis on data on the fly, and possibly perform an action as a result. We will look at a very atomic and specific example, that would be a great starting point for many use cases:

  • How to build a decoupling event ingestion layer that would work with multiple independent sources and receiving systems
  • How to do processing on streams of events coming from multiple input systems
  • How to react to outcomes of processing logic
  • How to do it all in a scalable, durable and simple fashion


Continue reading!
06.21.2018
|
apache spark apache kafka azure scala databricks kubernetes
Event stream processing architecture on Azure with Apache Kafka and Spark
Learn about combining Apache Kafka for event aggregation and ingestion together with Apache Spark for stream processing!
Continue reading!
04.01.2018
|
apache spark apache kafka azure scala databricks
Sentiment analysis on streaming data using Apache Spark and Cognitive Services
In this article, we will learn about performing transformations on Spark streaming dataframes. We will be doing stream processing using Spark Structured Streaming, and sentiment analysis on text data with Cognitive Services APIs as an example...
Continue reading!
02.7.2018
|
apache spark sentiment analysis streaming machine learning databricks
Getting more control over starting point of Spark Stream consumption with Offset and EnqueuedTime filters on EventHubs messages
It is convenient in case the consumer has stopped reading events and needs to continue from the exact point it left off earlier, or we might want to start consuming all events that were produced not earlier than a certain point in time. In this article we will learn how to have more flexibility around the starting point of consuming streams in Spark Structured Streaming. Previously, I wrote how we can...
Continue reading!
01.30.2018
|
apache spark event hubs streaming azure-event-hubs-spark databricks
Using Spark on Databricks to consume data from Event Hubs
In this article, we are going to set up a data ingestion system and connect to it from Spark to consume events to do further processing. As an example I’ve chosen to use Twitter data as a source, because it is freely available to anyone who creates a Twitter App, and new data comes in real-time...
Continue reading!
01.25.2018
|
apache spark event hubs streaming azure-event-hubs-spark databricks
Lena’s (distributed) snapshot of 2017!

What happened in 2017?
Continue reading!

01.01.2018
|
New Year
Running a Distributed Database on Kubernetes on Azure

What is the benefit of running a distributed database on Kubernetes? Time it takes to create a Kubernetes cluster with a distributed database on it is just few minutes. Same to destroy the cluster. It can take a long time to properly set up a bare metal cluster, or even a VM one, which is less flexible. If using a containerized distributed database, it can significantly reduce costs spent on clusters for development and testing, because the cluster doesn’t have to exist when unused and it provides more flexibility.
Continue reading!

11.15.2017
|
azure kubernetes containers cassandra distributed databases
Setting up a Kubernetes cluster with Azure Container Service:
Terraform, Azure Resource Manager, Azure CLI

When working on a distributed system with many moving parts and diverse platforms, there is always need to have an easy way to manage your system altogether. Azure provides a way to use well known and proven orchestrators: Kubernetes, DC/OS, Docker Swarm. If your services need to be adjustable, Azure Container Service (ACS) is a right tool with a lot of flexibility. It is possible to create an ACS cluster using Azure Portal in the UI. There are many options to set up an ACS cluster in an automated, repeatable, or immutable infrastructure paradigm, and we will go through them in this article.
Continue reading!

10.22.2017
|
azure terraform container service kubernetes
Exploring Azure Container Instances and Web Apps for Containers
Recently, Azure introduced Azure Container Instances and Web App for Linux and Containers. It makes it easier to host applications for more diverse application stacks. Azure Container Instances is the fastest way to deploy containers on Azure, because instead of building an underlying VM, ACI just assigns an existing VM to the container. It can also work with Kubernetes through an experimental ACI to Kubernetes connector...
Continue reading!
10.14.2017
|
web apps azure scala fsharp docker containers
"Women in Tech" is all about "Tech"
Reading through tons of papers about databases, networking and distributed systems I don't usually pay special attention to who wrote it, until I see lots and lots of really useful papers and same names. In this little post I wanted to make a list of my favorite women in tech, or software engineering, or computing - whatever you sounds better for you.The main thing that is common for all of the women below - they all work hard, they are fond of their work and research, and they don't give a fuck about what somebody else thinks they should be doing instead.
Continue reading!
06.09.2016
|
role model
What makes you feel weird within your primary language ecosystem
Recently there are a lot of conversations about whether you can or can't be a good .NET engineer, or Java engineer, or whatever other engineer. The word "good" means different things to different people. We can always look at this question from many different perspectives and the answer can be yes and no.
Continue reading!
16.06.2016
|
language ecosystems
Vital things an Architect must know about adjustable and flexible system design
If you're a good Architect, you can figure out the most reasonable architecture for some specific system at some specific point of time, based on known priorities. The major problem is - there is no general perfect architecture. Industry, science and business are moving forward fast as a light-speed rocket, so does technology. In the era of data all over the place, technologies and frameworks are polishing themselves, offering new innovative approaches. The moment will come when you have to make changes to the architecture that used to be perfect, and it's not bad - we just have to be armed and ready for that...
Continue reading!
28.03.2016
|
SOA microservices architecture distributed systems devops cloud-native infrastructure best practices
Cassandra on Docker using F#
In this blog post I will show you how to set up the Cassandra cluster on Docker and how to use F# to connect to it to execute any type of your application specific operations and queries. I am using Mac, having Docker Toolbox installed on OSX with F#. For the code example I am using F# ProjectScaffold template with Paket in Xamarin Studio on OS X...
Continue reading!
02.01.2016
|
cassandra docker fsharp
Learning the machine!
This post doesn't assume or require any knowledge of machine learning algorithms. It might be helpful to people who not always have time to implement the algorithms by themselves and prefer to use the available frameworks. I will go through the general topic of machine learning, explain the idea behind the several kinds of typical machine learning algorithms and show the code examples with F# and Accord.NET framework, which is fully open source...
Continue reading!
13.12.2015
|
fsharp machine learning data science accord.net functional programming
Hello West Coast!
This is a blog post that I was going to write few days ago but was very very busy with other things in life.So I have recently arrived in the USA, to speak at the PADNUG meet up. I have also spoken at Portland F# User Group and .NET Unboxed already. Next conference is QCon in San Francisco. Here I however would like to tell about some things that I've found fascinating, have impressed me, or that I have found different from what I am used to...
Continue reading!
30.10.2015
|
USA West Coast Honeymoon Thoughts
My reasons to love F#.
This blog entry contains a simple list of reasons why I find F# useful and suitable for various kinds of tasks.
Continue reading!
20.08.2015
|
fsharp functional programming love why reasons
Distributed computations and Big Data with F# cloud monads - Part III.
Part III gives an overview of the features of MBrace framework.
In this series you will get the concept of "cloud monads" or cloud computation expressions, find out how to create a cluster and configure the environment, explore the features and opportunities we have with new F# functional cloud programming model and view the code of MBrace examples.
Continue reading!
19.05.2015
|
fsharp cloud big data mbrace functional programming
Distributed computations and Big Data with F# cloud monads - Part II.
Part II describes the infrastructural options of creating the cluster for MBrace.
In this series you will get the concept of "cloud monads" or cloud computation expressions, find out how to create a cluster and configure the environment, explore the features and opportunities we have with new F# functional cloud programming model and view the code of MBrace examples.
Continue reading!
19.05.2015
|
fsharp cloud big data mbrace functional programming
Distributed computations and Big Data with F# cloud monads - Part I.
Part I gives an understanding of the approach and the idea of what happens behind the scenes when we use cloud computation expressions.
In this series you will get the concept of "cloud monads" or cloud computation expressions, find out how to create a cluster and configure the environment, explore the features and opportunities we have with new F# functional cloud programming model and view the code of MBrace examples.
Continue reading!
19.05.2015
|
fsharp cloud big data mbrace functional programming
Selling F# to enterprise.. Or, how to speak with managers?
When I asked the question on twitter about what problems people see in current way of F# development and growth.. one of the answers was: "Enterprise people who think C# is the best and only the best are a problem ;) But I wouldn't know how to reach them...". So I will try to express my thoughts on this topic here.
Continue reading!
19.04.2015
|
fsharp community enterprise profit
Monads O_o
Inspired by people, who don't get monads.. After reading this post you will know how to understand monads and see the step by step strategy of this process.
Continue reading!
31.03.2015
|
monads functional programming buzzwords learning
How I feel Big Data. Getting started!
I'm interested in Big Data for some period of time. Big Data - quite blurred, unclear and uncertain words. This story shows logical path I went to understand what it is about.
Continue reading!
07.07.2014
|
big data nosql hadoop
Hello World!
First test blog post! Hello World! I will be writing about art and magic of software engineering, about my opinion on different topics. Welcome!
Continue reading!
06.07.2014
|
hello world test yay
Yandex.Metrica

Data-intensive, distributed, cloud-native systems

Living in beautiful Seattle, Washington

Say hi!
lenokhall@gmail.com