Submitting Spark jobs using Kubernetes scheduler (Spark > 2.3.0)
Follow the discussion between Lena and Bernd and learn about a new Kubernetes scheduler option in the latest release of Spark 2.3.0. Lena gives Bernd a quick overview of Spark, and shows him a hands-on example of submitting a job to run on Kubernetes cluster in Azure Container Service (AKS). They go over the necessary steps to prepare for job submission, as well as details of what happens when the job is being created, running or completed. In addition to learning about technical aspects of the topic, enjoy the magnificent view of the pacific-northwestern nature and explore the Treehouses on Microsoft campus!
Data ingestion, stream processing and sentiment analysis pipeline using Twitter data, Event Hubs, Spark and Cognitive Services
Follow the conversation between Lena and Suz and learn about setting up a data ingestion and processing system consisting of event producer, reliable event aggregation and consumer using Twitter client, Event Hubs and Spark on Azure Databricks as an example. Lena and Suz are also discussing alternative options for stream processing, and how it can be used for various scenarios, including IoT, and how to apply machine learning to streaming data by showing an example of sentiment analysis on tweets coming in real-time.
Intuitive Distributed Algorithms with Examples
Presentation Natallia Dzenisenka and I did together at OSCON 2017 in Austin, Texas.
You can find slides here.
Keynote at Open F#: FSharp is the Future of OSS .NET