Spark Developemnt



Course Outline
Introduction to Apache Hadoop and the Hadoop Ecosystem

Introduction to Apache Hadoop and the Hadoop Ecosystem
Apache Hadoop Overview
Data Ingestion and Storage
Data Processing
Data Analysis and Exploration
Other Ecosystem Tools
Introduction to the Hands-On Exercises
Apache Hadoop File Storage

Apache Hadoop Cluster Components
HDFS Architecture
Using HDFS
Distributed Processing on an Apache Hadoop Cluster

YARN Architecture
Working With YARN
Apache Spark Basics

What is Apache Spark?
Starting the Spark Shell
Using the Spark Shell
Getting Started with Datasets and DataFrames
DataFrame Operations
Working with DataFrames and Schemas

Creating DataFrames from Data Sources
Saving DataFrames to Data Sources
DataFrame Schemas
Eager and Lazy Execution
Analyzing Data with DataFrame Queries

Querying DataFrames Using Column Expressions
Grouping and Aggregation Queries
Joining DataFrames
RDD Overview

RDD Overview
RDD Data Sources
Creating and Saving RDDs
RDD Operations
Transforming Data with RDDs

Writing and Passing Transformation Functions
Transformation Execution
Converting Between RDDs and DataFrames
Aggregating Data with Pair RDDs

Key-Value Pair RDDs
Map-Reduce
Other Pair RDD Operations


Querying Tables and Views with Apache Spark SQL

Querying Tables in Spark Using SQL
Querying Files and Views
The Catalog API
Comparing Spark SQL, Apache Impala, and Apache Hive-on-Spark


Working with Datasets in Scala

Datasets and DataFrames
Creating Datasets
Loading and Saving Datasets
Dataset Operations


Writing, Configuring, and Running Apache Spark Applications

Writing a Spark Application
Building and Running an Application
Application Deployment Mode
The Spark Application Web UI
Configuring Application Properties
Distributed Processing

Review: Apache Spark on a Cluster
RDD Partitions
Example: Partitioning in Queries
Stages and Tasks
Job Execution Planning
Example: Catalyst Execution Plan
Example: RDD Execution Plan
Distributed Data Persistence

DataFrame and Dataset Persistence
Persistence Storage Levels
Viewing Persisted RDDs
Common Patterns in Apache Spark Data Processing

Common Apache Spark Use Cases
Iterative Algorithms in Apache Spark
Machine Learning
Example: k-means
Apache Spark Streaming: Introduction to DStreams

Apache Spark Streaming Overview
Example: Streaming Request Count
DStreams
Developing Streaming Applications
Apache Spark Streaming: Processing Multiple Batches

Multi-Batch Operations
Time Slicing
State Operations
Sliding Window Operations
Preview: Structured Streaming


Apache Spark Streaming: Data Sources

Streaming Data Source Overview
Apache Flume and Apache Kafka Data Sources
Example: Using a Kafka Direct Data Source 

Comments

Post a Comment

Popular posts from this blog

Python Training in Hyderabad

Image
https://nareshit.in/python-training/ http://algorithmtraining.com https://www.pluralsight.com/browse/software-development/python?gclid=CjwKCAjwr-PYBRB8EiwALtjbz9vcNynyvlqfCZZxCRQP_CKHUUCgelXfb0bim268MVILIkS7NdNoyBoChHEQAvD_BwE&utm_term=&aid=7010a000002BPWoAAO&promo=&oid=&utm_source=non_branded&utm_medium=digital_paid_search_google&utm_campaign=US_Courses_Python_BMM&utm_content=&s_kwcid=AL!5668!3!241243437050!b!!g!!%2Bpython%20%2B%20training&ef_id=WuyHPwAAAJJ2RRAl:20180607221245:s Python Training webmaster Contents  [ hide ] 1  Python Training Overview 1.1  What are the Python Course Pre-requisites 1.2  Objectives of the Course 1.3  Who should do the course 1.4  Python Training Course Duration 2  Python Course Content 2.1  Core Python 2.1.1  Introduction to Script 2.1.2  Introduction to Python 2.1.3  Different Modes in PYTHON 2.1.4  PYTHON NEW IDEs 2.1....
Read more

python classes

What is Programming Language? -------------------------- 1.A programming language is a set of rules that provides a way of telling a computer what operations to perform. 2.A programming language is a set of rules for communicating an algorithm 3.It provides a linguistic framework for describing computations provides a way of telling a computer what operations to perform. Types of computer languages: -------------------------- Machine Language :      1.The fundamental language of the computer’s processor, also called Low Level Language.      2.All programs are converted into machine language before they can be executed.      3.Consists of combination of 0’s and 1’s that represent high and low electrical voltage. Assembly Language :      1.A low level language that is similar to machine language.  2.Uses symbolic operation code to represent the machine operation code.       Eg : ...
Read more

FULL STACK Python & Node JS Developer

Position Title: Full Stack Engineer (Python & NodeJS) Location: Mountain View, CA Client is looking for someone with Python and NodeJS experience and has worked on the complex algorithm and/or big data problems. Job overview Lead the development of a customer portal website serving hardware makers. Responsibilities Architect the web front end and implement code in Python, HTML, JavaScript, and CSS that meets the functional specifications. Collaborate with outsourced UI Designer to prototype the user experience during the design phase. Architect and implement database to meet capacity and performance specification. Collaborate with other members of engineering team to implement and optimize various back-end algorithms. Qualifications Bachelor degree in Computer Science or related field. Experience designing and developing web applications integrating various platforms through SOA. Experience designing and building RESTful APIs and micro-services. Ex...
Read more