SparkSession

The entry point to programming Spark with the Dataset and DataFrame API. A SparkSession can be used to create DataFrames, register DataFrames as tables, execute SQL over tables, cache tables, and read parquet files.

Syntax

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

Properties

Property Description
builder Interface for building the session configuration.
catalog Interface through which the user may create, drop, alter or query underlying databases, tables, functions, etc.
client Gives access to the Spark Connect client. Spark Connect only.
conf Runtime configuration interface for Spark.
dataSource Returns a DataSourceRegistration for data source registration.
profile Returns a Profile for performance/memory profiling.
read Returns a DataFrameReader that can be used to read data as a DataFrame.
readStream Returns a DataStreamReader that can be used to read data streams as a streaming DataFrame.
sparkContext Returns the underlying SparkContext. Classic mode only.
streams Returns a StreamingQueryManager that allows managing all active streaming queries.
tvf Returns a TableValuedFunction for calling table-valued functions (TVFs).
udf Returns a UDFRegistration for UDF registration.
udtf Returns a UDTFRegistration for UDTF registration.
version The version of Spark on which this application is running.

Methods

Method Description
createDataFrame(data, schema, samplingRatio, verifySchema) Creates a DataFrame from an RDD, a list, a pandas DataFrame, a numpy ndarray, or a pyarrow Table.
sql(sqlQuery, args, **kwargs) Returns a DataFrame representing the result of the given query.
table(tableName) Returns the specified table as a DataFrame.
range(start, end, step, numPartitions) Creates a DataFrame with a single LongType column named id, containing elements in a range.
newSession() Returns a new SparkSession with separate SQLConf, registered temporary views, and UDFs, but shared SparkContext and table cache. Classic mode only.
getActiveSession() Returns the active SparkSession for the current thread.
active() Returns the active or default SparkSession for the current thread.
stop() Stops the underlying SparkContext.
addArtifacts(*path, pyfile, archive, file) Adds artifact(s) to the client session.
interruptAll() Interrupts all operations of this session currently running on the server.
interruptTag(tag) Interrupts all operations of this session with the given tag.
interruptOperation(op_id) Interrupts an operation of this session with the given operationId.
addTag(tag) Adds a tag to be assigned to all operations started by this thread in this session.
removeTag(tag) Removes a tag previously added for operations started by this thread.
getTags() Gets the tags currently set to be assigned to all operations started by this thread.
clearTags() Clears the current thread's operation tags.
  • Last updated on