Interacting With HDFS from PySpark

One often needs to perform HDFS operations from a Spark application, be it to list files in HDFS or delete data. Because accomplishing this is not immediately obvious with the Python Spark API (PySpark), a few ways to execute such commands are presented below.

Using the Java Gateway

Even with Python applications, Spark relies on the JVM, using Py4J to execute Python code that can interface with JVM objects. Py4J uses a gateway between the JVM and the Python interpreter, which is accessible from your application’s SparkContext (sc below) object:

######
# Get fs handler from java gateway
######
URI = sc._gateway.jvm.java.net.URI
Path = sc._gateway.jvm.org.apache.hadoop.fs.Path
FileSystem = sc._gateway.jvm.org.apache.hadoop.fs.FileSystem
fs = FileSystem.get(URI("hdfs://somehost:8020"), sc._jsc.hadoopConfiguration())

# We can now use the Hadoop FileSystem API (https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html)
fs.listStatus(Path('/user/hive/warehouse'))
# or
fs.delete(Path('some_path'))

While this strategy doesn’t look too elegant, it is useful as it does not require any third party libraries.

Third party libraries

If using external libraries is not an issue, another way to interact with HDFS from PySpark is by simply using a raw Python library. Examples are the hdfs lib, or snakebite from Spotify:

from hdfs import Config

# The following assumes you have hdfscli.cfg file defining a 'dev' client.
client = Config().get_client('dev')
files = client.list('the_dir_path')

or

from snakebite.client import Client

client = Client(hdfs_hostname, hdfs_port)
client.delete('/some-path', recurse=True)

Shell subprocesses

For completion’s sake, this section shows how to accomplish HDFS interaction directly through the subprocess Python facilities, which allows Python to call arbitrary shell commands.

import subprocess

cmd = 'hdfs dfs -ls /user/path'.split() # cmd must be an array of arguments
files = subprocess.check_output(cmd).strip().split('\n')
for path in files:
  print (path)
Diogo Franco

Diogo Franco

I work in Big Data at Farfetch, and I love data, distributed systems, machine learning, code and science!