Going 10x faster with python-oracledb Data Frames – Christopher Jones
Oracle Database users gain significant performance and memory improvements by using python-oracledb’s new dataframe support. Here are some results.
Python-oracledb, the Python driver for Oracle Database, can efficiently select data for use with data frame libraries such as Apache PyArrow, Pandas, Polars, NumPy, PyTorch, or to write files in Apache Parquet format. See the documentation here.
The performance improvements are impressive. From a power user here:
Reporting some results now that I was able to test this out in the wild:
Creating (sparse) numpy ndarrays from DB table, between 2x and 3x faster, gain is both extract and being able to work directly in a more efficient format
Creating (large) numpy recarrays from DB table, between 3x and 10x faster. The larger the data, the larger the benefit (and this is with non zero-copy).
Creating PQ files from RecordBatch is about 3x faster and a lot more memory friendly
In general, this change enabled some serious performance improvements for me.
And another user here says:
I’m seeing, on average a 60% speed up in my extract scripts compared with pd.read_sql, some tables are as much as 78-85% faster.
These are great numbers! Thanks to both these users for sharing their results.
Our (Oracle’s driver team) work to improve data frames in python-oracledb is continuing. Projects to support the Oracle Database 23ai VECTOR data type, and to make it easier to insert data frames into Oracle Database, are under way.
If you’re interested in data frames, also check out my related blog posts python-oracledb 3.0 Data Frames — a new way to query data and Writing to Parquet and Delta Lake files from Oracle Database using Python.
Python-oracledb Resources
Going 10x faster with python-oracledb Data Frames was originally published in Oracle Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.
