Introduction:
Hi! today we are going to talk about a different and simple topic which I encountered recently in an interview. The topic is csv. As a data scientist, we use pandas daily to read csv files using pd.read_csv() functionality. And that is where the csv turns into a dataframe for us and we start data manipulation with the dataframe. But a csv is not a dataframe, and doesn't need pandas to be read. In this article, we will discuss a few ways to read a csv file, and will also take a deeper dive in how the csv and similar files are actually stored.
What is csv?
A csv or comma separated value file is a file where each data item is stored with comma separating one value from another. csv is the universally accepted format for tabular data. For a normal csv, each row is stored as a separate new line in the file; and each row contains each values separated by comma, but the row doesn't end with a comma.
How can we read a csv?
A csv can be read using pandas ( read_csv), csv package, and simple i/o file reading functionality as well. Let's get into a bit details of each processes.
for simply reading a csv file, we can use the following pandas code:
import pandas as pd
data = pd.read_csv(csv_file_path)
for using csv package, we can use the following code:
import csv
with open(filepath,'r') as file:
reader = csv.reader(file)
for row in reader:
print(row)
for using file i/o functionality, you will have to be a bit tricky. Here is a function I have written which returns a list of list for the csv file read. The list of list means that it is a list of all rows, and each row is also provided as a list of its elements.
def read_csv(filepath,where_to_start=0,how_many_lines = None):
"""
this function reads a csv file.
how_many_lines take number of lines to read from the beginning
or where_to_start.
if none is provided, then read the full file.
returns a list of list format of the read file.
"""
if how_many_lines is None:
f = open(filepath,mode = 'r')
all_of_it = f.read()
f.close()
list_of_lists = [line.split(',') for line in all_of_it.split('\n')]
else:
list_of_lists = []
f.seek(where_to_start,0)
for i in range(1,how_many_lines):
list_of_lists.append(f.readline().strip('\n').split(','))
f.close()
return list_of_lists
Now, let's discuss a few further questions which come up along with csv. For example,
what is faster to save? csv or text?
this question depends on a few aspects. But the main core thing is same. The core thing is the fact that a csv is nothing but a comma separated text file only. So saving it as a text file or csv file should take ideally same time. But in few cases, you may have to transform your data to save it as a text and thus end up wasting more time. Also another thing which may happen is that a csv can be saved from a complex object such as dataframe; which may take more time than saving a string into text of comparable size.
Important thing to consider if you are taking forever to save a file is that maybe you need some other format to save the file. Consider taking a look into hdf5 or parquet or feather formats. These are suitable for saving bigger data and also for longer storage of data.
what is faster? csv or xlsx?
xlsx type files are in the other hand, stored in very different way and therefore are much heavier than csv. So if the same file is saved as csv and xlsx, it may take more time in case of xlsx. Also, xlsx format can save much different things which csv can't such as multiple sheets, visualizations and other things.
Conclusion:
So in this article, we discussed about csv file format, saw a few ways to read it with or without pandas; and finally compared its speed of saving with text and xlsx format. Please comment and let us know what other questions regarding csv can be important and you would like us to answer.
Comments
Post a Comment