IntermediatePython · Lesson 6 of 9

Data Analysis with pandas

Load, filter, group, join and summarise tabular data with DataFrames.

pandas is Python's standard library for tabular data. A DataFrame is a table with named columns; each column is a Series. pd.read_csv loads a file (or text), df.head() shows the first rows and df.describe() summarises numeric columns.

Select columns with df["score"] or df[["name", "score"]], filter rows with a condition — df[df["score"] >= 30] — and add columns with vectorised arithmetic, which is far faster than Python loops.

groupby + agg summarises per group (average per subject), merge joins tables like SQL, and pivot_table reshapes data into report form. Install with pip install pandas. (pandas isn't available in the in-browser runner, so run these examples locally.)

TerminalShell
pip install pandas
analysis.pyPython
from io import StringIO

import pandas as pd

results_csv = StringIO("""name,form,subject,score
Amina,4,Maths,88
Amina,4,Biology,79
Juma,4,Maths,29
Juma,4,Biology,48
Neema,4,Maths,71
Neema,4,Biology,84
Ali,3,Maths,95
Ali,3,Biology,62
""")
results = pd.read_csv(results_csv)
fees = pd.DataFrame({"name": ["Amina", "Juma", "Neema", "Ali"], "balance": [0, 200000, 50000, 0]})

print(results.head(3))
print(results["score"].describe().round(1).to_dict())

passed = results[results["score"] >= 30]
print(f"Passed {len(passed)} of {len(results)} papers")

results["grade"] = pd.cut(results["score"], bins=[-1, 29, 44, 64, 74, 100], labels=list("FDCBA"))

per_subject = results.groupby("subject")["score"].agg(["mean", "min", "max"]).round(1)
print(per_subject)

report = results.pivot_table(index="name", columns="subject", values="score")
report["average"] = report.mean(axis=1).round(1)
report = report.merge(fees, left_index=True, right_on="name").set_index("name")
print(report.sort_values("average", ascending=False))

Key points

  • A DataFrame is a table of named columns; read_csv, head and describe get you started.
  • Filter with boolean conditions and compute with whole-column arithmetic, not loops.
  • groupby/agg, merge and pivot_table cover most reporting needs.

Exercise

Using the results data above, find each student's best subject, the pass rate (score ≥ 30) per subject as a percentage, and the number of each grade per form. Save the per-student report to report.csv with to_csv.

Show solution

Try the exercise yourself first — then compare your approach with this one.

idxmax on the pivot table gives each student's best subject. The pass rate is the mean of a True/False column, times 100. crosstab counts grades per form, and to_csv saves the report.

pandas_solution.pyPython
from io import StringIO

import pandas as pd

results = pd.read_csv(StringIO("""name,form,subject,score
Amina,4,Maths,88
Amina,4,Biology,79
Juma,4,Maths,29
Juma,4,Biology,48
Neema,4,Maths,71
Neema,4,Biology,84
Ali,3,Maths,95
Ali,3,Biology,62
"""))
results["grade"] = pd.cut(results["score"], bins=[-1, 29, 44, 64, 74, 100], labels=list("FDCBA"))

scores = results.pivot_table(index="name", columns="subject", values="score")
best = scores.idxmax(axis=1)
print(best.to_dict())                     # {'Ali': 'Maths', 'Amina': 'Maths', 'Juma': 'Biology', 'Neema': 'Biology'}

pass_rate = (results["score"] >= 30).groupby(results["subject"]).mean().mul(100).round(1)
print(pass_rate.to_dict())                # {'Biology': 100.0, 'Maths': 75.0}

print(pd.crosstab(results["form"], results["grade"]))

report = scores.assign(average=scores.mean(axis=1).round(1), best_subject=best)
report.to_csv("report.csv")
print(open("report.csv", encoding="utf-8").read())

Check your understanding

  1. How do you keep only rows where score is at least 30?

  2. What does df.groupby("subject")["score"].mean() give?

  3. Why prefer df["score"] * 2 over a Python loop over rows?

  4. Which joins two DataFrames on a shared column, like SQL JOIN?

Ask AI