Logging, environment variables, and config

Your scripts work. Now make them work at 3 a.m. on a server you cannot see. That is what these three habits are for.

Logging instead of print

print() is fine in the playground. In an Airflow task or a Databricks job, you need to know when something happened, how serious it is, and which part of the code said it. That is logging.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger("pipeline")

log.info("loaded %s rows from %s", 120, "offers.csv")
log.warning("3 rows skipped: bad salary")
log.error("API returned 500, retrying")

Levels, from quiet to loud: DEBUG, INFO, WARNING, ERROR, CRITICAL. basicConfig(level=INFO) shows INFO and above and hides DEBUG. In production you flip one setting to see more or less, without touching the code.

Two habits:

• Log counts at each step: rows read, rows written, rows skipped. When the numbers disagree, you have found the bug.
• Pass values as arguments (log.info("rows: %s", n)), not with an f-string. Logging formats the message only if the level is enabled.

Airflow, Databricks, and cloud consoles all collect this output. Without logging, a failed job is a black box. With it, you read the story.

Secrets in environment variables

Never put a password in code. Code goes to GitHub. Code gets pasted into AI assistants. Code is read by everyone on the team.

import os

db_password = os.environ["DB_PASSWORD"]          # error if missing: good, fail fast
db_host = os.environ.get("DB_HOST", "localhost") # default if missing

The variable is set outside the code: in your shell, in a .env file that is git-ignored, in the Airflow connection, in the cloud secret manager.

export DB_PASSWORD="..."         # shell, for one session

For local work, the python-dotenv package reads a .env file into os.environ. Add .env to .gitignore before you create it.

Never log a secret either. Log its length or its last characters if you must confirm it is set.

Config files

Settings that are not secret and change between environments belong in a config file, not in the code: batch sizes, table names, the dev vs prod flag.

# config.yaml
env: prod
batch_size: 500
source_table: raw.offers

import yaml                        # pip install pyyaml

with open("config.yaml") as f:
    config = yaml.safe_load(f)

config["batch_size"]               # 500

YAML is the most common in data tools (Airflow, dbt, Kubernetes). TOML is Python’s own choice (pyproject.toml). JSON works too. All three load into a dict, then it is just dict access.

The professional checklist

Habit	Why
logging, not print	You can see what happened after the fact.
Secrets in os.environ	Nothing sensitive in Git or chat.
Settings in a config file	Change behaviour without changing code.
Git for every project	History, review, rollback.
PEP 8 style, a linter (ruff)	Code looks like everyone else’s.
Tests with pytest	You know it works before it runs at night.

These are not Python skills. They are engineering skills. In data teams they are what separates “writes scripts” from “ships pipelines”.