Types of Financial Data and How to Obtain Them (with Practical Examples)

上一篇：Why Financial Data Cannot Be Modeled with “Standard Machine Learning”

中文版：【模块1 建立认知2】金融数据的类型与获取方式（附实战）

目录

Classification of Financial Data

Common Data Sources

Python Data Acquisition (Practical Guide)

Yahoo Finance

FRED: Federal Reserve Economic Data

Tushare

Data Structure Standardization (Preparing for Modeling)

Summary

Classification of Financial Data

In financial data science, data can generally be understood from two perspectives: data source and modeling purpose. The most fundamental category is market data, which is directly generated from trading activities and serves as the core input for almost all financial analyses. Typical examples include price data (open, close, high, low) as well as trading volume and turnover. These data cover a wide range of assets, such as equities, indices, commodities, and foreign exchange. A key characteristic of market data is its high frequency—usually daily or even intraday—which makes it widely applicable in return modeling, volatility analysis, and trading strategy research.

Another important category is macroeconomic data, which reflects the overall economic environment and plays a crucial role in explaining movements in financial markets. Common indicators include interest rates (such as SHIBOR, LIBOR, or government bond yields), GDP, inflation measures (CPI/PPI), unemployment rates, and exchange rates. Unlike market data, macroeconomic data is typically low-frequency, often published on a monthly or quarterly basis. As a result, it is more suitable for medium- to long-term analysis, such as modeling the relationship between macroeconomic conditions and asset prices.

The third category is derived data, which is not directly observed but constructed by transforming raw data into meaningful features. Examples include log returns, rolling volatility, technical indicators (such as moving averages, RSI, and MACD), and various financial factors (such as size, value, and momentum). Derived data is arguably the most valuable component in financial data science, as models typically rely on these engineered features for training and prediction.

Common Data Sources

In the Chinese market context, if the research involves A-share equities or domestic macroeconomic data, professional data platforms such as Wind and Tonghuashun iFinD can be used. These platforms provide comprehensive and high-quality datasets, but they are usually subscription-based. For an open-source alternative, Tushare is a strong option. It offers a relatively complete set of Chinese financial data interfaces and integrates well with the Python ecosystem, making it particularly suitable for academic research and project-based work.

Python Data Acquisition (Practical Guide)

After understanding the types of financial data, a more practical question arises: how can these data be obtained in practice? For beginners in financial data science, the typical workflow does not start with modeling. Instead, it begins with a structured process: data acquisition, data inspection, data cleaning, and data storage. Only after completing these steps can subsequent tasks—such as return calculation, stationarity testing, volatility modeling, and multivariate analysis—be carried out reliably.

In practice, different data sources correspond to different use cases. For global market assets such as stocks, indices, and ETFs, Yahoo Finance is a highly convenient and free data source. For macroeconomic variables such as interest rates, GDP, or inflation, the FRED database is a reliable and high-quality choice. If the focus is on the Chinese financial market, Tushare is generally more suitable for Python users. The following sections demonstrate how to retrieve these types of financial data using Python.

Yahoo Finance

We begin with market data retrieval using Yahoo Finance. One of its main advantages is its broad coverage and ease of use, making it particularly suitable for teaching demonstrations, coursework, and rapid prototyping of trading strategies. With the help of the yfinance library, we can conveniently download historical price data for a given stock or index over a specified time range.

import yfinance as yf
import pandas as pd
 
# ------------------------------
# Step 1: Define the asset ticker and time range
# ------------------------------
# Here we use Apple Inc. (AAPL) as an example
ticker = "AAPL"
 
# Set the start and end dates
start_date = "2020-01-01"
end_date = "2024-01-01"
 
# ------------------------------
# Step 2: Download historical data using yfinance
# ------------------------------
# The download() function returns a DataFrame
# It typically includes columns such as Open, High, Low, Close, Adj Close, and Volume
df = yf.download(ticker, start=start_date, end=end_date)
 
# ------------------------------
# Step 3: Inspect the basic structure of the data
# ------------------------------
print("First 5 rows of the data:")
print(df.head())
 
print("\nLast 5 rows of the data:")
print(df.tail())
 
print("\nShape of the dataset:")
print(df.shape)
 
print("\nMissing values summary:")
print(df.isna().sum())
 
# ------------------------------
# Step 4: Reset the index to convert the date index into a column
# ------------------------------
# By default, yfinance stores the date as the index
# For consistency in further processing, we convert it back to a regular column
df.reset_index(inplace=True)
 
# ------------------------------
# Step 5: Save the data to an Excel file
# ------------------------------
# index=False ensures that the DataFrame index is not written to the file
output_file = "AAPL_stock_data.xlsx"
df.to_excel(output_file, index=False)
 
print(f"\nData has been saved to: {output_file}")

This piece of code accomplishes a basic yet essential task: downloading historical stock price data from an external data source and saving it locally. For beginners, the key point is not merely to “obtain the data,” but to learn how to inspect its structure. Many people start modeling immediately after downloading data, without first verifying column names, date formats, or missing values. This often leads to errors in subsequent analysis. A well-structured data acquisition workflow should always include checking the first few rows, examining the dataset dimensions, and summarizing missing values.

In addition to stocks, Yahoo Finance also supports indices, ETFs, commodities, and some foreign exchange data. For example, the S&P 500 index can be accessed using ^GSPC, the gold ETF using GLD, and the crude oil ETF using USO. In other words, once you are familiar with this approach, it can be easily extended to a wide range of asset classes.

FRED: Federal Reserve Economic Data

Next, we turn to macroeconomic data acquisition. While financial market prices reflect outcomes, macroeconomic data are often used to explain the underlying economic mechanisms behind those outcomes. For instance, when analyzing stock market fluctuations, changes in interest rates can significantly affect market valuations; when studying exchange rates, indicators such as inflation and economic growth also play an important role. For this type of data, a commonly used and reliable source is FRED, the Federal Reserve Economic Data database.

Using the pandas_datareader library, we can directly retrieve macroeconomic indicators from FRED. For example, the following code demonstrates how to download the Federal Funds Rate (FEDFUNDS).

from pandas_datareader import data as pdr
import pandas as pd
import datetime
 
# ------------------------------
# Step 1: Define the time range
# ------------------------------
start = datetime.datetime(2015, 1, 1)
end = datetime.datetime(2024, 1, 1)
 
# ------------------------------
# Step 2: Retrieve macroeconomic data from FRED
# ------------------------------
# "FEDFUNDS" is the code for the Federal Funds Rate in the FRED database
df_rate = pdr.DataReader("FEDFUNDS", "fred", start, end)
 
# ------------------------------
# Step 3: Inspect the data
# ------------------------------
print("First 5 rows of the Federal Funds Rate data:")
print(df_rate.head())
 
print("\nLast 5 rows of the Federal Funds Rate data:")
print(df_rate.tail())
 
print("\nShape of the dataset:")
print(df_rate.shape)
 
print("\nMissing values summary:")
print(df_rate.isna().sum())
 
# ------------------------------
# Step 4: Reset the index to convert the date index into a column
# ------------------------------
df_rate.reset_index(inplace=True)
 
# Rename columns to more descriptive names
df_rate.columns = ["date", "fed_funds_rate"]
 
# ------------------------------
# Step 5: Save the data to an Excel file
# ------------------------------
output_file = "fed_rate.xlsx"
df_rate.to_excel(output_file, index=False)
 
print(f"\nMacroeconomic data has been saved to: {output_file}")

Compared to market data, macroeconomic data generally has a lower frequency. Some series are available at a daily frequency, while others are published monthly or quarterly. Therefore, after obtaining macroeconomic data, it is not appropriate to immediately merge it with market price data. Instead, one should first understand its release frequency and economic meaning. For example, GDP is typically reported on a quarterly basis, whereas stock prices are usually available daily. Aligning these two types of data requires additional processing. In other words, working with macroeconomic data is not only a technical issue but also involves frequency matching and economic interpretation.

Tushare

If the research focuses on the Chinese market, Tushare is a highly practical tool. It provides strong support for A-share data and also offers access to certain macroeconomic, fund, financial, and index datasets. For students working on academic papers or course projects, Tushare has a significant advantage in its seamless integration with the Python ecosystem. Its unified API design also makes it more suitable for batch data processing compared to manually downloading data from web sources.

The following code example demonstrates how to use Tushare to retrieve daily price data for Kweichow Moutai.

import tushare as ts
import pandas as pd
 
# ------------------------------
# Step 1: Set the Tushare token
# ------------------------------
# Replace this with your own token obtained from the Tushare website
ts.set_token("your_token")
 
# Initialize the pro API interface
pro = ts.pro_api()
 
# ------------------------------
# Step 2: Retrieve daily stock data
# ------------------------------
# ts_code='600519.SH' represents Kweichow Moutai (Shanghai Stock Exchange)
# start_date and end_date should be in YYYYMMDD format
df = pro.daily(
    ts_code='600519.SH',
    start_date='20200101',
    end_date='20240101'
)
 
# ------------------------------
# Step 3: Inspect the raw data
# ------------------------------
print("First 5 rows of raw data:")
print(df.head())
 
print("\nColumn names of raw data:")
print(df.columns.tolist())
 
print("\nShape of the dataset:")
print(df.shape)
 
print("\nMissing values summary:")
print(df.isna().sum())
 
# ------------------------------
# Step 4: Sort data in ascending order by date
# ------------------------------
# Tushare often returns data in descending order by default
# For time series analysis and return calculations, ascending order is required
df = df.sort_values(by="trade_date", ascending=True)
 
# ------------------------------
# Step 5: Rename columns for consistency
# ------------------------------
# Tushare uses different field names compared to Yahoo Finance
# Select commonly used fields and rename them to a standardized format
df = df[['trade_date', 'open', 'high', 'low', 'close', 'vol']]
df.columns = ['date', 'open', 'high', 'low', 'close', 'volume']
 
# ------------------------------
# Step 6: Convert date format
# ------------------------------
# trade_date is originally a string (e.g., 20200102)
# Convert it to datetime for easier filtering, sorting, and visualization
df['date'] = pd.to_datetime(df['date'], format='%Y%m%d')
 
# ------------------------------
# Step 7: Save the processed data to an Excel file
# ------------------------------
output_file = "maotai.xlsx"
df.to_excel(output_file, index=False)
 
print(f"\nProcessed A-share data has been saved to: {output_file}")

This piece of code is closer to a real-world data processing workflow than simply “downloading data.” In practical research, the goal is not just to obtain raw datasets, but to quickly transform data from different sources into a unified format. For example, in Tushare the date column is named trade_date, whereas in Yahoo Finance the date is typically stored as the index; Tushare uses vol to represent trading volume, while Yahoo Finance uses Volume. If these inconsistencies are not addressed, it will become cumbersome to perform multi-asset comparisons or batch processing later on.

To make this section more practical, we provide a more complete example below. Starting from downloading data via Yahoo Finance, we directly perform basic preprocessing steps, including selecting commonly used fields, renaming columns, converting date formats, sorting the data, and saving the results. In this way, the resulting dataset is already in a standardized form that is ready for subsequent analysis.

import yfinance as yf
import pandas as pd
 
# ------------------------------
# Step 1: Download raw stock data
# ------------------------------
ticker = "MSFT"
df = yf.download(ticker, start="2020-01-01", end="2024-01-01")
 
# ------------------------------
# Step 2: Convert the date index into a regular column
# ------------------------------
df.reset_index(inplace=True)
 
# ------------------------------
# Step 3: Keep only the commonly used fields for analysis
# ------------------------------
df = df[['Date', 'Open', 'High', 'Low', 'Close', 'Volume']]
 
# ------------------------------
# Step 4: Standardize column names
# ------------------------------
df.columns = ['date', 'open', 'high', 'low', 'close', 'volume']
 
# ------------------------------
# Step 5: Convert the date column to datetime format
# ------------------------------
df['date'] = pd.to_datetime(df['date'])
 
# ------------------------------
# Step 6: Sort the data in ascending order by date
# ------------------------------
df = df.sort_values(by='date', ascending=True)
 
# ------------------------------
# Step 7: Inspect the processed data
# ------------------------------
print("First 5 rows of the processed data:")
print(df.head())
 
print("\nData types:")
print(df.dtypes)
 
# ------------------------------
# Step 8: Save the processed dataset
# ------------------------------
df.to_excel("MSFT_standardized.xlsx", index=False)
 
print("\nThe standardized dataset has been saved.")

From this process, it becomes clear that financial data acquisition is not an isolated task, but rather the starting point of the entire data analysis pipeline. What truly matters is not simply whether the data can be downloaded, but whether it can be immediately used for analysis after retrieval. Therefore, a well-designed financial data acquisition script should typically accomplish at least four tasks: first, clearly define the data source and time range; second, verify the completeness of the downloaded data; third, standardize column names and date formats; and fourth, save the data in a reusable format for subsequent analysis.

If you want this section to read more like a tutorial within a series blog, you can also include the following transition:

After completing data acquisition, we now have the raw inputs required for modeling. However, raw price data cannot be directly used in most financial time series models. Further steps are required, including return calculation, missing value handling, trading day alignment, and stationarity testing. For this reason, data acquisition is not the endpoint, but rather the true beginning of financial data analysis.

Data Structure Standardization (Preparing for Modeling)

In financial data analysis, obtaining data does not mean you can immediately proceed to modeling. In practice, raw data often comes from different platforms and APIs, with inconsistencies in column naming, date formats, sorting order, and frequency structure. Without proper standardization, subsequent tasks—such as return calculation, visualization, or building models like VAR, GARCH, and DCC—are prone to errors. Therefore, data structure standardization is not an optional step, but one of the most critical preparations before financial modeling.

The essence of data standardization is to transform datasets from different sources and formats into a unified, structured, and reusable form. For financial time series, this process typically involves several key tasks: standardizing column names, unifying date formats, sorting data chronologically, handling missing values, constructing return variables, and aligning trading days in multi-asset analysis. Only after completing these steps can the data truly become suitable input for modeling.

The first issue to address is inconsistent column naming. Different data sources often use different labels for the same concept. For example, the date column may be labeled Date in Yahoo Finance but trade_date in Tushare; trading volume may appear as Volume or vol. If these differences are not resolved, it becomes difficult to reuse analysis code across datasets. Therefore, it is essential to establish a standardized naming convention from the beginning. In practice, it is recommended to adopt concise and consistent field names such as date, open, high, low, close, and volume.

The following code demonstrates how to transform raw market data into a standardized format with consistent column naming:

import pandas as pd
 
# ------------------------------
# Example: Assume we already have a raw DataFrame
# This DataFrame may come from Yahoo Finance or other data sources
# ------------------------------
# Here we assume the original column names are Date, Open, High, Low, Close, Volume
# If your column names are different, simply adjust them during selection and renaming
 
# Read data from an Excel file (can also be replaced with CSV)
df = pd.read_excel("raw_market_data.xlsx")
 
# ------------------------------
# Step 1: Inspect original column names and preview data
# ------------------------------
# Before standardization, always check the structure of the dataset
# This is a critical habit in data analysis
print("Original column names:")
print(df.columns.tolist())
 
print("\nFirst 5 rows of raw data:")
print(df.head())
 
# ------------------------------
# Step 2: Select core fields commonly used in analysis
# ------------------------------
# The purpose is to remove unnecessary columns and reduce noise
# For example, some data sources may include Adjusted Close, Turnover, etc.
# If they are not needed at this stage, they can be excluded
df = df[['Date', 'Open', 'High', 'Low', 'Close', 'Volume']]
 
# ------------------------------
# Step 3: Standardize column names
# ------------------------------
# Convert different naming conventions into a unified format
# This ensures that the same analysis code can be reused across datasets
df.columns = ['date', 'open', 'high', 'low', 'close', 'volume']
 
print("\nStandardized column names:")
print(df.columns.tolist())

After standardizing the column names, the next step is to process the date format. Financial data analysis is almost always time-based, so the date column must be clear, consistent, and correctly recognized by Python. If the date is still stored as a string or has inconsistent formatting, it can cause issues in subsequent operations such as sorting, filtering, visualization, and time alignment. The standard approach is to convert the date column into the datetime type.

import pandas as pd
 
# ------------------------------
# Load the preprocessed dataset
# ------------------------------
df = pd.read_excel("raw_market_data.xlsx")
 
# Standardize column names for consistency
df = df[['Date', 'Open', 'High', 'Low', 'Close', 'Volume']]
df.columns = ['date', 'open', 'high', 'low', 'close', 'volume']
 
# ------------------------------
# Step 1: Convert the date column to datetime format
# ------------------------------
# pd.to_datetime() can automatically recognize common date formats
# If the format is unusual, you can specify the format parameter explicitly
df['date'] = pd.to_datetime(df['date'])
 
# ------------------------------
# Step 2: Check data types after conversion
# ------------------------------
print("Data types of each column:")
print(df.dtypes)
 
# ------------------------------
# Step 3: Sort data in ascending order by date
# ------------------------------
# Financial time series analysis typically requires data to be ordered chronologically
# Otherwise, calculations such as returns, rolling statistics, or lagged variables may be incorrect
df = df.sort_values(by='date', ascending=True)
 
# ------------------------------
# Step 4: Reset the index
# ------------------------------
# After sorting, the original index may become non-sequential
# Resetting the index ensures a clean and consistent structure
df = df.reset_index(drop=True)
 
print("\nFirst 5 rows after processing:")
print(df.head())

After standardizing the date format and column names, the next step is to check for missing values. Missing data is very common in financial datasets and may arise from non-trading days, trading suspensions, API retrieval issues, or misalignment when merging multiple data sources. If missing values are not properly handled, many models may fail to run or, more subtly, introduce bias into the results. Therefore, it is essential to first assess the extent of missing data and then choose appropriate handling methods based on the characteristics of the dataset before proceeding to modeling.

import pandas as pd
 
# ------------------------------
# Assume df has already been standardized in terms of column names and date format
# ------------------------------
df = pd.read_excel("standardized_market_data.xlsx")
 
# ------------------------------
# Step 1: Count missing values in each column
# ------------------------------
print("Missing values count for each column:")
print(df.isna().sum())
 
# ------------------------------
# Step 2: Identify rows containing missing values
# ------------------------------
# This helps to examine where missing values occur (dates and variables)
missing_rows = df[df.isna().any(axis=1)]
print("\nRows with missing values:")
print(missing_rows)
 
# ------------------------------
# Step 3: Handle missing values
# ------------------------------
# The appropriate method depends on the specific context
# Here are several common approaches
 
# Option 1: Drop rows with missing values
df_dropna = df.dropna()
 
# Option 2: Forward fill using the previous valid observation
# Suitable for certain continuous time series
df_ffill = df.fillna(method='ffill')
 
# Option 3: Linear interpolation
# Suitable for relatively smooth series, but should be used cautiously for price data
df_interp = df.interpolate(method='linear')
 
print("\nShape after dropping missing values:", df_dropna.shape)
print("Shape after forward fill:", df_ffill.shape)
print("Shape after interpolation:", df_interp.shape)

In financial research, handling missing values is not about making the data as “smooth” as possible, but about respecting the nature of the data itself. For example, missing stock prices on non-trading days are normal, and it would be inappropriate to mechanically fill these gaps with interpolated values, as this would create “artificial prices.” In contrast, for low-frequency macroeconomic data with occasional missing observations, interpolation may sometimes be acceptable. Therefore, data standardization is not purely a technical task; it also requires an understanding of the economic meaning behind the data.

Once the raw price data has been properly cleaned, it is usually necessary to construct return variables. In most financial models, the primary object of analysis is not the price level itself, but the rate of change in prices. Price series often exhibit trends and non-stationarity, whereas returns are typically closer to a stationary process and are therefore more suitable for statistical modeling. In particular, in return modeling, volatility modeling, and correlation analysis, log returns are the most commonly used form.

import pandas as pd
import numpy as np
 
# ------------------------------
# Load standardized price data
# ------------------------------
df = pd.read_excel("standardized_market_data.xlsx")
 
# Ensure the date column is in datetime format and sorted in ascending order
df['date'] = pd.to_datetime(df['date'])
df = df.sort_values(by='date', ascending=True).reset_index(drop=True)
 
# ------------------------------
# Step 1: Calculate simple returns
# ------------------------------
# Simple return = (current price - previous price) / previous price
# Useful for intuitive interpretation of price changes
df['simple_return'] = df['close'].pct_change()
 
# ------------------------------
# Step 2: Calculate log returns
# ------------------------------
# Log return = ln(Pt / Pt-1)
# More commonly used in financial modeling due to time additivity
df['log_return'] = np.log(df['close'] / df['close'].shift(1))
 
# ------------------------------
# Step 3: Inspect the results
# ------------------------------
print("First 5 rows with returns:")
print(df.head())
 
# ------------------------------
# Step 4: Remove the first row with missing values
# ------------------------------
# The first observation has no previous price, so returns are NaN
df = df.dropna(subset=['simple_return', 'log_return']).reset_index(drop=True)
 
print("\nFirst 5 rows after removing NaN values:")
print(df.head())
 
# ------------------------------
# Step 5: Save the processed dataset
# ------------------------------
df.to_excel("market_data_with_returns.xlsx", index=False)
print("\nDataset with returns has been saved.")

In single-asset analysis, at this stage it is usually possible to proceed to the next step of modeling. However, if the research involves multiple assets—such as analyzing the interactions among oil prices, exchange rates, and gold—an additional issue must be addressed: time alignment across different assets. Since trading calendars are not perfectly synchronized across markets, directly merging multiple time series will result in a large number of missing values and may even lead to misaligned model inputs. Therefore, before conducting multi-asset modeling, it is essential to align the data using common trading dates.

The following code example demonstrates how to merge two asset datasets based on shared trading days:

import pandas as pd
 
# ------------------------------
# Load two standardized datasets
# ------------------------------
# Assume df1 contains oil data and df2 contains gold data
df1 = pd.read_excel("oil_data_standardized.xlsx")
df2 = pd.read_excel("gold_data_standardized.xlsx")
 
# ------------------------------
# Step 1: Standardize date format
# ------------------------------
df1['date'] = pd.to_datetime(df1['date'])
df2['date'] = pd.to_datetime(df2['date'])
 
# ------------------------------
# Step 2: Sort data by date
# ------------------------------
df1 = df1.sort_values(by='date').reset_index(drop=True)
df2 = df2.sort_values(by='date').reset_index(drop=True)
 
# ------------------------------
# Step 3: Rename variables to avoid column name conflicts
# ------------------------------
# If both datasets contain columns like 'close' or 'return',
# merging directly will cause duplicate column names
# Therefore, add asset-specific prefixes before merging
df1 = df1[['date', 'close', 'log_return']].copy()
df1.columns = ['date', 'oil_close', 'oil_return']
 
df2 = df2[['date', 'close', 'log_return']].copy()
df2.columns = ['date', 'gold_close', 'gold_return']
 
# ------------------------------
# Step 4: Perform inner join on common dates
# ------------------------------
# how='inner' means keeping only the dates that exist in both datasets
# This is the most appropriate method for multi-variable financial modeling
df_merged = pd.merge(df1, df2, on='date', how='inner')
 
# ------------------------------
# Step 5: Inspect the merged dataset
# ------------------------------
print("First 5 rows after merging:")
print(df_merged.head())
 
print("\nShape of the merged dataset:")
print(df_merged.shape)
 
print("\nMissing values summary:")
print(df_merged.isna().sum())
 
# ------------------------------
# Step 6: Save the merged dataset
# ------------------------------
df_merged.to_excel("merged_oil_gold_data.xlsx", index=False)
print("\nThe aligned multi-asset dataset has been saved.")

If the analysis involves more assets—for example, oil, exchange rates, and gold simultaneously—the procedure remains the same: first standardize each dataset individually, then unify date formats and variable names, and finally merge them step by step based on common dates. This step is particularly critical for multivariate methods such as VAR, Granger causality tests, and DCC models, since these models require that observations at each time point are strictly aligned across all variables.

To ensure that this step truly serves as preparation for modeling, real-world projects usually include a final validation stage. This involves checking the integrity of the processed data, such as verifying whether duplicate dates exist, whether missing values remain, and whether returns contain infinite or abnormal values. Performing these checks helps identify potential issues early, before entering the modeling stage.

import pandas as pd
import numpy as np
 
# ------------------------------
# Load the final processed dataset
# ------------------------------
df = pd.read_excel("market_data_with_returns.xlsx")
 
# ------------------------------
# Step 1: Check for duplicate dates
# ------------------------------
duplicate_dates = df['date'].duplicated().sum()
print("Number of duplicate dates:", duplicate_dates)
 
# ------------------------------
# Step 2: Check for remaining missing values
# ------------------------------
print("\nMissing values summary:")
print(df.isna().sum())
 
# ------------------------------
# Step 3: Check for infinite values
# ------------------------------
# In extreme cases, return calculations may produce inf or -inf
inf_count = np.isinf(df.select_dtypes(include=[np.number])).sum().sum()
print("\nNumber of infinite values:", inf_count)
 
# ------------------------------
# Step 4: Descriptive statistics
# ------------------------------
# Provides a quick overview of the distribution of numerical variables
print("\nDescriptive statistics of numerical variables:")
print(df.describe())
 
# ------------------------------
# Step 5: Remove infinite values if necessary
# ------------------------------
df = df.replace([np.inf, -np.inf], np.nan)
df = df.dropna().reset_index(drop=True)
 
print("\nShape of dataset after cleaning:", df.shape)

From a more fundamental perspective, the purpose of data structure standardization is not merely to “make the table look cleaner,” but to establish a unified data interface for the entire modeling pipeline. Only when column names are consistent, date formats are standardized, time ordering is correct, missing values are properly controlled, return variables are constructed, and multi-asset series are accurately aligned can the outputs of subsequent models be considered reliable. Otherwise, even if the code runs successfully, the results may lack genuine statistical validity.

Therefore, in financial data analysis, standardization is not a mechanical preprocessing step, but rather a bridge connecting raw data to formal modeling. In many cases, poor model performance is not due to insufficient model complexity, but because the input data has not been properly structured. For beginners, the most important skill to develop is not mastering advanced models first, but learning how to transform raw data into a form that models can genuinely utilize.

Summary

The core of financial data processing does not lie in the volume of data, but in the consistency of its structure and the effectiveness of its features. Market data provides direct price information, macroeconomic data offers contextual explanations, and derived data forms the core inputs for modeling. In practice, data acquisition is only the first step; the more critical tasks are data standardization and feature construction. Only by establishing a well-structured data processing pipeline can subsequent analyses—such as return modeling, volatility modeling, and multivariate models (e.g., VAR, GARCH, DCC)—be carried out effectively.

For further exploration, one can build upon the processed data in this chapter to conduct stationarity tests (such as ADF and KPSS) and proceed to return modeling. This represents a natural extension of financial time series analysis.