Linear Algebra
Course overview
Linear algebra is the language for representing directions, organizing data, and transforming batches of numbers. Features, model parameters, embeddings, retrieval scores, and batched computations can all be represented as vectors, matrices, or higher-dimensional tensors. This path is designed for genuine beginners: it builds intuition for dot products and distance first, then covers shapes, spaces, linear layers, projection, and numerical stability, and finally verifies least-squares geometry with a tested standard-library project.
Where this fits in the overall path
Linear Algebra sits in the Mathematics and data foundations stage. It is a shared prerequisite for vector fundamentals, machine learning, deep learning, and embeddings. It explains how data and parameters are organized, which directions are independent, and how matrices transform information in batches.
Learning objectives
- Calculate dot products, norms, distances, and cosine similarity by hand, and explain their semantic limits.
- Read vectors, matrices, shapes, and matrix multiplication.
- Use linear combinations, column spaces, and rank to explain whether a system is solvable.
- Understand in neural networks as a spatial transformation.
- Understand least squares through projection, while distinguishing algebraic fitting from statistical conclusions.
- Use condition numbers and singular values to recognize ill-posed problems, and understand the limits of eigenvalues, SVD, and dimensionality reduction.
Prerequisites
Basic arithmetic, square roots, and one-variable linear equations are enough. The code exercises require lists, functions, and exceptions from Python Fundamentals. You do not need calculus first. Vector-retrieval indexes, thresholds, and evaluation are covered next in Vector Fundamentals.
Recommended order
- Coordinates, dot products, norms, and similarity: begin with coordinate semantics and vector geometry.
- Vectors, matrices, and shapes: build a shape ledger and distinguish matrix multiplication from elementwise operations.
- Linear combinations, systems, and rank: use column spaces, null spaces, bases, and rank to reason about information and solutions.
- Linear transformations and neural networks: understand , batch-direction conventions, and the role of nonlinearity.
- Orthogonality, projection, and least squares: understand fitting, residuals, and normal equations geometrically.
- Numerical stability and condition numbers: distinguish singularity, ill-conditioning, and algorithmic instability.
- Eigenvalues, SVD, and the least-squares project: understand principal directions and dimensionality reduction, then complete a tested least-squares project.
Hands-on entry points
- Implement and test dot products, norms, and cosine similarity in Coordinates, dot products, norms, and similarity.
- Keep a shape ledger for every formula: decide first whether an operation is valid, then what it means.
- Continue to Eigenvalues, SVD, and the least-squares project, run the standard-library least-squares program and its eight tests, verify residual orthogonality, and experiment with rank deficiency, ill-conditioning, and outliers.
Mastery criteria
- I can write vector and matrix shapes and check that inner dimensions agree before computing.
- I can explain matrix multiplication as linear combinations rather than elementwise multiplication.
- I can use rank to identify redundant features or a non-unique solution.
- I can explain why least-squares residuals are orthogonal to the feature space through projection.
- I can distinguish singularity, ill-conditioning, and algorithmic instability, and explain why explicitly inverting a matrix is not the default.
- I can explain the use and limits of eigenvalues, singular values, and SVD in dimensionality reduction.
- I can fit a one-variable linear model with the standard library, explain its input contract, and verify it in normal mode, with
-O, and with unit tests.
Connections to other knowledge bases
| Knowledge base | Connection |
|---|---|
| Vector Fundamentals, Embeddings | Apply dot products, norms, normalization, and batched matrix operations to semantic vectors. |
| Semantic Search | Similarity, dimensionality reduction, and index thresholds depend on vector geometry and evaluation with real data. |
| Calculus Fundamentals | Gradients determine how matrix parameters are updated; the chain rule runs through composed transformations. |
| Machine Learning | Least squares, rank, condition numbers, regularization, and PCA are foundational tools. |
| Deep Learning | Linear layers, embedding tables, and attention all rely on matrix and tensor operations. |
Primary references
- MIT OpenCourseWare: 18.06SC Linear Algebra
- MIT 18.06SC Syllabus
- NumPy: Linear algebra routines
- NumPy:
linalg.lstsq - PyTorch:
torch.nn.Linear
Verified on 2026-07-14. The MIT course covers stable mathematical concepts. NumPy and PyTorch are changing engineering interfaces; this path records only the semantics verified on that date. Consult current official documentation and record versions before use. The local project requires only the Python 3.11.9 standard library; it does not require NumPy or PyTorch.