≡

Introduction

Why read this handbook?

Interviews for the generative artificial intelligence field aren't standardized like software engineering interviews are. There isn't a bank of questions online you can practice or even an established set of topics to study. This handbook aims to fix that. The exercises here stem from real-world interview questions — with additional exercises that are actually related to your day-to-day job as an AI researcher or engineer.

Part of the problem is how quickly the AI field evolves. New information went stale within months, and as a result, it was unclear what an interview should look like — both from the interviewee's and the interviewer's perspective. What would you ask in an interview, if what's hot today is moot tomorrow?

Now, several years after ChatGPT's debut, the passage of time has introduced some stability. The fundamentals of a transformer have converged to a universally-accepted architecture. The systems required to train and run inference have begun to take shape. Of course, even more exciting developments are on the horizon, but with time, parts of AI have solidified and begun to form the pillars of future developments. As a result of this, AI interview questions have also begun to show familiar, recurring patterns across companies and roles.

To this end, this handbook is a live document. Its current instantiation is just a start, and as more and more of the job description and interview pipeline becomes concrete, this handbook will expand to include more exercises and more documentation — ideally growing into a compilation of exercises that prepare you for both the interviews and also the job.

What do I need to know?

This handbook is geared towards technical roles, specifically researchers and engineers that work directly with a large generative model. This means that you're dealing with training, fine-tuning, or inference on a day-to-day basis. In particular, the explanations are written for coders: I try to explain using code snippets rather than equations.

Prerequisites. I will assume you already know Python, basic linear algebra, and the fundamentals of machine learning. You should know what a matrix multiplication is, the difference between training and inference, what a probability distribution is etc. Just a surface-level understanding suffices, but with that said, the deeper your pre-existing knowledge, the more exercises you'll be able to complete. The hardest practice exercises require deep familiarity with these topics.

💡

Note: Missing any of these pre-requisites? Here are resources to help you with these topics. For Python, I would recommend

For the fundamentals of machine learning, I would recommend

There are also high quality resources for generative AI. Think of these resources as your textbook for all the details, and think of this handbook as your cheat sheet. It's a Sparknotes-style summary and index for other resources across the web — just enough to jog your memory.

Must-know topics. With this handbook, you'll learn several critical topics — how the transformer works at a mechanical level, including the intuition for its current design; how these models are trained, tuned, and served; and different types of interview questions, including a large repository of practice exercises. I've also included a series of exercises purely for the sake of learning — through these, you'll gain a deeper understanding, which can take you from "passing" to "strong hire".