--- tags: - devops - l1 - flashcard-deck - ai-devops-tools --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [AI Tools for DevOps](../../../../library/portal/topics.md) | **Domain:** DevOps & Tooling id category difficulty tags question answer source_path generativeai/18fed098d703 generativeai medium generativeai, llm, embeddings Mathematical Intuition of Attention Block? Each input is projected into three vectors: **Query (Q)**, **Key (K)**, and **Value (V)**. The attention score is computed as:\n\n`Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V`\n\nThe dot product of Q and K measures relevance between tokens, scaling by `sqrt(d_k)` prevents large values from dominating softmax, and the result weights the V vectors to produce context-aware outputs. **Multi-head attention** runs multiple independent attention functions in parallel and concatenates their outputs, allowing the model to attend to different representation subspaces simultaneously. projects/knowledge/interview/generativeai/001-mathematical-intuition-of-attention-block.txt generativeai/37f6ac6c3d46 generativeai medium generativeai, embeddings, evaluation Advanced Explanation of FAISS? FAISS indexes large sets of dense vectors for fast similarity search. Key components:\n\n- **Distance metrics**: L2 (Euclidean) or inner product (cosine similarity when normalized)\n- **Flat Index**: Brute-force exact search — accurate but slow for large datasets\n- **IVF (Inverted File Index)**: Partitions vectors into clusters; searches only nearest clusters for speed\n- **Product Quantization (PQ)**: Compresses vectors into compact codes to reduce memory; approximates distances\n- **HNSW**: Graph-based index for fast approximate nearest neighbor search\n\ projects/knowledge/interview/generativeai/015-advanced-explanation-of-faiss.txt generativeai/3fc8f8a47ceb generativeai medium generativeai, rag, embeddings What are vector databases and what problems do they solve? Vector databases store high-dimensional embedding vectors and are optimized for similarity search (nearest neighbor retrieval), unlike traditional databases which handle structured queries and exact matches.\n\nKey concepts:\n- **ANN search**: Approximate nearest neighbor using LSH, HNSW graphs, or IVF to trade some accuracy for speed\n- **Dimensionality handling**: Specialized indexes (HNSW, PQ) plus optional PCA/t-SNE reduction\n- **Popular systems**: Pinecone (managed), Milvus (open-source, scalable), Weaviate (GraphQL API)\ projects/knowledge/interview/generativeai/009-vector-databases.txt generativeai/499b0955c466 generativeai medium generativeai, embeddings, evaluation Generative AI Fundamentals? Key fundamentals of Generative AI:\n\n- **Discriminative vs Generative**: Discriminative models learn decision boundaries; generative models learn data distributions to create new samples\n- **Latent space**: Lower-dimensional encoding of data features that enables meaningful sample generation\n- **Evaluation metrics**: Inception Score (quality/diversity), FID (statistical similarity to real data), human evaluation\n- **Mode collapse** (GANs): Mitigated with mini-batch discrimination, spectral normalization, WGAN-GP loss\ projects/knowledge/interview/generativeai/010-generative-ai-fundamentals.txt generativeai/5da9dea375c4 generativeai easy generativeai, embeddings, llm Beginner Explanation? "FAISS is a library that allows you to quickly find similar items in a large dataset of vectors. For example, if you have a sentence embedding vector for the query ""I like to play football"", FAISS can efficiently search through millions or billions of other sentence embedding vectors to find the ones that are most similar.\n\nTo use FAISS, you first need to create an index from your dataset of vectors. This involves some preprocessing to optimize the index for fast similarity search." projects/knowledge/interview/generativeai/003-beginner-explanation.txt generativeai/68192aaf716f generativeai hard generativeai, llm, embeddings The Transformer Architecture? "Introduced in ""Attention Is All You Need"" (2017), the Transformer eliminates recurrence in favor of **self-attention**, processing all tokens in parallel.\n\nKey components:\n- **Self-attention**: Each token attends to all others, capturing dependencies regardless of distance\n- **Positional encoding**: Sinusoidal or learned embeddings inject word-order information\n- **Encoder-decoder structure**: Encoder processes input; decoder generates output; both use self-attention + feed-forward layers\n\" projects/knowledge/interview/generativeai/007-the-transformer-architecture.txt generativeai/6ad57f75470e generativeai easy generativeai, fine-tuning, llm Learning the Data Distribution? Generative models learn the probability distribution of the training data. This allows them to generate new samples that are statistically similar to the original data[2].\n\nAnalogy: Like a music student who studies thousands of songs until they can compose new melodies that sound right — they learned the distribution of notes and rhythms.\n\nRemember: Generative models learn P(data) — the probability distribution. Discriminative models learn P(label|data) — the decision boundary. projects/knowledge/interview/generativeai/011-learning-the-data-distribution.txt generativeai/70fb19ff313b generativeai easy generativeai, llm, prompting Evolution of Next Word Prediction Models? Before Transformers, sequence models evolved through:\n\n- **RNNs**: Process sequences via hidden states but struggle with long-range dependencies (vanishing gradients)\n- **LSTMs**: Add memory cells to retain information over longer sequences, solving the vanishing gradient problem\n- **GRUs**: Simplify LSTMs by merging cell/hidden state — more efficient with similar long-range capability\n\nAll three are limited by sequential processing, preventing parallelization and limiting scalability. projects/knowledge/interview/generativeai/006-evolution-of-next-word-prediction-models.txt generativeai/accdcc359b6c generativeai easy generativeai, llm, prompting Sampling from the Learned Distribution? Once the model has learned the data distribution, it can sample from this distribution to generate new samples. This sampling process introduces randomness, which allows the model to produce varied outputs for the same input[1]. projects/knowledge/interview/generativeai/012-sampling-from-the-learned-distribution.txt generativeai/ba86be4a6362 generativeai hard generativeai, embeddings, llm FAISS and Its Applications? FAISS (Facebook AI Similarity Search) is a library for efficient similarity search and clustering of dense vectors.\n\nKey points:\n- **Index types**: Flat (exact), IVFFlat (approximate with clusters), HNSW (graph-based), PQ (compressed)\n- **ANN search**: Limits search to nearby clusters/graph neighbors for speed, trading some accuracy\n- **Key parameters**: `nlist` (number of clusters), `nprobe` (clusters searched per query) — tune for speed/accuracy tradeoff\n- **GPU support**: FAISS can run on NVIDIA GPUs for significant speedup\ projects/knowledge/interview/generativeai/002-faiss-and-its-applications.txt generativeai/c1b299d770d5 generativeai medium generativeai, embeddings, llm Intermediate Explanation? FAISS searches a large vector dataset (e.g., 1 billion embeddings) efficiently:\n1. **Preprocessing**: Builds an index by clustering vectors and encoding them with product quantization to reduce memory\n2. **Searching**: For a query vector, identifies the nearest clusters first, then only compares within those clusters\n3. **Ranking**: Returns the top-k most similar vectors by score\n\nOptimized with multi-threading and GPU acceleration for fast search even at billion-vector scale. projects/knowledge/interview/generativeai/004-intermediate-explanation.txt generativeai/c5ba9c03cdee generativeai hard generativeai, llm, embeddings Transformer Architectures? Key Transformer architecture concepts:\n\n- **Components**: Encoder/decoder layers, multi-head attention, feed-forward networks, layer norm + residual connections\n- **Self-attention**: Weighted sum of values based on query-key compatibility; captures long-range dependencies\n- **Positional encoding**: Sinusoidal or learned embeddings to inject sequence order\n- **Architecture variants**: Encoder-only (BERT — understanding), Decoder-only (GPT — generation), Encoder-decoder (T5 — seq2seq)\n- **Variable-length input**: Handled via padding tokens and attention masks\ projects/knowledge/interview/generativeai/008-transformer-architectures.txt generativeai/dce734b53e4e generativeai medium generativeai, fine-tuning, llm Variational Autoencoders (VAEs)? Variational Autoencoders (VAEs) learn a latent representation of data and use it to generate new samples. They are trained to maximize the likelihood of training data under the learned generative model. Unlike GANs, VAEs provide an explicit probabilistic framework with an encoder (data to latent space) and decoder (latent space to data). projects/knowledge/interview/generativeai/014-variational-autoencoders-vaes.txt generativeai/e8d71f791fc6 generativeai medium generativeai, fine-tuning, llm Adversarial Training (GANs)? One popular type of generative model is the Generative Adversarial Network (GAN). GANs consist of two neural networks - a generator and a discriminator. The generator generates new samples, while the discriminator tries to distinguish between real and generated samples. Through this adversarial training process, the generator learns to produce more realistic samples that can fool the discriminator[2]. projects/knowledge/interview/generativeai/013-adversarial-training-gans.txt generativeai/f41c35dc8f18 generativeai hard generativeai, rag, embeddings Advanced Explanation? FAISS uses advanced indexing for efficient similarity search:\n\n- **IVF (Inverted File Index)**: Partitions vector space into Voronoi cells; narrows search to nearest cells\n- **Product Quantization (PQ)**: Decomposes vectors into subvectors, quantizes each separately for compact RAM storage\n- **HNSW graph**: Multi-layer navigable small-world graph for fast traversal to nearest neighbors\n\nThe most accurate combination is IVF+PQ. These techniques enable state-of-the-art similarity search for semantic search, recommendations, and content retrieval. projects/knowledge/interview/generativeai/005-advanced-explanation.txt --- ## Wiki Navigation ### Related Content - [AI Tools for DevOps](../../../../library/topics/ai-devops-tools/index.md) (Topic Pack, L1) — AI Tools for DevOps - [AI-Assisted DevOps Cookbook](../../../../library/guides/ai-devops-cookbook.md) (Reference, L1) — AI Tools for DevOps - [The Ops of AI/ML Workloads](../../../../library/topics/ai-ml-ops/index.md) (Topic Pack, L2) — AI Tools for DevOps