Google

PART 1: ALGORITHMIC PROBLEM - CONCURRENT WEB CRAWLER BASE PROBLEM You are building a web crawler to index pages for a search engine. You are given a starting URL and a helper function fetch_urls(url) that returns a list of all hyperlinks found on that webpage. Task: Write an algorithm to crawl all reachable URLs within the same domain as the starting URL. You must return a list of all unique URLs visited. How do you prevent infinite loops (cycles) if page A links to page B, and page B links back to page A? FOLLOW-UP 1 Your crawler is working, but it's processing URLs sequentially, which is too slow. You decide to introduce multi-threading to fetch multiple URLs at the same time. However, site administrators are complaining that your crawler is overwhelming their servers with hundreds of concurrent requests. Task: Introduce a "Politeness Policy". You must ensure that your crawler never makes more than one request to the same domain per second. How do you design your URL processing queue and manage your worker threads so that if a domain is "locked" for the next second, your threads don't just sit idle, but actively pick up URLs from other domains? FOLLOW-UP 2 The internet is massive. A single machine, even with multi-threading, cannot crawl it all. You are now scaling your crawler to a distributed cluster of 1,000 machines. Task: How do you manage the "URL Frontier" (the queue of URLs to be crawled) and the "Visited Set" globally? Discuss the trade-offs of using a centralized database versus partitioning the URLs across the 1,000 machines (e.g., using consistent hashing on the domain name). Furthermore, storing billions of visited URLs in memory is impossible; explain how you would use a 'Bloom Filter' to efficiently check if a URL has been crawled before, and what the inherent trade-off of a Bloom Filter is. PART 2: AI DISCUSSION & TERMINOLOGY Practical Application & Infrastructure: If you use an AI coding assistant to generate Infrastructure-as-Code (like Terraform or AWS CloudFormation scripts) to deploy your company's servers, what are the massive security and financial risks involved, and how do you review the AI's output to prevent them? General Knowledge: What is "Model Collapse" (or AI Inbreeding)? Why are AI researchers concerned about training future models as the internet becomes increasingly filled with AI-generated content? Basic Terminologies (Briefly explain the following concepts): Catastrophic Forgetting: A phenomenon where an AI model completely loses its ability to perform a previously learned task after being fine-tuned on new, unrelated data. Its new neural weights "overwrite" the old knowledge too aggressively. ReAct (Reason + Act): An advanced prompt engineering framework used for AI Agents. Instead of just answering a question, the AI is prompted to output an internal "Thought" (reasoning about what to do), take an "Action" (like searching a database or using a calculator), observe the result, and repeat until the problem is solved. Instruction Tuning: The crucial training step that turns a raw text-predictor into a helpful assistant. The model is fed thousands of examples of (Instruction -> Response) pairs so it learns to follow commands rather than just mindlessly completing sentences. Jailbreaking: The act of crafting highly complex or manipulative prompts specifically designed to bypass an AI model's safety filters, ethical guidelines, or system instructions, forcing it to generate forbidden or restricted content.