AI Security and Trustworthy AI (4280/6280)
Build AI that is effective for security, secure against attack, and trustworthy in deployment.
Fall 2026 · Mondays and Wednesdays, 2:00–3:30 PM · Instructor: Long Cheng
The Three Pillars
This course is organized around three interconnected pillars at the intersection of artificial intelligence and cybersecurity:
- AI for Security — using AI/ML to defend systems: network intrusion and anomaly detection, malware detection and classification, phishing/fraud detection, and AI-assisted security operations.
- Security of AI — protecting ML models from attack: adversarial examples, data poisoning and backdoors, model extraction/inversion and membership inference, and LLM security.
- Trustworthy AI — making AI safe, fair, robust, transparent, and accountable: interpretability, fairness, privacy-preserving ML, robustness certification and formal verification, and AI governance.
A single running example threads through all three: an ML-based detector deployed in a SOC must work (Pillar 1), resist evasion and poisoning (Pillar 2), and be explainable and accountable (Pillar 3).
Schedule at a Glance
| Week | Dates | Pillar | Topic | Reading | Due |
|---|---|---|---|---|---|
| 1 | Week 1 | Overview | Introduction and the Three Pillars of AI and Security | — | HW1 out (Wed) |
| 2 | Week 2 | AI for Security | Foundations of Machine Learning for Cybersecurity | — | HW1 due (Wed); HW2 out |
| 3 | Week 3 | AI for Security | Network Intrusion and Anomaly Detection | — | HW2 due (Wed); HW3 out |
| 4 | Week 4 | AI for Security | Malware Detection and Classification | — | HW3 due (Wed); HW4 out |
| 5 | Week 5 | AI for Security | Phishing/Fraud Detection and AI for Security Operations | — | HW4 due (Wed); HW5 out |
| 6 | Week 6 | Security of AI | Threat Modeling for ML Systems and Adversarial Examples | — | HW5 due (Wed); HW6 out; Midterm Project Proposal out |
| 7 | Week 7 | Security of AI | Advanced Evasion, Transferability, and Data Poisoning/Backdoors | — | HW6 due (Wed); Midterm Project work session |
| 8 | Week 8 | Security of AI | Midterm Project Work Session and Presentations | — | Midterm Project Report & Presentation due (Wed) |
| 9 | Week 9 | Security of AI | Model Extraction, Inversion, Membership Inference, and LLM Security | — | HW7 out (Mon) |
| 10 | Week 10 | Security of AI | Adversarial Defenses and Robustness | — | HW7 due (Wed); HW8 out |
| 11 | Week 11 | Trustworthy AI | Interpretability and Explainability | — | HW8 due (Wed); Capstone Project Proposal out |
| 12 | Week 12 | Trustworthy AI | Fairness: Definitions, Metrics, and Detection | — | Capstone Project Proposal due (Wed); Capstone Work Begins |
| 13 | Week 13 | Trustworthy AI | Fairness: Mitigation and Bias Correction | — | — |
| 14 | Week 14 | Trustworthy AI | Privacy-Preserving Machine Learning (Differential Privacy, Federated Learning) | — | HW9 out (Mon); Capstone Progress Report due (Wed) |
| 15 | Week 15 | Trustworthy AI | Robustness Certification, Formal Verification, and Safety | — | HW9 due (Wed) |
| 16 | Week 16 | Trustworthy AI | Transparency, Accountability, and AI Governance; Three-Pillar Synthesis | — | — |
| 17 | Week 17 | Synthesis | Capstone Presentations and Closing | — | Capstone Report due (Mon); Capstone Presentations (Mon–Wed) |
See the full schedule for the pillar grouping and lecture links.
Course Overview
This advanced, hands-on course is organized around three pillars at the intersection of artificial intelligence and cybersecurity. Students learn to apply machine learning to defensive security problems (AI for Security), to identify and model threats to AI systems and implement attacks and defenses (Security of AI), and to audit and improve AI systems for fairness, interpretability, privacy, robustness, and accountability (Trustworthy AI). Through programming assignments, case studies, and a sustained capstone project, students develop the technical and analytical skills to build AI systems that are effective for security, secure against attack, and trustworthy in deployment.
Learning Objectives
Upon successful completion of this course, students will be able to:
- Apply machine learning to cybersecurity problems (AI for Security) — build and evaluate detectors for network intrusion, malware, and phishing/fraud, accounting for class imbalance, concept drift, and operational constraints.
- Design and execute threat models for AI systems, identifying attack vectors, adversaries, capabilities, and assets at risk across the ML lifecycle.
- Implement adversarial attacks and defenses (Security of AI) using Python and deep learning frameworks (TensorFlow/PyTorch), including evasion, poisoning/backdoors, and model-extraction/privacy attacks.
- Apply interpretability and explainability techniques (e.g., LIME, SHAP, attention, Integrated Gradients) to understand and audit model decisions.
- Conduct fairness audits, detect and quantify bias, and implement fairness mitigation strategies.
- Design and implement privacy-preserving techniques such as differential privacy (DP-SGD), federated learning, and secure aggregation.
- Verify robustness and reason about AI governance using formal verification/certification techniques and transparency/accountability practices (model cards, audits, regulatory frameworks).
- Evaluate AI holistically across the three pillars, balancing detection effectiveness, security against attack, and trustworthiness trade-offs.
- Architect and present a capstone project that integrates multiple pillars on a real-world or benchmark AI system.
Assessment
| Component | Weight |
|---|---|
| Homework and Labs (9 assignments) | 35% |
| Quizzes and Participation | 10% |
| Midterm Project | 20% |
| Individual Capstone Project | 35% |
| Total | 100% |
See the Syllabus for full policies on late work, regrades, and academic integrity.