██FR█████ █INTELL███████████

As a New Hire, I’m Leading the Company Platform’s Facial Authentication System Development


This content originally appeared on DEV Community and was authored by wintrover

🎯 Project Overview

I’ve been tasked with developing a facial recognition-based user authentication system for the company platform, working independently on this critical project. This goes beyond simple feature development, encompassing real-time processing, multi-environment adaptation, and web-based implementation across various technical challenges.

🚀 Phase 1: Core Technology Development & Performance Optimization (July 30)

Deepface + MediaPipe Hybrid Architecture Implementation

I developed a real-time facial recognition prototype based on Deepface’s ArcFace model. However, initial testing revealed significant recognition rate drops in various environments (lighting, angles, glasses, etc.).

Solution: Applied MediaPipe 3D landmark technology to dramatically improve recognition rates across diverse conditions.

Performance Optimization Tasks

  • Processing Speed: Enhanced to 5.5 FPS
  • Memory Management: Resolved memory leaks that occurred during technology integration
  • Real-time Processing: Implemented real-time face detection based on webcam streaming

🏗 Phase 2: KYC System Architecture Design & Requirements Analysis (July 31 – August 1)

Requirements Gathering & System Design

Through development process reporting, I gathered specific requirements for the KYC procedure:

  • User Guidance: Real-time guidance like “Look left, look right”
  • Data Storage: Multi-frame capture and video storage capabilities
  • Precision Analysis: Similarity analysis for specific facial regions (eyes, nose, mouth) in addition to full face

Architecture Transition

Shifted from desktop environment to web browser-based webcam system implementation.

Technical Review & Collaboration Environment Setup

  • Patent Analysis: Compared ETRI and Apple patents to evaluate technical utility
  • Collaboration Environment: Established Docker and GitHub-based development environment sharing
  • Version Control: Developed branching strategies for efficient collaboration

🌐 Phase 3: Web-based System Implementation & Technical Challenge Resolution (August 4-5)

Backend Technology Stack Migration

To enable webcam functionality in web environment, migrated backend from Python to Express.js.

Key Changes:

  • Python Flask ? Node.js Express.js migration
  • WebSocket implementation for real-time communication
  • RESTful API design

MediaPipe Web Integration Problem Resolution

Encountered several technical issues during MediaPipe web integration:

3.1 CSP (Content Security Policy) Conflicts 

// Problem: CDN resources blocked by CSP
// Solution: Updated helmet middleware configuration
app.use(helmet({
  contentSecurityPolicy: {
    directives: {
      scriptSrc: ["'self", "https://cdn.jsdelivr.net"],
      connectSrc: ["'self", "https://cdn.jsdelivr.net"],
      workerSrc: ["'self", "blob:"]
    }
  }
}));

3.2 File Loading Timing Issues

  • Resolved WASM file loading delays
  • Implemented Promise-based initialization for async loading
  • Optimized module dependency management

3.3 Landmark Data Processing Errors 

// Problem: Landmark coordinates returned as normalized values
// Solution: Added logic for pixel coordinate conversion
drawLandmarkPoints() {
  this.ctx.arc(
    landmark.x * this.canvas.width, 
    landmark.y * this.canvas.height, 
    this.landmarkStyle.radius, 
    0, 
    2 * Math.PI
  );
}

🔧 Phase 4: Debugging & Performance Optimization

Key Issues & Resolution Process

4.1 Yaw Angle Measurement Error

Problem: Yaw angle measured abnormally high (85 degrees) when facing forward

Root Cause: Default values (0,0) from generateDummyLandmarks function included in face width calculation

Solution: Modified calculateAngles function to skip dummy landmarks

// Exclude dummy landmarks during face width calculation
const validLandmarks = landmarks.filter(landmark => 
  landmark.x !== 0 || landmark.y !== 0
);
const faceWidth = Math.max(...validLandmarks.map(l => l.x)) - 
                  Math.min(...validLandmarks.map(l => l.x));

4.2 Landmark Rendering Performance Issues

Problem: Performance degradation when rendering 478 landmark points in real-time

Solutions:

  • Removed 95% of unnecessary console logs
  • Canvas rendering optimization
  • Implemented smooth animations using requestAnimationFrame

4.3 Server Connection Issues

Problem: FaceRecognitionClient‘s testServerConnection method causing infinite loops

Solution: Added retry limits (20 attempts, ~2 seconds) and proper error handling for initialization failures

📊 Current Status & Remaining Tasks

✅ Completed Features

  • Deepface + MediaPipe hybrid facial recognition system
  • Web-based webcam integration
  • Real-time landmark rendering (478 points)
  • Basic KYC workflow implementation
  • Performance optimization (5.5 FPS)

🔄 In Progress

  • Yaw Angle Measurement Error: Currently debugging, need algorithm improvement for accurate angle measurement
  • Quality Assessment System: Completing real-time quality evaluation functionality when server connected
  • User Interface: Improving intuitive UI/UX for end users

🎯 Next Phase Plans

  1. Yaw Angle Measurement Accuracy Enhancement
  2. Multi-face Processing Feature Addition
  3. Liveness Detection Implementation for Security Enhancement
  4. Mobile Environment Support
  5. Production Server Deployment

💡 Key Learnings

Technical Growth

  • Hybrid Architecture: Designed optimal solutions combining Deepface and MediaPipe strengths
  • Web-based Real-time Processing: Gained experience implementing real-time facial recognition systems via webcam
  • Performance Optimization: Enhanced skills in memory management and rendering optimization for large-scale data processing

Project Management

  • Requirements Management: Realized the critical impact of requirements convergence in early design phases
  • Technical Debt Management: Understood the importance of balancing performance and functionality
  • Problem-solving Abilities: Improved systematic analysis and resolution of complex technical challenges

This project has been a significant growth opportunity, going beyond simple feature development to encompass real-time system design, performance optimization, and user experience improvement. I plan to continue developing and improving the system to create a more stable and accurate facial authentication solution.


This content originally appeared on DEV Community and was authored by wintrover