👁️ Building FaceML: Local-First Face Tracking and Recognition in Rust using Apple Vision

Most face-recognition demos start with an API call.

Upload an image.
Wait for a response.
Receive a face ID.

That is convenient, but it also means the most sensitive part of the image leaves the machine.

I wanted to explore the opposite design:

Camera frame
    ↓
Apple Vision landmarks
    ↓
Rust tracking and geometry
    ↓
Optional polygon or depth face mask
    ↓
Optional on-device face embedding
    ↓
Local SQLite identity gallery

That experiment became FaceML. The Rust crate and executables are named facefeature.

It is a local-first face geometry, tracking, and optional identity pipeline for Apple Silicon. It calls Apple’s Vision framework directly from Rust, uses a native AppKit camera window, and can run an SFace ONNX model through Core ML. There is no Swift bridge project, WebView, cloud API, or image upload in the pipeline.

The source code is available in the FaceML repository.

This post is about how the pieces fit together, and the problems that appeared once I moved beyond detecting a rectangle around a face.

The Pipeline

The system has two related but deliberately separate paths.

The geometry path runs continuously. It detects faces and landmarks, associates detections across frames, smooths movement, predicts through display latency, and draws the overlay. It can also turn the same landmarks into a dense polygon mesh or a pseudo-3D depth wireframe.

The identity path is optional. It waits for a usable face, aligns a small crop, creates several embeddings, checks whether those embeddings agree, and only then queries or updates the local gallery.

The live pipeline is callback-driven. Rust configures one AVCaptureSession, then AVFoundation sends each CMSampleBuffer to FrameDelegate::did_output_frame() on a serial queue. That callback extracts the CVPixelBuffer, runs Apple Vision, updates the Rust tracker, and optionally submits identity work. Only the final layer update crosses back to the main dispatch queue.

At the same time, AVCaptureVideoPreviewLayer renders the camera session directly. The final window is a composition of that preview, zero or more face-mask CAShapeLayers, and the diagnostic shape and text layers produced by Rust.

The layer stack is created explicitly. The preview is the base image, optional mask layers sit above it, and the ordinary landmark overlay remains on top:

let preview = unsafe { AVCaptureVideoPreviewLayer::layerWithSession(&session) };
let overlay = CAShapeLayer::layer();
let face_mask_layers = match self.ivars().face_mask {
    FaceMaskMode::None => Vec::new(),
    FaceMaskMode::Polygon => vec![make_mask_layer(0.72, 0.85)],
    FaceMaskMode::Depth => (0..DEPTH_LAYER_COUNT)
        .map(|index| {
            let intensity = index as f64 / (DEPTH_LAYER_COUNT - 1) as f64;
            make_mask_layer(0.22 + intensity * 0.56, 0.58 + intensity * 0.26)
        })
        .collect(),
};

preview.setFrame(content_view.bounds());
for layer in &face_mask_layers {
    layer.setFrame(content_view.bounds());
}
overlay.setFrame(content_view.bounds());

root_layer.addSublayer(&preview);
for layer in &face_mask_layers {
    root_layer.addSublayer(layer);
}
root_layer.addSublayer(&overlay);

AVFoundation owns frame delivery, but Rust chooses the callback queue and delegate:

let frame_delegate = FrameDelegate::new(
    &preview,
    &face_mask_layers,
    self.ivars().face_mask,
    &overlay,
    self.ivars().face_id.clone(),
);
let frame_queue =
    DispatchQueue::new("dev.facefeature.camera.frames", DispatchQueueAttr::SERIAL);

unsafe {
    output.setSampleBufferDelegate_queue(
        Some(ProtocolObject::from_ref(&*frame_delegate)),
        Some(&frame_queue),
    );
}

Keeping these paths separate matters. A tracker answers:

Is this detection the same moving face as a moment ago?

Face recognition answers:

Does this face match a person stored from an earlier session?

Those are different questions with different failure modes.

Calling Apple Vision Directly From Rust

The first layer is intentionally small: a backend-neutral detector trait and a macOS implementation.

pub trait FaceGeometryDetector {
    fn name(&self) -> &'static str;
    fn detect_path(&self, image_path: &Path) -> Result<Detection, DetectorError>;
}

On macOS, AppleVisionDetector creates a VNDetectFaceLandmarksRequest, runs it through a VNImageRequestHandler, and translates every VNFaceObservation into portable Rust data.

For a live camera frame, the detector wraps the existing Core Video buffer instead of first encoding or copying it into another image format:

pub fn detect_pixel_buffer(
    &self,
    pixel_buffer: &CVPixelBuffer,
) -> Result<Detection, DetectorError> {
    autoreleasepool(|_| {
        let options = empty_options();
        let handler = unsafe {
            VNImageRequestHandler::initWithCVPixelBuffer_options(
                VNImageRequestHandler::alloc(),
                pixel_buffer,
                &options,
            )
        };
        detect_with_handler(&handler)
    })
}

The shared handler path constructs and performs the actual Vision request:

let request = unsafe { VNDetectFaceLandmarksRequest::new() };
let request_for_handler: Retained<VNRequest> =
    request.clone().into_super().into_super();
let requests = NSArray::from_retained_slice(&[request_for_handler]);

handler
    .performRequests_error(&requests)
    .map_err(|error| DetectorError::Backend(error.to_string()))?;

let observations = unsafe { request.results() }
    .ok_or_else(|| DetectorError::Backend("Vision returned no results array".to_owned()))?;

The result includes:

• face and landmark confidence;
• normalized bounding boxes;
• roll, yaw, and pitch when Vision exposes them;
• named regions such as the eyes, eyebrows, pupils, nose, lips, and face contour;
• scale-independent measurements such as inter-eye distance divided by face width.

Vision returns landmark points relative to the face box. The rest of the application should not need to remember that detail, so the backend converts them immediately into normalized full-image coordinates:

let points = normalized_points
    .iter()
    .map(|point| Point {
        x: bounds.x + point.x * bounds.width,
        y: bounds.y + point.y * bounds.height,
    })
    .collect();

The public coordinate system is therefore always [0, 1] with the origin at the lower-left, matching Vision. Rendering code performs the final conversion into window pixels.

This boundary makes the model data portable even though the current detector backend is macOS-specific.

A Camera Window Without a WebView

The live executable is a native AppKit application assembled from Rust.

AVFoundation captures 1280x720 BGRA frames. An AVCaptureVideoPreviewLayer displays the camera, while Core Animation layers draw face boxes, landmarks, pose values, confidence, and identity labels.

The delegate method is intentionally tiny. It hands the sample buffer to the Rust processing pipeline:

unsafe fn did_output_frame(
    &self,
    _output: &AVCaptureOutput,
    sample_buffer: &CMSampleBuffer,
    _connection: &AVCaptureConnection,
) {
    self.process_frame(sample_buffer);
}

Inside process_frame(), the data types make each boundary visible:

let Some(pixel_buffer) = (unsafe { sample_buffer.image_buffer() }) else {
    return;
};

let detector = AppleVisionDetector;
let detection = match detector.detect_pixel_buffer(&pixel_buffer) {
    Ok(detection) => detection,
    Err(_) => return,
};

let observed_faces = match self.ivars().tracker.lock() {
    Ok(mut tracker) => tracker.update_at(detection.faces, observed_at),
    Err(_) => return,
};
let tracked_faces = observed_faces
    .iter()
    .map(|face| face.predicted(inference_seconds + PRESENTATION_DELAY_SECONDS))
    .collect::<Vec<_>>();

The real function handles errors and mutex access explicitly; the shortened excerpt above highlights the transformation from CMSampleBuffer to Detection to TrackedFace.

The frame callback is deliberately backpressure-aware:

• incoming frames are accepted at up to 30 FPS;
• late video frames are discarded;
• face-ID work uses a bounded queue;
• if the embedding worker is busy, the current sample is skipped instead of building a growing latency backlog;
• diagnostic labels refresh at 5 FPS even though geometry can update more often.

A live vision UI feels wrong when it faithfully draws an old result. Keeping the newest useful frame is more important than processing every frame.

Rendering is the only part dispatched back to the main queue:

DispatchQueue::main().exec_async(move || {
    let preview = unsafe { &*(preview_layer as *const AVCaptureVideoPreviewLayer) };
    let face_mask_layers = face_mask_layers
        .iter()
        .map(|layer| unsafe { &*(*layer as *const CAShapeLayer) })
        .collect::<Vec<_>>();
    let overlay = unsafe { &*(overlay_layer as *const CAShapeLayer) };

    update_overlay(
        preview,
        &face_mask_layers,
        face_mask_mode,
        overlay,
        &tracked_faces,
        &face_id_matches,
        capture_status.as_ref(),
        refresh_labels,
        inference_seconds * 1_000.0,
    );
});

update_overlay() builds separate Core Graphics paths for the optional mask and the normal annotations, then swaps all of them without implicit animation:

CATransaction::begin();
CATransaction::setDisableActions(true);
for (layer, path) in face_mask_layers.iter().zip(&face_mask_paths) {
    layer.setPath(Some(&path));
}
overlay.setPath(Some(&path));

if refresh_labels {
    replace_face_labels(/* preview, tracks, identity results, timing */);
}

CATransaction::commit();

Turning Landmarks Into a Face Mask

The new face-mask modes reuse Vision’s geometry; they do not add another recognition model.

The polygon mode builds a dense wireframe over the live face:

cargo run --release --bin facefeature-camera -- --face-mask polygon

• -mask-only hides the camera preview while leaving capture and analysis active. That distinction matters: it changes presentation, not what the camera pipeline processes.

cargo run --release --bin facefeature-camera -- \
  --face-mask polygon --mask-only

The renderer selects the path builder from the CLI mode:

for tracked_face in tracked_faces {
    match face_mask_mode {
        FaceMaskMode::None => {}
        FaceMaskMode::Polygon => {
            if let Some(path) = face_mask_paths.first() {
                add_polygon_face_mask(path, preview, &tracked_face.geometry);
            }
        }
        FaceMaskMode::Depth => {
            add_depth_face_mask(&face_mask_paths, preview, &tracked_face.geometry);
        }
    }
}

The mesh starts with Vision’s dense allPoints region, adds dedicated eye and eyebrow anchors that can otherwise disappear at steep yaw, synthesizes roll-aware forehead rows, interpolates edge midpoints, and repeatedly runs Delaunay triangulation. Triangles crossing the eyes or inner mouth are removed so those openings remain readable.

The depth mode uses the same 2D draw anchors but computes a small 3D lighting model:

cargo run --release --bin facefeature-camera -- \
  --face-mask depth --mask-only

Each landmark receives a smooth dome depth, with the nose and lips pulled forward and the eyes slightly recessed. Yaw, pitch, and roll rotate the surface normals once; the original Vision points still anchor the lines, which avoids applying yaw twice. Eight low-contrast shape layers quantize the lighting into a restrained wireframe.

This is intentionally pseudo-3D. It is a visualization inferred from one camera image, not measured facial depth. At strong yaw, the mesh also warps the self-occluded side toward the face axis, and it is not clipped to the axis-aligned detection box because a rotated contour can legitimately extend beyond that box.

Detection Indexes Are Not Tracking IDs

Vision may return the same people in a different order from one frame to the next. Treating the array index as an identity makes labels jump between faces.

FaceML assigns each active face a tracking ID. For every update it constructs a cost matrix between existing tracks and new detections. The match cost combines:

60% bounding-box overlap
30% relative center distance
10% normalized landmark-shape distance

That weighting appears directly in the association cost:

fn match_cost(track: &Track, detection: &FaceGeometry, observed_at: Instant) -> f64 {
    let predicted = predicted_bounds(track, observed_at);
    let observed = detection.bounding_box;
    let overlap = intersection_over_union(predicted, observed);
    let average_diagonal = ((diagonal(predicted) + diagonal(observed)) / 2.0).max(0.05);
    let center_distance = center(predicted).distance(center(observed));
    let relative_center_distance = center_distance / average_diagonal;

    if overlap < 0.05 && relative_center_distance > 1.25 {
        return INVALID_MATCH_COST;
    }

    let landmarks = landmark_shape_distance(&track.geometry, detection).unwrap_or(0.5);
    0.6 * (1.0 - overlap)
        + 0.3 * relative_center_distance.min(1.0)
        + 0.1 * landmarks.min(1.0)
}

The Hungarian algorithm solves the assignment globally. This is important when multiple faces cross or stand close together: choosing the nearest detection independently for every track can produce a locally reasonable but globally inconsistent assignment.

The tracker also keeps an unmatched option. A detection that is too far from a predicted track is allowed to become a new track instead of being forced into a bad match.

After association, geometry is smoothed and velocity is updated. A track remains eligible for reassociation for 15 missed detection updates, which gives it a short grace period through occlusion or a temporarily missed face.

Compensating for Inference Delay

Smoothing removes jitter, but it adds lag. Inference and presentation add more.

The result is a common webcam-overlay problem: when the subject moves, the box follows behind the face.

FaceML measures the detection time and predicts the smoothed geometry forward using the track velocity:

pub fn predicted(&self, pipeline_delay_seconds: f64) -> Self {
    let prediction_seconds =
        (pipeline_delay_seconds.max(0.0) + self.smoothing_lag_seconds).min(0.25);

    // Translate the box and every landmark using the measured velocity.
}

The prediction horizon includes both measured pipeline delay and the estimated lag introduced by exponential smoothing. It is capped at 250 ms so a stale velocity cannot launch an overlay across the screen.

This is a small detail, but it changes the result from a diagnostic drawing into something that feels attached to the face.

Tracking Is Not Recognition

A tracking ID is temporary by design. Once its grace period expires, the next detection gets a new ID even if the same person walked away and returned.

Persistent recognition lives in a separate layer:

This separation also keeps the default mode lightweight. Plain camera mode never loads the recognition model or opens a biometric gallery.

To enable automatic local matching and enrollment:

cargo run --release --bin facefeature-camera -- --face-id

To match without changing the gallery:

cargo run --release --bin facefeature-camera -- --read-only

In read-only mode, a match keeps its stored person ID and name. An unmatched face is shown as Unknown, and no identity, centroid, counter, name, or timestamp is written.

Aligning a Face Before Embedding It

The SFace model expects an aligned 112x112 face, not an arbitrary camera crop.

FaceML derives five anchor points from Vision’s landmark regions:

left eye
right eye
nose
left mouth corner
right mouth corner

It solves a similarity transform from those points to the standard SFace template, then inverse-maps the destination pixels into the camera frame with bilinear sampling. The output is an RGB/NCHW tensor.

The alignment loop works backward from every destination pixel into the source frame:

let source = five_landmarks(face, width as f32, height as f32)?;
let transform = similarity_transform(&source, &SFACE_TEMPLATE)?;
let mut tensor = vec![0.0; 3 * SFACE_WIDTH * SFACE_HEIGHT];
let plane = SFACE_WIDTH * SFACE_HEIGHT;

for destination_y in 0..SFACE_HEIGHT {
    for destination_x in 0..SFACE_WIDTH {
        let (source_x, source_y) =
            transform.inverse(destination_x as f32 + 0.5, destination_y as f32 + 0.5);
        let [red, green, blue] = sample_bgra(
            bgra,
            width,
            height,
            bytes_per_row,
            source_x - 0.5,
            source_y - 0.5,
        );

        let offset = destination_y * SFACE_WIDTH + destination_x;
        tensor[offset] = red;
        tensor[plane + offset] = green;
        tensor[2 * plane + offset] = blue;
    }
}

Alignment reduces variation caused by face position, rotation, and scale before the model sees the image. Without it, the embedding can describe the crop as much as it describes the person.

The bundled SFace model runs through ONNX Runtime with the Core ML execution provider. Compute units are set to all, so Core ML can schedule compatible operations on the M1 CPU, GPU, and Neural Engine. Unsupported operations may still fall back to the CPU.

Apple Vision makes its own execution-device decisions. Apple does not expose a guarantee that the built-in face-landmark request runs on the Neural Engine, so the application does not claim that it does.

One Frame Is Weak Evidence

A technically valid crop can still be poor recognition evidence. The subject may be blinking, blurred, underexposed, too small, or turning too far away.

Before inference, the identity pipeline rejects samples with weak geometry. It checks detection confidence, landmark confidence, face area, roll, yaw, and the required landmark groups.

The aligned tensor then receives additional checks for exposure, contrast, texture, and eye openness.

For each new tracking ID, FaceML gathers observations at 90 ms intervals. Three samples are enough when they agree. If they do not, it collects up to five observations and searches for the most mutually consistent group of three.

The selection score is mostly embedding agreement, with a smaller contribution from image quality:

score = mean_pairwise_similarity × 0.9 + quality × 0.1

For each possible trio, the weakest pair becomes the consistency value. This makes the selection sensitive to an outlier instead of hiding it inside an average:

let similarities = [
    cosine_similarity(&samples[first].embedding, &samples[second].embedding),
    cosine_similarity(&samples[first].embedding, &samples[third].embedding),
    cosine_similarity(&samples[second].embedding, &samples[third].embedding),
];

let consistency = similarities.into_iter().fold(f32::INFINITY, f32::min);
let mean_similarity = similarities.into_iter().sum::<f32>() / 3.0;
let score = mean_similarity * 0.9 + quality * 0.1;

if best.is_none_or(|(_, best_score, _, _)| score > best_score) {
    best = Some((indices, score, consistency, quality));
}

The minimum pairwise similarity inside the chosen trio becomes the consistency value. This stops one attractive but inconsistent frame from dominating the result.

The important design choice is that ordinary matches never rewrite a stored centroid. Gallery updates happen only through deliberate guided capture or consensus-backed enrollment of a new identity. A borderline match therefore cannot slowly drag a known identity toward the wrong person.

Guided Multi-Pose Enrollment

Automatic enrollment is convenient, but a deliberate capture session produces better templates.

cargo run --release --bin facefeature-camera -- \
  --capture --name "Radit"

The overlay guides the subject through five poses:

straight → left → right → up → down

Each pose must be held steadily for 350 ms. The pipeline captures three embeddings per pose, for 15 samples in total.

Guided capture now requires exactly one visible face. Zero faces and multiple faces are explicit error states, and either one resets the stable-pose hold instead of allowing a stale candidate to continue:

let tracked_face = match tracked_faces {
    [] => {
        client.report_capture_problem("NO FACE — move one person into view");
        return;
    }
    [tracked_face] => tracked_face,
    _ => {
        client.report_capture_problem("MULTIPLE FACES — only one person is allowed");
        return;
    }
};

The capture banner also shows invalid pose, quality, alignment, and camera-frame problems in red. Successful progress remains separate from those actionable errors.

Nothing is committed while the capture is incomplete. When all poses succeed, the centroid and pose samples are written together in a SQLite transaction. A failed or cancelled session does not leave a partially enrolled identity behind.

At recognition time, FaceML classifies the query as center, horizontal, vertical, or unknown. It compares against the main centroid plus only the pose-template group relevant to that query. A horizontally turned face does not need to compete with unrelated up/down templates.

The matcher expresses that rule by chaining the centroid with only accepted pose templates and keeping the best cosine similarity:

fn identity_similarity(
    identity: &Identity,
    embedding: &[f32],
    query_pose: FaceQueryPose,
) -> f32 {
    std::iter::once(identity.centroid.as_slice())
        .chain(
            identity
                .pose_templates
                .iter()
                .filter(|template| query_pose.accepts(template.pose))
                .map(|template| template.embedding.as_slice()),
        )
        .map(|template| cosine_similarity(template, embedding))
        .max_by(f32::total_cmp)
        .unwrap_or(f32::NEG_INFINITY)
}

Some Vision revisions do not provide native pitch. In that case, FaceML estimates vertical pose from the proportions between the eyes, nose, and mouth. The UI marks the value with ~ to make it clear that it is an approximation rather than a native Vision measurement.

The Local Identity Gallery

The SQLite gallery stores:

• a numeric person ID;
• an optional editable name;
• a compact fingerprint used as a human-readable gallery label;
• a normalized embedding centroid;
• enrollment sample counts;
• pose-specific templates;
• schema, embedding-format, and model-checksum metadata.

The model checksum matters. Embeddings produced by different models should not silently share the same vector space. FaceML rejects a gallery when its stored model checksum does not match the active model.

The fingerprint is not a raw face hash and should not be treated as a universal identifier. It is generated from a quantized embedding and is useful only as a compact label for a gallery entry.

Most importantly, normalized embeddings are still biometric data.

Local-first does not mean harmless.

The database and its WAL files are excluded from Git, but the user still needs to protect, relocate, or delete them like any other sensitive local data. FaceML stores no face image in the gallery and uploads neither images nor embeddings.

Image Mode and JSON Output

The native camera is only one interface. The small CLI can inspect a still image and serialize portable geometry as JSON:

cargo run --release -- detect /path/to/photo.jpg --pretty

The response reports its backend and coordinate system explicitly:

{
  "backend": "apple_vision",
  "coordinate_system": "normalized_lower_left",
  "faces": [
    {
      "confidence": 1.0,
      "bounding_box": {
        "x": 0.24,
        "y": 0.52,
        "width": 0.10,
        "height": 0.18
      },
      "yaw_radians": 0.0,
      "landmarks": []
    }
  ]
}

The shortened example omits most landmark points, but the real response keeps them grouped by named region.

Benchmarking the Full Pipeline Without a Camera

Live camera timings are useful, but they mix code performance with movement, lighting, autofocus, and whatever happens to be in front of the webcam. FaceML now has a repeatable headless benchmark:

cargo run --release --bin facefeature-camera -- --benchmark

The benchmark embeds a fictional 160x90 BGRA face fixture in the binary, expands it in memory to a 1280x720 CVPixelBuffer, and runs the same major stages as the application. It does not open the webcam and does not read or write the identity database.

let detection = AppleVisionDetector.detect_pixel_buffer(&pixel_buffer)?;
let tracked = tracker.update_at(detection.faces, Instant::now());
let geometry = benchmark_depth_mesh_and_paths(&tracked[0].geometry);
let tensor = benchmark_aligned_tensor(&pixel_buffer, &tracked[0].geometry)?;
let embedding = engine.embed(tensor)?;
let similarity = engine.match_embedding(&embedding);

If a future Vision revision rejects the fictional fixture, the benchmark reports that fact and feeds deterministic fallback geometry into the downstream stages. The fallback rate is part of the output, so a successful timing run cannot silently pretend Vision detected a face.

One 100-iteration M1 run, after five warmups, produced:

That is about 49.5 complete synthetic frames per second, with an average mesh of 433 vertices, 791 triangles, and 1,225 unique edges. The embedding has 128 dimensions; Vision detected one face per request and fallback geometry was used in 0% of that run.

These numbers describe one machine and one run, not a universal promise. Temperature, power mode, background load, macOS, and Core ML cache state can all move them. --benchmark-iterations N changes the measured run count when a longer sample is useful.

Project Shape

The code is divided by responsibility:

src/model.rs                  portable geometry types
src/detector/mod.rs           backend-neutral detector interface
src/detector/apple_vision.rs  direct Apple Vision implementation
src/tracker.rs                assignment, smoothing, and prediction
src/face_id.rs                alignment, embeddings, and SQLite gallery
src/main.rs                   image-to-JSON CLI
src/bin/facefeature-camera.rs camera UI, masks, capture, and benchmark
assets/benchmark_face_fixture.bgra deterministic benchmark input
.github/workflows/apple-silicon-ci.yml native test, benchmark, and package job

The portable tracker and model types can be tested without a camera. The current suite reports 50 passing tests and one ignored hardware-heavy Core ML test. It covers global assignment, detector reordering, brief occlusion, track expiry, latency prediction, jitter smoothing, face alignment, sample consensus, pose-aware matching, non-adaptive ordinary matches, SQLite reopening, schema migration, atomic guided capture, polygon and depth geometry, mask-only options, benchmark statistics, and capture alerts.

What I Learned

The model was not the hardest part.

The difficult work lived around it:

• converting coordinate systems once and keeping the contract explicit;
• preventing detection order from becoming identity;
• handling backpressure so old frames do not accumulate;
• smoothing geometry without making the overlay feel delayed;
• deriving a useful 3D-looking surface from 2D landmarks without claiming measured depth;
• rejecting weak evidence before inference;
• separating temporary tracking state from persistent biometric identity;
• making gallery writes intentional and transactional;
• benchmarking the complete path without camera or database variability;
• exercising native Apple framework code on Apple Silicon CI;
• stating hardware-acceleration limits accurately.

A face demo can look convincing with one good frame. A usable local pipeline has to behave sensibly across movement, missed detections, changing poses, bad lighting, application restarts, and an overloaded worker.

That is the direction of FaceML: not a cloud face API rebuilt locally, but a small native vision system whose data flow, persistence, and failure modes remain visible.