Iris Look-to-Pay
eye crop128×128 grayHOG · 1,764-dPCAcosine ≥ 0.70pay or explain
Browser port checked against the Python original: HOG error ≈ 1e-7
Human–computer interaction · computer vision
Ho Chi Minh City --:--Your time --:--
I build interactive systems that act on what they perceive, and study how they should behave when that perception is uncertain.
01Research
Two papers accepted in 2026. Each opens to a page on how the method works, with the paper's numbers and an example you can run.
Nhan Thanh Ho*, Anh Dong Nguyen*, Thuan Duc Tran*, Thang Van Nguyen, Bui Anh Thinh, Nguyen Bao Tran, Long Nhat Ho * equal contribution
Every field a vision pipeline perceives (when, where, who, what) is stored with a calibrated confidence, and retrieval marginalises over whether perception was right. Under heavy noise: 3.3× Recall@5 over exact matching.
My part Probabilistic model · perception pipeline (YOLOv8, ByteTrack, homography uncertainty) · writing and figures
Thanh Nhan Ho*, Dong Anh Nguyen*, Minh Tai Pham Nguyen, Minh Khue Phan Tran, Hoang Minh Nguyen, Nguyen Bao Tran, Trong Nhan Le * equal contribution
Sparse loop detectors lend pseudo-labels to the roads that only taxis cover, by similarity of their 24-hour patterns; a graph network learns the whole city. hotspot@100 = 0.97 on 838 road segments.
My part Graph models (GraphSAGE, multi-scale GNN, baselines) · evaluation and hotspot maps
02Research direction
Perception is never certain, yet most interfaces turn a probability into a yes or a no before the person is involved. I want to carry uncertainty all the way to the decision.
Calibrated, per-field confidence for vision pipelines, so one wrong detection does not decide for the user.
ACCV 2026Borrowing supervision from reliable sensors by similarity, and learning the rest on a graph.
ICISN 2026Next: wearable and egocentric input, glasses, rings and hand tracking, where the interface must decide to act, ask or wait.
Research statement03Systems
Each runs in the browser, with public code, tests and a write-up of the mechanism, not only the screens.
eye crop128×128 grayHOG · 1,764-dPCAcosine ≥ 0.70pay or explain
Browser port checked against the Python original: HOG error ≈ 1e-7
video frameYOLO11n · ONNXseat zonesstate machinestaff alert
On-device detection in the browser, about 150 ms per frame on WebAssembly
1,441 OSM cafésbilingual labellerevidence · 1−Π(1−c)rankexplain
20 unit tests in CI; unknown is scored as unknown, not as no
04About
Ho Chi Minh City, Vietnam
I work where computer vision meets interaction. My papers model perceptual uncertainty and learn from sparse labels; my systems put those ideas in front of people. Industrial management taught me how systems run, English how people say what they mean, telecommunications how signals travel.
05Contact
Open to PhD positions
If your lab is asking what comes after the screen, I would love to talk. Pick a reason. Change the words. Send.
ToDong Anh Nguyen
donganh1409a@gmail.com
Ho Chi Minh City, Vietnam
Case study 02 · 2024
Know there is a seat before you leave home.
SeatScout turns the cameras cafés already have into a live seat map.
or see how it works firstSystem at a glance
Video frameCCTV clip, sample or webcam
YOLO11nONNX Runtime Web, 512 px, WebGPU or WASM
Filterperson + 11 belongings (bag, laptop, cup…)
Seat zonesperson covers ≥ 25% of a zone, or a thing is ≥ 50% inside
State machineFree → Taken → Away → Check, with timers
Map + alertsfloor plan, seat list, staff check
Design rationale
How it works
Drawn for this portfolio to explain the idea, not footage from the detector.
01 · Context and goal
Next to my school in Dubai is a restaurant that fills up fast. The only way to know if there is a seat is to walk over. Some days you find a table. Some days you find a queue, and the break is gone.
Quest
Show every seat, live, without asking anyone to do anything new.
02 · My role
Dong Anh Nguyen
Founder · UX and computer-vision design
Srihari Murali
Player 2 · AI Engineer, Deloitte
Coded the 2024 detector in YOLO11.
03 · Process and solution
Level 1
YOLO11 finds people, and the bags, cups and laptops they leave.
Level 2
Each seat is drawn once, on the camera view and on the floor plan.
Level 3
Every taken seat shows how long its guest has stayed.
Level 4
Order a drink and book a seat before you leave.
Live demo
git clone https://github.com/SEATSCOUT-SOCIAL-APP-TEAM/AI-Engine-Prototype
cd AI-Engine-Prototype/docs
python3 -m http.server 8000
# open localhost:8000/seatscout/
YOLO11n runs in the tab with ONNX Runtime Web. Webcam frames never leave your device.
04 · How it's built
A guest goes to the restroom. Free the seat and someone takes it; hold it forever and the map lies. So the seat waits, and what the guest left behind decides how long.
Free
Clock14:00
Stayed0 min
Away–
| Layer | 2024 | Live demo, 2026 |
|---|---|---|
| Input | Video I filmed at the restaurant | Sample clips or your webcam |
| Detect | YOLO11 in Python, coded by Srihari | YOLO11n, ONNX Runtime Web |
| Seat logic | Seat zones, four states, the clock and the step-away rules: my design, in both | |
| App | Python, Streamlit | Plain JavaScript, GitHub Pages |
The live demo waits before it changes a seat: one second of sitting before Taken, so a passing waiter does not count, and a second and a half of absence before Away, so a missed frame is not a guest leaving.
What it taught me
A camera in a café already feels like being watched. A product built on it has to feel like a favour, so the map shows seats, never faces.
Seeing seats was half the answer. Which café is right for you at all? That question became Cafe Finder.
Case study 03 · 2025
Say how you feel. Cafe Finder reads what people say about 1,441 real cafés in Saigon, learns the places you go back to, and tells you why it picked each one.
System at a glance
1,441 real cafésOpenStreetMap extract of Saigon, 31 Oct 2026
Readtags, opening hours, names, descriptions, notes
Labelbilingual lexicon with negation; each label keeps its evidence
Combine1 − Π(1 − c) over sources; tags 0.90, text 0.70, name 0.45
Rank0.5 words + 0.35 taste · words + 0.15 nearness
Explainevery label opens to its source
Design rationale
Try it
git clone https://github.com/donganh1409/perfect-cafe-finder
cd perfect-cafe-finder
npm ci
npm run dev # localhost:8080/perfect-cafe-finder/
npm test # the ranking tests
Real cafés from OpenStreetMap. Your visits and notes stay in your browser.
01 · Context and goal
Maps rank cafés by rating and distance. People choose them by feeling: quiet enough to think, busy enough to feel alive, cheap enough for a long afternoon. The answer is already in the places each person keeps going back to.
The goal
Suggest the next café the way a good friend would: from what you love, not from what is popular.
02 · My role
With the SeatScout team. I wrote the engine's keyword step; Srihari Murali, AI Engineer at Deloitte, laid out its architecture.
03 · Process and solution
Every place gets the same short profile.
The places you return to add up to your taste.
A sentence becomes keywords. This one is from the engine's prompt.
Your words, your taste and the walk, in one score.
A pick you can't explain is a pick nobody trusts.
The app
Concept screens for the phone, with sample data. The working web app is linked above.
04 · How it's built
“The all-Vietnamese house band performs every night… it's always jammed with student-aged groupies.”
Cà Phê Acoustic
“Quiet upstairs with lots of old books. Không ồn.”
Live music84% sure
Quiet85%
Books85%
Livelyargued both ways
Sources that agree add up like witnesses: 1 − (1 − 0.70)(1 − 0.45) = 0.84. “Không ồn”, not noisy, counts against lively.
function facetMatch(cafe, keyword) {
const label = best(cafe.labels, keyword.keys);
if (!label) return 0.25; // nobody has said
if (label.score > 0)
return 0.55 + 0.45 * label.score; // how sure we are
return 0; // evidence against
}
words = mean(keywords.map(k => facetMatch(cafe, k)));
taste = (cos(centred(cafe), yourTaste) + 1) / 2 * words;
score = .50 * words + .35 * taste + .15 * near;
Why not Google reviews?
Google's terms do not allow storing review text, or showing it on a map that is not Google's. So each café links to its reviews, and what someone learns from them comes back as a note, in their own words.
What I would test next
Label fifty cafés by hand, then measure how often the reader agrees, and whether people trust a pick more when they can open its evidence.
Data: © OpenStreetMap contributors (ODbL), descriptions from Wikivoyage (CC BY-SA). The engine's keyword step lives in AI-Engine-Prototype.
ACCV 2026Asian Conference on Computer Vision · accepted
Problem
Video search systems store each event as fields (when, where, who, what), filled by detectors, trackers and classifiers that are sometimes wrong. Exact matching throws away the right event after a single wrong field.
Idea
Treat whether each field is correct as a hidden variable. Store a calibrated confidence with every field, and score events by marginal likelihood, so reliable fields compensate for doubtful ones.
Result
Under the heaviest attribute noise, 3.3× higher Recall@5 than exact matching (0.394 vs 0.120) on CAVIAR. The same rule transfers to outdoor VIRAT without recalibration (0.221 vs 0.085).
My part
Co-first author. I worked on the probabilistic model, built the perception pipeline (YOLOv8 + ByteTrack, track windows, homography-based spatial uncertainty), and wrote the paper and its figures.
01
A query asks for four fields. Perception got the target event's action wrong. Move the slider: how sure was the action classifier when it made that mistake?
Exact matching (Hard-AND)
Marginal likelihood (ours)
A toy example computed live with the paper's scoring rule, log P(Q|E) = Σk log(ĉksk + (1−ĉk)bk), with bk = 1/8. If the classifier was unsure, the target stays on top. If it was confidently wrong, an event with a doubtful match overtakes it. That is the behaviour the model is designed to have.
02
1 · Structured memory with confidence
Each event keeps its interpretable fields, and each field carries the reliability needed to reason about perceptual error.
E = { (t, ĉt), (l, Σl), (a, ĉa), (s, ĉs) }
Events are stored as ExplicitCells in a multi-tier memory indexed along time, topic, identity and space.
2 · Spatial uncertainty in metres, not pixels
A low-confidence box gives a looser foot-point. Its image covariance is pushed through the camera homography to the ground plane, and a location query becomes a Mahalanobis test.
σpx = σ0 + σ1(1 − conf), Σground = JH Σimage JH⊤
σ0 = 2 px, σ1 = 10 px.
3 · A hidden “was perception right?” per field
If the field is correct, the query should match it; if not, fall back to a background likelihood instead of vetoing. Summing logs over fields gives the retrieval score.
P(qk|ek) = ĉk sk(qk, ek) + (1 − ĉk) bk(qk)
A first-order Taylor expansion around ĉk = 0 gives a linear surrogate, Score = Σk wk(Q) ĉk(E) mk(E, Q), within 0.002 Recall@5 of exact inference on CAVIAR.
4 · Calibrate before you trust
Raw detector and classifier scores are not probabilities. Isotonic regression, fitted with two-fold cross-fitting, maps them to calibrated ĉk so that fields are comparable.
ECE, location: 0.552 → 0.000 · action: 0.113 → 0.018
AUROC 0.811 for action confidence: it ranks correct above incorrect predictions.
03
04
05
06
Every interface that acts on perception, from gesture input to smart glasses, faces the same question this paper answers for retrieval: what should a system do with an observation it is only partly sure of? Here the answer is to carry calibrated uncertainty through to the decision instead of thresholding it away early. I build interactive systems on the same principle: Iris pays only above a similarity threshold and explains every refusal, and SeatScout holds a seat in an Away state instead of freeing it on one missed frame.
@inproceedings{ho2026probabilistic,
title = {Probabilistic Structured Event Modeling for Video Event
Recognition under Perceptual Uncertainty},
author = {Ho, Nhan Thanh and Nguyen, Anh Dong and Tran, Thuan Duc and
Nguyen, Thang Van and {Bui Anh Thinh} and {Nguyen Bao Tran} and
Ho, Long Nhat},
booktitle = {Asian Conference on Computer Vision (ACCV)},
year = {2026},
note = {Accepted}
}ICISN 2026International Conference on Intelligent Systems and Networks · accepted
Problem
Loop detectors measure traffic accurately but only on a few roads. Taxi GPS covers almost every road but is noisy and biased. Most road segments have no ground-truth label.
Idea
Normalise both sources into 24-hour patterns. Give every unlabelled segment a pseudo-label from the five loop segments whose daily pattern it resembles most, fuse it with the taxi profile, and learn on the road graph.
Result
On 838 road segments in San Francisco, hotspot@50 = 0.92 and hotspot@100 = 0.97, above every GNN baseline in hotspot ranking.
My part
Co-first author. I built and trained the graph models (the fusion GraphSAGE and multi-scale GNN, and the baselines they are compared with), and ran the evaluation: hotspot@k metrics and the hotspot maps.
01
Loop detectorsLargeST, Greater Bay Area · 2,352 sensors · accurate, sparse
Taxi trajectoriesSFMTA taxi trips · dense, noisy
24-hour patternsnormalise, map-match to segments, drop irregular days
Pseudo-labelstop-5 loop segments by cosine similarity, weighted average
Fusionỹ = 0.7 · pseudo + 0.3 · taxi
Graph learningGraphSAGE, multi-scale GNN over 838 segments
24-h flow + hotspotsR², hotspot@50, hotspot@100
Highlighted in green: the parts I built.
Similaritysi = cos(yiloop, pjtaxi), i ∈ N5(j)
Pseudo-labelyjpseudo = Σi wi yiloop, wi = si / Σk sk
Fusionỹj = α · yjpseudo + (1 − α) · pjtaxi, α = 0.7
02
Five loop-equipped roads with similar daily rhythms vote, weighted by similarity. The vote is then blended with the road's own taxi profile. Move α to see the trade-off between the stable sensor pattern and the road's own variation.
pseudo-label taxi profile fused label
Illustration with synthetic profiles, computed live with the paper's equations. The paper's data are the LargeST loop detectors and SFMTA taxi trips.
03
| Method | R² | hotspot@50 | hotspot@100 |
|---|
How I read it: fusion helps most where it matters for city operations, ranking the congestion hotspots. Fusion-GraphSAGE reaches hotspot@100 = 0.97 and the multi-scale variant hotspot@50 = 0.92, while plain GraphSAGE keeps a slightly higher R² (0.82 vs 0.81). The two fusion models are complementary: one is better at the very top of the ranking, the other over the wider set.
04
05
@inproceedings{ho2026hgnn,
title = {24H-GNN: A Pattern-Driven Pseudo-Label Fusion Graph Network
for City-Wide Traffic Flow Prediction},
author = {Ho, Thanh Nhan and Nguyen, Dong Anh and Pham Nguyen, Minh Tai and
Phan Tran, Minh Khue and Nguyen, Hoang Minh and
{Nguyen Bao Tran} and Le, Trong Nhan},
booktitle = {International Conference on Intelligent Systems and Networks (ICISN)},
year = {2026},
note = {Accepted}
}StatementDong Anh Nguyen · 2026
The systems we interact with increasingly act on what they perceive: a gesture, a gaze, a face, a scene. Perception is never certain, yet most interfaces turn a probability into a yes or a no long before the person is involved. I want to study what interaction looks like when uncertainty is carried all the way to the decision, and to build systems that behave well because of it.
On-body and egocentric interaction is where this matters most. Smart glasses, rings and hand tracking turn small, noisy signals into commands, often with no screen to show what the system understood. Three questions drive me:
I would answer them the way I have worked so far: build the system end to end, from the perception pipeline to the interface, then measure it, both with benchmarks and with people.
What if a glance could verify who you are?
Smart glasses that recognise their wearer by the iris, and use it to approve a payment. Before any hardware, I wrote the verification software as a small prototype to test the idea. Source on GitHub.
System at a glance
Eye imagecamera frame or sample, cropped to the guide box
Grayscale, 128×128OpenCV weights, bilinear resize
HOG16 px cells, 2×2 blocks, 9 bins, L2-Hys: 1,764-d
PCAfitted on the enrolled set, exported to JSON
Cosine matchbest enrolled identity; accept if ≥ 0.70
Pay or explainapprove, or say why not and what to do
Design rationale
The view through the glasses. The small frame is the camera facing your eye. Results here are simulated with sample photos, and nothing leaves this page.
Place your eye inside the frame
Start a simulated scan to pay.
Simulated prototype result: sample photos compared in your browser, not a live eye.
Concept: the glasses, the café and the payment are imagined. The comparison is real code running on photos. With your own eyes, the camera only turns on when you ask, and one frame is compared in your browser and then discarded.
Each photo becomes a list of numbers, and two lists are compared by the angle between them. Open a photo on the desktop below and the steps run on it.
The repository, laid out as a desktop. Drag things around. Open a photo to run it through the steps, or a file to read the real code.
A close-up from the webcam or an uploaded file.
Colour is dropped and the whole photo is squeezed to a small square.
HOG measures edges in 9 directions for every 16-pixel cell: 1,764 numbers.
PCA keeps the 3 directions in which the training photos differ most.
Cosine similarity scores the angle between two lists. Above 0.7, the prototype reports a match.
train_pca.py.Close to the saved model, not identical.Where to set the 0.7 line is also a design question. Set it lower and strangers pass more often; set it higher and the owner is refused more often. The demo compares edge features directly, without PCA, which gives higher scores, so its line is 0.90.
The PCA step shows the main limit. It was fitted on three photos, so it can spread them along only two directions, and the third holds nothing.
PCA was fitted on three images, so it has only two useful directions.
It was never tested on enough people to know how often it lets a stranger in or refuses the owner.
Lashes, eyelids and reflections outweigh the iris itself in the edge features.
Nothing stops a printed photo of the owner's eye from being accepted.
An iris can't be changed like a password, so a refusal is hard to work around. It is also likely to happen in public, at a counter with people waiting. I expect people to blame themselves when it happens, but I haven't tested this yet.
The prototype's messages were written for me as the developer. Here is what I would show the person at the counter instead.
| The prototype says | The glasses should say | Reason |
|---|---|---|
| No match (Score: 0.62) | Could not verify identity. Nothing was charged. | States the outcome for the money first and keeps the score private. |
| Error: {e} | Part of your eye was outside the frame. Move it inside and try again. | Checks the framing before matching, so the message can name the fix. |
| Matched: anh (Score: 0.86) | Paid 45.000 ₫ to Cà phê Hạt. | The wearer cares about the payment, not the match. |
| A delete button in the sidebar that removes the person at once | Remove your eye from these glasses? The photo and features will be deleted. | Erasing a biometric should be easy and never accidental. |
For each person the prototype saves the original photo in data/images/ and the feature numbers in iris_features.joblib, both unencrypted. That is acceptable for a test and unacceptable for glasses. I would keep only the features, never the photo, store them encrypted on the device, and explain at enrolment what is kept and how to erase it.
Matching three photos shows the pipeline runs. It says almost nothing about accuracy on real people.
The threshold decides who pays for mistakes. If it is low, the shop loses money to strangers. If it is high, the wearer is turned away. Choosing it needs testing with people, alongside the maths.
Most of the design work I found was in the failure path: telling the wearer what happened to the money, and giving them a way to recover.