The" Visual gist System Contextual machine-vision system Low-dimensional representation of overall scene Gabor-filter outputs at multiple scales, orientations, locations Dimensionality reduction via PCA “ Feature vector for an image the gist of the scene Compute 12 x 30=360 dim feature vector Or use steerable filter bank 6 orientations. 4 scales averaged over 4x 4 regions==384 dim feature vector Reduce to -80 dimensions using PCA NPCA DIiva Torralba ICV 2001
Contextual machine-vision system • Low-dimensional representation of overall scene: – Gabor-filter outputs at multiple scales, orientations, locations – Dimensionality reduction via PCA Feature vector for an image: the “gist” of the scene – Compute 12 x 30 = 360 dim. feature vector – over 4x4 regions = 384 dim. feature vector – Reduce to ~ 80 dimensions using PCA The “Visual Gist” System Or use steerable filter bank, 6 orientations, 4 scales, averaged Oliva & Torralba, IJCV 2001
Low-dimensional representation for image context 80-dimensional representation Hardware set-up · Wearable systen Gives immediate feedback to the user Must handle general camera view Computer: Sony laptop Capable of wireless link for audience display Designed for utility, not fashion
Low-dimensional representation for image context Images 80-dimensional representation Hardware set-up • • Computer: Sony laptop • Wearable system – Gives immediate feedback to the user – Must handle general camera view – Capable of wireless link for audience display Designed for utility, not fashion…
Our mobile rig, version 1 tt 4 Kevin Murphy Courtesy of Kevin Murphy. Used with permission. Our mobile rig, version 2 Antonio Torralba (Courtsey of Antonio Torralba. Used with permission
Our mobile rig, version 1 Kevin Murphy Our mobile rig, version 2. Antonio Torralba (Courtesy of Kevin Murphy. Used with permission.) (Courtsey of Antonio Torralba. Used with permission.)
Experiments T Rooms and halls on gth floor of 200 Tech Square Outdoors Test Interior of 200 Tech square, gth floor(seen in training Interior of 400 Tech Square(unseen) Outdoors(unseen places) Goals Identify previously seen locations Identify category of previously unseen locations Ground truth 200 out street 400 Short street (Q,IvG) Specific location corridor 6b Location office 400/628 iice400627 recognition offe400625 for mobil elevator 40 elevator 400/ vision system kitche pen spice P(C,IviD Location category street Indoor/outdoor
Experiments – th floor of 200 Tech. Square – • Test: – th floor (seen in training) – – • – – Specific location Location category Indoor/outdoor Ground truth System estimate Location recognition for mobile vision system • Train: Rooms and halls on 9 Outdoors Interior of 200 Tech. Square, 9 Interior of 400 Tech. Square (unseen) Outdoors (unseen places) Goals: Identify previously seen locations Identify category of previously unseen locations