If you have used Facebook at all in the past few years you are probably familiar with the app’s remarkable Artificial Intelligence capabilities. This is largely present in the face-recognition and friend tag suggestion abilities. Apparently, though, the company has improved this AI so that it is now even more impressive.
Indeed, Facebook’s director of applied machine learning, Joaquin Candela, shared, in a blog post, that Facebook can now recognize not only particular objects in your photographs, but also scenes and actions now, too. And this is true even within your photo posts that bear no words at all.
The software behind this update is Lumos, an artificially intelligent program that acts as the ocular sense to allow a computer to “see” the details of any image, even sans relevant text description. This machine learning engine is actually just part of several image-recognition features on Facebook, ranging from fighting spam to flagging nudity.
But today, this already spectacular system gets an upgrade that can recognize actions in addition to objects. In addition, the network’s alt text system—which is used as a proprietary means for describing a photo for users who are visually impaired—will now be able to recognize a dozen different actions. This could include walking and dancing to other actions than simply be described with a verb and an object (like “playing a guitar”).
Although the program was originally—and remains—intended to assist the visually impaired, the technology is available to all users, now providing the ability to search for an image of a specific place or action or even to find a particular article of clothing. This means, of course, that image searches will be even more accurate (as you can search for not only “objects” but combinations of actions and objects as well as object qualifiers (like “playing a black guitar”).
Candela goes on to notes that this idea will also continue to advance into automatically labeling videos down the road. He explains, “While these new developments are noteworthy, we have a long and exciting road ahead and are just scratching the surface of what is possible with a self-serve computer vision platform. With computer vision models getting pixel perfect and Facebook advancing into video and other immersive formats, Lumos will help unlock new possibilities in a reliable, fast, and scaleable way and pave the road for richer product experiences in the near future.”