Live Text is perhaps the coolest feature in iOS 15, macOS Monterey and iPadOS 15. It pulls the words out of images and lets you paste them into notes, emails, etc.
I’ve tested this on both iPhone and iPad. Here’s why I find it so amazing.
Like so many people, I’ve gotten in the habit of taking a picture of anything I want to remember later. Live Text makes this even more useful. Rather than saving a whole image, I can turn the text in it into a note or a reminder.
For example, while out walking around, you can grab the name of a restaurant you’re interested in right off the sign. Use it to start a web search.
Or suppose the coffee shop has the Wi-Fi password written on a board. Grab it and paste it into Settings in couple of seconds.
And students can copy everything off a whiteboard as text and paste it into their notes. That should come in handy for people who fill up whiteboards during meetings, too.
How to use Live Text in iOS 15
The best part of Live Text is it’s so easy to use. There’s essentially no learning curve.
Consider an example. You find a recipe you like in a magazine. Take out your iPhone or iPad, open the camera app, and act like you’re going to take an image of the text. Wait a second or two and a small icon will appear in the lower right corner. Tap on it and a pop-up window will appear with the text in it. You can then select the words, sentences, etc. you want to copy.See a recipe you like? Accurately copy the text into Notes in just a few seconds.
Photo: Ed Hardy/Cult of Mac
The same process work with images in the Photos app. In this case, all the text will be highlighted in place, on the picture. You can then select the ones you want. Sometimes, the text icon doesn’t show up. In that case, tap and hold on the text you want to select. It’ll get selected unless the characters are too distorted.
And Live Text can often pull words out of images on directly web pages. In my testing, this option doesn’t work reliably, but I’m using iOS/iPadOS 15 beta 1. There’s time for improvement.
Live Text works almost shockingly well. When copying something printed, it’s often 100% accurate. Even under bad conditions, you can usually get almost everything.
And it works almost as well with handwriting. Even cursive. The accuracy depends on the text being reasonably clear, but it doesn’t have to be perfect — just not chicken scratch. I tried it with a couple of notes sent to me recently and the results were about 95% correct.
And translate, too
Live Text is likely to come in handy for travelers because it also ties in with Apple’s translation service. For me, the best option is to take a picture of the menu, then open it and tap and hold on a phrase to select it. A pop-up window gives the option to covert the words to English (or your preferred language.)
I did a bit of playing around with this and it seems reasonably useful. Translations aren’t perfect. But I’ve been in many restaurants in countries where I don’t speak the local language — I know any guide to figuring out what to order is welcome.
LiveText is OCR for everyone, everywhere
Live Text is, of course, just an application of optical character recognition, tech has been around for decades. But iOS 15, macOS Monterey and iPadOS 15 do it quite well. It’s convenient and very quick.
And because this is Apple, the process is done with privacy in mind. The word recognition process happens on your device. The image is not uploaded to a remote server. Which also means you don’t need a fast internet connection — or any connection at all.
When Apple releases its next operating system updates for iPhone, Mac and iPad, give Live Text a try. I think you’ll be as impressed as I am.