Build your first Machine Learning Model in iOS using CoreML

Core ML Sample App Overview

This app is pretty simple. Our app lets user either capture a image of something or select a image from their photo library. Then, the machine learning algorithm will try to predict what the object is in the picture.The result may not be accurate, but you will get an idea how you can implement Core ML in your app.

iOS Machine Learning

Machine learning is a type of artificial intelligence where computers “learn” without being explicitly programmed. Instead of coding an algorithm, machine learning tools enable computers to develop and refine algorithms, by finding patterns in huge amounts of data.

Let’s Start

Let’s begin! First of all go to Xcode 9 and create a new project. Select the Single View App option for this project, and make sure the language is set to Swift.

Making the User Interface

First of all select the Main.storyboard Choose the view controller in Main.storyboard, and then go up to the Xcode menu. Click Editor-> Embed In-> Navigation Controller.

Once you have done that, you should see a navigation bar appear at the top of your view. you can also gives any color and title to it.

Next, Select the View Controller and add some UI elements to the view. Add two Bar Button Item for Camera and Photo Library. These two buttons let users choose a image from his/her photo library or capture by using camera. The other two elements you need are a UILabel and a UIImageView. Then add required components outlets in View Controller.

Adding a Core ML Model Into Your App

This tutorial uses the Inception v3 model, which you can download from Apple’s Machine Learning page. Scroll down to Working with Models, and download that one.

Adding a Model to Your Project

After you download Inceptionv3.mlmodel,

drag it from Finder into the Resources group

in your project’s Project Navigator:

Implementing the Camera and Photo Library Functions

Now, let’s move onto the implementation. In ViewController.swift, first adopt the UINavigationControllerDelegate protocol that will be required by the UIImagePickerController class

import UIKit
class ViewController: UIViewController,UINavigationControllerDelegate{
    @IBOutlet weak var imageView: UIImageView!
    @IBOutlet weak var classifier: UILabel!
    override func viewDidLoad() {
        super.viewDidLoad()
    }
    override func didReceiveMemoryWarning() {
        super.didReceiveMemoryWarning()
    }
}

Next, you need to create the respective actions from clicking the both bar button items. Now insert the following action methods in the ViewController class:

@IBAction func Camera(_ sender: Any)
{
    if !UIImagePickerController.isSourceTypeAvailable(.camera){
        return
    }
    let cameraPicker = UIImagePickerController()
    cameraPicker.delegate = self
    cameraPicker.sourceType = .camera
    cameraPicker.allowsEditing = false
    present(cameraPicker, animated: true)
}

@IBAction func Library(_ sender: Any) {
    let picker = UIImagePickerController()
    picker.allowsEditing = false
    picker.delegate = self
    picker.sourceType = .photoLibrary
    present(picker, animated: true)
}

After that, Now we present the UIImagePickerController to the user. For take image (either from the photo library or the camera). For that we need to implement UIImagePickerControllerDelegate class method to ViewController.swift.

extension ViewController : UIImagePickerControllerDelegate
{
    func imagePickerControllerDidCancel(_ picker: UIImagePickerController)
    {
        dismiss(animated: true, completion: nil)
    }
}

To access your camera and photo library, there is still one last thing you must do. Go to your Info.plist and two entries: Privacy – Camera Usage Description and Privacy – Photo Library Usage Description. Starting from iOS 10, you will need to specify the reason why your app needs to access the camera and photo library.

Now, let’s add the model in our code. Go back to ViewController.swift. First, import the CoreML framework at the very beginning:

import CoreML

Next, declare a model variable in the class for the Inceptionv3 model, and initialize it in the viewWillAppear() method:

var model: Inceptionv3!
override func viewWillAppear(_ animated: Bool) {
    model = Inceptionv3()
}

Converting the Images

In the extension of ViewController.swift, update the code like below. We implement the imagePickerController(_:didFinishPickingMediaWithInfo) method to process the selected image:

extension ViewController : UIImagePickerControllerDelegate{
    func imagePickerControllerDidCancel(_ picker: UIImagePickerController) {
        dismiss(animated: true, completion: nil)
    }

    func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
        picker.dismiss(animated: true)
        classifier.text = "Analyzing Image...."
        guard let image = info["UIImagePickerControllerOriginalImage"] as? UIImage else {
            return
        }
        UIGraphicsBeginImageContextWithOptions(CGSize(width: 299, height: 299), true, 2.0)
        image.draw(in: CGRect(x: 0, y: 0, width: 299, height: 299))
        let newImage = UIGraphicsGetImageFromCurrentImageContext()!
        UIGraphicsEndImageContext()
        let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
        var pixelBuffer : CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(newImage.size.width), Int(newImage.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
        guard (status == kCVReturnSuccess) else {
            return
        }
        CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pixelData, width: Int(newImage.size.width), height: Int(newImage.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
        context?.translateBy(x: 0, y: newImage.size.height)
        context?.scaleBy(x: 1.0, y: -1.0)
        UIGraphicsPushContext(context!)
        newImage.draw(in: CGRect(x: 0, y: 0, width: newImage.size.width, height: newImage.size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        imageView.image = newImage
    }
}

Now, if you do not understand most of that code, no worries. This is really some advanced Core Image code, which is out of the scope of this tutorial. All you need to know is that we converted the image taken into something which the data model can take into. I would recommend you playing around with the numbers and seeing what the results in order to have a better understanding.

Using Core ML

Anyway, let’s shift the focus back to Core ML. We use the Inceptionv3 model to perform object recognition. With Core ML, to do that, all we need is just a few lines of code. Paste the following code snippet below the imageView.image = newImage line.

guard let prediction = try? model.prediction(image: pixelBuffer!) else {
    return
}

if let accuracy = String(format: "%.0f", ((prediction.classLabelProbs as? NSDictionary)?.value(forKey: prediction.classLabel) as? Double)! * 100) as? String{
    classifier.text = "I think \(accuracy)% this is a \(prediction.classLabel)."
}

Finally it’s time to test the app! Build and run the app in the simulator or your iPhone (with iOS 11 installed). Choose a image from your photo library or click a photo using camera. The app will tell you what the image is about.