AI: Exploration of Cognitive Service with Computer Vision and Natural Language Processing

Background

Last week, I got the chance to share my learning about Cognitive Service – Computer Vision, Custom Vision as well as Language Understanding Intelligent Service (LUIS) at the Cloud AI Bootcamp @ Microsoft Singapore. By following the hands on lab, crowd had found it hardly to image how those pieces of technology can work along. Therefore, I make to demonstrate how to put Computer Vision and LUIS together, specifically after analysis using the console app developed in lab 1.1 Computer Vision and the train model on LUIS in lab 1.5. In this demonstration, we will be creating ASP.NET Core Project to link up both technologies and the scenario would be an user would like to request some photos (sample processed photo in lab 1.1) by typing human language, such as “Find mountain pics”.

Start…

  1. First, we need to create a ASP.NET Core Web API Project with API template with Visual Studio 2017.
    ASP.NET Core with API template
  2. Secondly, install Microsoft.Azure.DocumentDB.Core from the NuGet package manager. As this point of writing, it is version 1.9.1.
  3. Third,  create a Repository folder and a class DocumentDBRepository.cs for accessing Azure Cosmos DB which was done quite similar with the code in Cloud AI Bootcamp lab notes with some little tweaks. This class contains some functions that will create database or collection if it doesn’t exist. Also, it contains function that will read or find data from the database.
    public static class DocumentDBRepository where T : class {
      private static readonly string DatabaseId = "images";
      private static readonly string CollectionId = "metadata"; 
      private static readonly string Endpoint = "<your cosmosdb end point here>";
      private static readonly string AuthKey = "<your cosmosdb auth_key here>";
      private static DocumentClient client;
      
      public static void Initialize() 
      {
         client = new DocumentClient(new Uri(Endpoint), AuthKey);
         CreateDatabaseIfNotExistsAsync().Wait();
         CreateCollectionIfNotExistsAsync().Wait();
      }
      private static async Task CreateDatabaseIfNotExistsAsync()
      {
         try
         {
            await client.ReadDatabaseAsync(UriFactory.CreateDatabaseUri(DatabaseId));
         }
         catch (DocumentClientException e)
         {
            if (e.StatusCode == System.Net.HttpStatusCode.NotFound)
            {
               await client.CreateDatabaseAsync(new Database { Id = DatabaseId });
            }
            else
            {
               throw;
            }
          }
      }
    
      private static async Task CreateCollectionIfNotExistsAsync()
      {
         try
         {
            await client.ReadDocumentCollectionAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, CollectionId));
         }
         catch (DocumentClientException e)
         {
            if (e.StatusCode == System.Net.HttpStatusCode.NotFound)
            {
               await client.CreateDocumentCollectionAsync(
                            UriFactory.CreateDatabaseUri(DatabaseId),
                            new DocumentCollection { Id = CollectionId },
                            new RequestOptions { OfferThroughput = 1000 });
            }
            else
            {
               throw;
            }
        }
     }
     public static async Task<IEnumerable> GetItemsAsync(Expression<Func<T, bool>> predicate)
     {
         IDocumentQuery query = client.CreateDocumentQuery(
                    UriFactory.CreateDocumentCollectionUri(DatabaseId, CollectionId))
                    .Where(predicate)
                    .AsDocumentQuery();
    
         List results = new List();
         while (query.HasMoreResults)
         {
            results.AddRange(await query.ExecuteNextAsync());
         }
    
         return results;
     }
     public static IQueryable FindMatchingDocuments(string query)
     {
         var queryOptions = new FeedOptions() { MaxItemCount = -1 };
         return client.CreateDocumentQuery(
                    UriFactory.CreateDocumentCollectionUri(DatabaseId, CollectionId), query, queryOptions);
     }
    }
    
  4. Next, create a API controller under Controllers folder, name it as “ImageController” with the following code.
     
        [Route("api/[controller]")]
        public class ImageController
        {
            public ImageController()
            {
                DocumentDBRepository.Initialize();
            }
    
            private IEnumerable IndexAsync()
            {
                var items = DocumentDBRepository.FindMatchingDocuments("select * from Images");
                return items;
            }
    
            [HttpGet("{param}")]
            public async Task<IEnumerable> GetImages(string param)
            {
                string LuisEndPoint = "<obtain your endpoint from LUIS publish setting>";
                List images = new List();
                HttpClient _httpClient = new HttpClient();
                _httpClient.DefaultRequestHeaders.Accept.Add(
                    new MediaTypeWithQualityHeaderValue("application/json"));
                var response = await _httpClient.GetAsync(LuisEndPoint + param);
                if (response.IsSuccessStatusCode)
                {
                    string str = await response.Content.ReadAsStringAsync();
                    var a = JsonConvert.DeserializeObject(str);
    
                    foreach(var b in IndexAsync().Select(x => x))
                    {
                        foreach(var c in b.Tags)
                        {
                            foreach(var ent in a.entities)
                            {
                                if (c.Contains(ent.entity))
                                {
                                    images.Add(b.BlobUri.ToString()); //return Url of Images
                                }
                            }
                        }
                    }
    
    
                }
                return images.Distinct();
            }
        }
    
  5. Next, create a new Model folder and create two following model classes under that folder, LuisData.cs for deserializing the response from LUIS API and ImageMetaData.cs for the data from the Azure Cosmos DB that contains images’ details.LuisData.cs
        public class LuisData
        {
            public string query { get; set; }
            public TopScoringIntent topScoringIntent { get; set; }
            public List intents { get; set; }
            public List entities { get; set; }
        }
    
        public class TopScoringIntent
        {
            public string intent { get; set; }
            public double score { get; set; }
        }
    
        public class Intent
        {
            public string intent { get; set; }
            public double score { get; set; }
        }
    
        public class Entity
        {
            public string entity { get; set; }
            public string type { get; set; }
            public int startIndex { get; set; }
            public int endIndex { get; set; }
            public double score { get; set; }
        }
    
    

    ImageMetaData.cs

    public class ImageMetadata
        {
            public ImageMetadata(string imageFilePath, string fileName)
            {
                this.LocalFilePath = imageFilePath;
                this.FileName = fileName;
                this.Id = this.FileName; // TODO: Worry about collisions, but ID can't handle slashes.
            }
    
            public ImageMetadata()
            {
    
            }
    
            public void AddInsights(ImageInsights insights)
            {
                this.Caption = insights.Caption;
                this.TranslatedCaption = insights.TranslatedCaption; //Marvin
                this.Tags = insights.Tags;
    
            }
    
            [JsonProperty(PropertyName = "id")]
            public string Id { get; set; }
    
            public Uri BlobUri { get; set; }
    
            public string LocalFilePath { get; set; }
    
            public string FileName { get; set; }
    
            public string Caption { get; set; }
    
            public string TranslatedCaption { get; set; }//Marvin
    
            public string[] Tags { get; set; }
    
            public override string ToString()
            {
                return JsonConvert.SerializeObject(this);
            }
        }
    
        public class ImageInsights
        {
            public string ImageId { get; set; }
            public string Caption { get; set; }
            public string TranslatedCaption { get; set; }
            public string[] Tags { get; set; }
    
        }
    
  6. Let’s build and run now. Visual Studio should run it with browser with a localhost:portnumber/api/values by default. We shall then test our API by “find mountain pics” with http://localhost:portnumber/api/image/find%20mountain%20pics in order to retrieve some mountain pictures that we have processed during the labs.

The sample code can be found on GitHub @ https://github.com/hmheng/LuisCVAPI

 

Don’t forget to follow me @
Twitter: @hmheng
More slides @ SlideShare: https://www.slideshare.net/HiangMengHengMarvin
Blog: http://hmheng.pinsland.com

You may also like...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: