Machine Learning for Mobile Poets - Meet the eMiLy app

Remember way back in high school, when your English teacher tried to engrave upon your memory the importance of appreciating 19th-century poetry? Or how about college, when you were encouraged to memorize the prologue of the Canterbury Tales so that you could enliven a cocktail party?

This actually happened for me, and it never fails to entertain - thank you, Professor Lynch!

You may be a developer, an accountant, or a football player now, but it's never wrong to sit quietly and ponder the words carefully chosen by a poet who wanted to inspire you and make you think. To celebrate National Poetry Month in August, I built a mobile app that might help jog your memory about a poet that you might have forgotten, or never heard about, but someone who evokes strong feelings in many folks who hold good memories from their literature classes. Let's talk about Emily Dickinson.

Why a mobile app? Poetry is a great medium to consume on a mobile device. Often compact, these literary gems can fit on a small screen. You can also leverage a mobile device's hardware capabilities to enhance your experience, like taking a selfie or using the accelerometer. Also I like building mobile apps.

Emily Dickinson was a poet active in the mid 19th century in Amherst, Western Massachusetts, on the East Coast of the USA. She was well known even in her life for being an eccentric recluse, sometimes only talking to visitors through closed doors. She was famous not so much for her poetry during her lifetime (much was published posthumously) but rather for her gardening.

How can we use new technology to bring the work of a 19th century poet back to our consciousness? In the eMiLy app, I designed an interface to encourage discovery, interaction, and creativity.

• From a simple random query, discover a poem of the day from an obscure collection.
• Use a slider to reflect the user's mood display a poem whose emotion correlates to the mood
• Take a selfie, and display a correlating happy, sad, neutral or anxious poem
• And finally, display poetry generated by a Recurrent Neural Network, based on Dickinson's literary work, to see if her meter and style can be imitated. Stay tuned for an explanation of this in a separate article.

Building the eMiLy app allowed me to explore and dismantle the preconcieved notions about Dickinson's poetry that remained from school, idées fixes about her rhyme, meter, mood, and the themes that she touched in her poetry.

Scaffolding a mobile app

My JavaScript framework of choice is Vue.js, and an excellent way to build a native mobile app using Vue is NativeScript-Vue. Community-built (thanks Igor!) it is supported by Progress's Open Source project, NativeScript. Install NativeScript on your local machine, following these instructions
and use it from the command line:

tns create my-new-app

You'll be prompted to select your framework of choice and a basic layout.

I usually choose a blank layout, and then create custom tabs where my pages will be injected. Once scaffolded, you can start working in your editor of choice.

//components/Home.vue

<template>
  <Page class="page" actionBarHidden="true">
    <GridLayout rows="*,auto" columns="*">
      <!-- main components all on top of each other, since only 1 will be visible at any given time -->
      <component
        v-for="component in componentsArray"
        v-show="component === currentComponent"
        :key="component"
        :is="component"
        row="0"
        col="0"
      />
      <GridLayout row="1" col="0" rows="*" columns="*,*,*,*">
        <!-- Bottom navigation -->
        <StackLayout row="0" col="0" :class="navigationButtonClasses('DailyPoem')">
          <Image @tap="currentComponent = 'DailyPoem'" src="~/assets/pencil.png" height="30" />
        </StackLayout>
        <!--more bottom navigation-->
      </GridLayout>
    </GridLayout>
  </Page>
</template>

<script>
import DailyPoem from "../components/DailyPoem";
import MyMood from "../components/MyMood";
import MoodSlider from "../components/MoodSlider";
import NewPoetry from "../components/NewPoetry";

export default {
  data() {
    return {
      currentComponent: "DailyPoem",
      componentsArray: ["DailyPoem", "MyMood", "MoodSlider", "NewPoetry"]
    };
  },
  computed: {
    navigationButtonClasses() {
      return component => ({
        "nav-btn": true,
        colored: component === this.currentComponent
      });
    }
  },

  components: {
    DailyPoem,
    MyMood,
    MoodSlider,
    NewPoetry
  }
};
</script>

The Home.vue single-file component is the heart of this app; each page is loaded into this manufactured tab layout to be displayed. Assuming Xcode and/or Android Studio is installed on your local machine, you can also emulate the app as you build it; hot module replacement is available to you and speeds up the process:

tns run ios/android

With your tabs in place, your bottom navigation ready to go, and your pages ready, it's time to build out the four screens of this app.

Dickinson Data

The first question faced when building an app based on literature or art is always where to source data. Fortunately, a database of poems exists with over 300 Emily Dickinson poems (out of over 1800; she was a prolific writer). By exporting them in Json format, you can build a poem-of-the-day screen for your mobile app. On the first screen, I display a randomly-chosen poem out of the dataset that I exported from poetrydb.org.

//components/DailyPoem.vue

<script>
import { mapActions, mapState } from "vuex";

export default {
  computed: {
    ...mapState(["poem"])
  },
  methods: {
    ...mapActions(["getPoem"])
  },
  created() {
    this.getPoem();
  }
};
</script>

Using Vuex, I'm able to control the app's state, showing a new poem on app refresh. You could change this to reflect the day, or even enhance the app to reflect seasonality. For now, I'm simply displaying a random poem from a group of 300 saved to Json, tidying up its format for display:

//store/index.js - this is my Vuex store

getPoem({ commit }) {
   let randomPoem = Math.floor(Math.random() * 300);
   let poemObject = poetry[randomPoem];
   let parsedPoem = JSON.stringify(poemObject.text);
   let selectedPoem = parsedPoem.split(',').join('\n');
        commit('displayPoem', selectedPoem);
}

In the DailyPoem.vue's template block, then, I use NativeScript's XML-style markup to show the poem:

<template>
  <StackLayout>
    <Label class="action-bar" text="Discover Emily Dickinson"></Label>
    <ScrollView class="card">
      <StackLayout>
        <Label horizontalAlignment="center" textWrap="true" class="title">Welcome</Label>
        <Image src="~/assets/logo.png" width="400px" />
        <StackLayout class="inner-card">
          <Label horizontalAlignment="center" textWrap="true" class="subtitle">My Daily Poem</Label>
          <Label textWrap="true" :text="poem" />
        </StackLayout>
      </StackLayout>
    </ScrollView>
  </StackLayout>
</template>

This is the simplest use case for showing a random poem in a screen. The result looks like this:

The Measure of Mood

Emily Dickinson's poetry seems to be best-known for being sad; if you ask a student about words that her poems evoke, you might hear "dark" or "gloomy". But was Emily's work universally sad? It's risky to try to evaluate the mental health of people long past their lifetime solely through their literary output, but it has been surmised that Emily might have suffered from seasonal-affective disorder, given the wide variation of mood that her poetry evokes depending on whether it was written in winter or summer.

To help answer the question of whether Emily's work is predominantly sad or happy, I made use of a quick text-mining project I created in Azure Notebooks. You can fork this project and mine your own texts for most commonly-used words; if I run it in one of Azure's free colabs, I find that the most common word is:

(Can you guess it?)

LIKE

I believe that this is because Emily makes such frequent use of simile; the soul is like a flower, the heart is like the sun...

Further analysis shows that she uses the word 'life' more often than 'death', and 'day' more often than 'night'. She draws heavily on natural imagery: 'summer', 'day', 'sun', 'nature', 'time', 'sea', 'flower', 'rose'. A little machine-aided text-mining helps to busting presuppositions about a poet's intention and overall mood.

Given that we now know that Emily's poetry ranges greatly in emotion, as do we ourselves, we can consider building an emotive experience for a mobile user based on her poems' sentiment. Let's display a happy or sad poem, based on the user's preference.

To do this, I built a slider - slide right for happy poems, slide left for sadder poems. But how can one tell which of Emily's poems are happy, and which are sad?

Microsoft cognitive services text sentiment analysis to the rescue!

Text can tell a lot about a user's sentiment. Designed for things like analyzing user feedback and restaurant reviews, the service can also be used - as I discovered - to get a measure of the sentiment of a given poem. Using the poetry that I had exported from poetrydb.com, I ran several poems through the service and saved the sentiment score to the Json object where I store the selection of poems used in the app.

Then, I built a slider for the user to manipulate:

//components/MoodSlider.vue

<Slider
    borderWidth="5"
    margin="5"
    color="#00416a"
    backgroundColor="#00416a"
    minValue="0"
    maxValue="1"
    :value="value"
    @valueChange="onValueChanged($event.value)"
/>

As the slider moves, the value to which it is bound also changes as does the color of the surrounding card:

//components/MoodSlider.vue

onValueChanged(e) {
      this.showCard = true;
      this.clearMoodPoem();
      this.value = e.toFixed(2);
      if (this.value < 0.5) {
        this.moodClass = "inner-card blue";
      } else {
        this.moodClass = "inner-card rose";
      }
    },

When the user taps a button, the Vuex store provides a poem with a similar sentiment score, based on the saved value in the Json dataset.

The Enigmatic Face

To further personalize the project, what if someone could take a selfie, have the sentiment of the face analyzed, and have a poem matched to your mood as depicted by your face's expression? Once again, turning to Microsoft cognitive services, we have a nice solution handy. Rather than relying on pre-processed data, this time we will send an API call to Azure and get back information about a face's sentiment. To try this API, upload a photo here.

To get this working, you need a cognitive service setup for and a key to access the provided endpoint.

The user will snap a selfie:

//components/MyMood.vue

async runFaceDetect() {
      this.clearSelfiePoem();
      this.complete = false;
      const imageAsset = await takePicture({
        width: 300,
        height: 500,
        keepAspectRatio: true,
        saveToGallery: false,
        cameraFacing: "front"
      });
      //process the asset
      const filePath = await this.getFilePath(imageAsset);
      const imageFile = File.fromPath(filePath);
      this.selfie = imageAsset;
      //send it to Azure
      this.result = await this.sendRequest(imageFile);
    },

and then the image file will be sent to cognitive services where the Face API is queried for the emotion facial attribute:

//components/MyMood.vue

sendRequest(file) {
      return new Promise((resolve, reject) => {
        const ses = session("image-upload");

        const request = {
          url:
            "https://emilyemotions.cognitiveservices.azure.com/face/v1.0/detect?returnFaceLandmarks=false&returnFaceAttributes=emotion",
          method: "POST",
          headers: {
            "Content-Type": "application/octet-stream",
            "Ocp-Apim-Subscription-Key": "provided-key"
          },
          description: "Uploading " + file.name
        };

        ...
      });
    }

The Face API returns an array of sentiments, and you can match the top returned value to the sentiments already saved in the .json dataset prior. Learn more about using the Face API here.

So far, the app shows a progression from displaying one poem through leveraging pre-processed sentiments to reading sentiments from users' faces and displaying a matching poem. In the next article, I'll discuss how I built the final screen of this app using poems generated by a Recurrent Neural Network, to see if it could detect, match and reproduce the 'ballad meter' often used by Dickinson. Stay tuned for the second part of this article series!

I hope that I've inspired you, if not to go back and read through Emily Dickinson's thought-provoking works, then perhaps to look through old literature and see how you can introduce it to new readers via the technology that we build every day. Find the assets for this work, including presentation slides and video and the full app codebase here.

Learn more about:
🤳 Face API
✍️ Text Analytics
👩🏻 Emily Dickinson