Digistump Oak-Based System Monitor

Pro Micro Code

In my setup, the Oak is the I2C master, and the Pro Micro is the I2C slave. The Pro Micro handles mostly “middleman” tasks:

  • Accept and store report data via USB (Serial)
  • Transmit commands via USB (Keyboard)
  • Communicate with the Oak via I2C to get keyboard commands
  • Communicate the report data to the Oak, also via I2C

This doesn’t sound too bad, but there are plenty of gotchas despite the low number of lines of code to be written.

Arduino IDE Warning There are two Pro Micro boards in the Arduino IDE. One is for the 5v variant, and one is for the 3.3v variant. If you use the wrong one, the board will boot but fail to load, and you’ll get a “Device Not Recognized” error if you’re hooking up to Windows. There’s a fix for this on this SparkFun help page that involves hitting reset twice in a row quickly, which will give you a short window to upload new code.

I’ll start by explaining the loop() code. Remember again that all this code is up on GitHub:

void loop() {
  while(Serial.available()) 
    /* Read bytes of the report from the PC */
    byte byteRead = Serial.read();
    /* There are no newlines in the PC's report, but I might want 
     *  to test the program with the serial monitor.  So, treat a 
     *  newline as the end of the report.
     */
    if(byteRead == '\n') byteRead = 0;
    reportBuffer[reportBufPos] = byteRead;

    /* Is this the end of the report?  If so, calculate the size. */
    if(byteRead == 0) {
      reportBufPos = 0;
      reportLen = strlen((const char*)reportBuffer);
    }
    else if(reportBufPos<REPORT_BUFFER_SIZE-1) reportBufPos++;
  }
  /* If the serial port isn't ready, make note of that for our
   *  status.
   */
  if(Serial) serialState = 1;
  else serialState = 0;
}

The loop’s job is just to read from serial. I store the bytes read in a global buffer. If a newline or null byte is written, I consider that to be the end of the report. Then, I calculate the size for later when I need to transmit that info up to the Oak. Note that the Pro Micro doesn’t care what the data is, or what format it’s in. It’s just reading whatever is written over serial, and storing it for transmission via I2C.

I have some rudimentary logic here to keep the buffer from overflowing – if the PC writes too many bytes, it will just overwrite the last byte over and over again.

Next, let’s look at the requestEvent function, called when the Oak requests bytes over I2C:

/* This function is called when the Oak requests bytes over I2C */
void requestEvent() {
  byte dataTmp[33];
  int i;

  if(dataLenWritten == 0) {
    /* The first transaction is to get the size of the report.     
     *  This is a 2-byte transfer, so the max report size is
     *  ~65k bytes
     */
    int dataLenCount = 0;
    dataLenCount = reportLen;

    dataLenCount++;
    dataTmp[0] = (dataLenCount>>8)&0x0f;
    if(serialState) dataTmp[0] |= 0x80;
    dataTmp[1] = dataLenCount&0xff;
    Wire.write(dataTmp,2);
    dataLenWritten = 1;
    bufferWritten = 0;
  } else {
    /* We already wrote the size, write the report.
     *  The max I2C transfer supported by Arduino's 
     *  libraries is only 32 bytes, so we have to send
     *  the report in 32-byte chunks.
     */
    for(i=0; i<I2C_REQUEST_MAX; i++) {
      dataTmp[i] = reportBuffer[bufferWritten];
      if(reportBuffer[bufferWritten] == 0) {
        dataLenWritten = 0;
        i++;
        break;
      }
      bufferWritten++;      
    }
    dataTmp[i] = 0;
    Wire.write(dataTmp, i);
  }
    
}

The I2C transfer occurs in two phases. In the first phase, the Oak requests 2 bytes, and it wants the size of the report that the Pro Micro has buffered. After that has been written, the Oak requests a series of 32 byte (or less) transfers in order to get the report from the Pro Micro. The report has to be broken down into 32-byte transfers because of limitations in the Arduino Wire library, which unfortunately aren’t well documented.

One important thing to note for this function – there’s not a lot of room for doing anything but writing bytes over I2C! If you stop and do a Serial.print, or even just do a bunch of math in this function, the Oak will read back all 1’s (0xFF bytes) instead of the data you intended to write. I believe this is because the Oak has a software I2C implementation, and/or both sides do a bad job of clock stretching to indicate they’re not ready for sending/receiving data.

Design Decision Note that the Pro Micro is always the slave. It receives data from the PC on the PC’s schedule, and the Oak requests the report from the Pro Micro on its schedule. I haven’t run into any conditions where the PC is writing when the Oak is trying to read, but it’s possible to have this happen.

Design Decision The Pro Micro also has to store the entire report in its SRAM, and the Pro Micro’s SRAM is rather small. The limit to the buffer size is less than 2 kilobytes. This is plenty for my purposes, but if you wanted to transfer arbitrary size data, you might consider having the Pro Micro push the data across to the Oak as soon as it gets it from the PC. The downside to the approach is timing – there’s a very narrow timing window for I2C as I mentioned previously, so more experimentation and troubleshooting might be required for this approach.

The next important function is the function for when the Oak writes data to the Pro Micro via I2C:

/* This function is called when the Oak sends bytes via I2C
   These bytes represent keyboard commands to send to the PC */
void receiveEvent(int count) {
  while(Wire.available()) {
    byte data = Wire.read();
    if(data == 0) {
      /* if the byte is 0, reset everything */
      cmdByte = 0;
      dataLenWritten = 0;
      bufferWritten = 0;
      Keyboard.releaseAll();
      continue;
    }
    if(cmdByte == 0) {
      /* The first byte is the "command byte" in my I2C protocol.
         It is 4 bits of command and 4 bits of key modifiers. */
      cmdByte = data;
      dataLenWritten = 0;
    } else {
      /* The second byte contains only data, the keycode to send */
      byte command = cmdByte>>4;
      if(command == 1) {
        byte mod = cmdByte&0xf;
        if(mod&1) {
          Keyboard.press(KEY_LEFT_CTRL);
        }
        if(mod&2) {
          Keyboard.press(KEY_LEFT_SHIFT);
        }
        if(mod&4) {
          Keyboard.press(KEY_LEFT_ALT);
        }
        if(mod&8) {
          Keyboard.press(KEY_LEFT_GUI);
        }
        Keyboard.write(data);
        Keyboard.releaseAll();
      }
      cmdByte = 0;
    }
  }
}

If the Oak sends a null byte, then the Pro Micro effectively “soft resets.” This is useful for times when the two boards get out of sync. For instance, if the Oak reboots after sending a I2C request but before finishing it, the Pro Micro might be expecting to finish the current transfer, while the Oak assumes it is starting from scratch. The Oak, therefore, can send a null byte when it boots to ensure that the Pro Micro is in a good state.

The other reason the Oak wants to write data over I2C to the Pro Micro is to communicate a keyboard command. The Oak writes 2 bytes for every event. The first byte contains 4 bits of the command plus 4 bits of modifier keys. Right now, the only command is 0x1 = “send keyboard stroke,” but the protocol could be extended should there be anything else I want to do with this device. The second byte is the key code to send to the PC. For simplicity’s sake, I’m using the key codes defined in Arduino’s Keyboard.h throughout the stack.

Security Warning Take a minute to stop and think about this for a second. This system accepts keyboard strokes via WiFi and transmits them to your PC. That’s a huge security hole. We’ll talk about this more in the Oak section, but ideally you’d have the WiFi control link protected via SSL and a strong password. If you just want the system monitor (which is a lot less of a security risk), you can remove the keyboard code from this sketch and you’ll be a lot better off.

That’s most of the important Pro Micro code, so now we’re down to the Oak’s portion of the software. This is the most complex piece, though!

Digistump Oak Firmware

The Oak’s tasks are by far the most difficult:

  • Request and accept I2C data from the Pro Micro to get the report
  • Send commands from the web interface over I2C to the Pro Micro
  • Send HTML and JS to the client browser
  • Respond to many, many events the browser can generate

The Oak’s flash & SRAM size is big enough that I can store the HTML and JS as part of the image, so that’s what I’m going to do. If I needed more memory on the Oak, I could always set up a separate webserver to serve these files, but it’s easier and more self contained to store it on the Oak, in my opinion.

I’ll start with the Arduino/Processing code for the Oak, full code on GitHub, etc etc:


void getReport() {
  // This function requests the report from the PC,
  // which is cached in the Pro Micro
  int i = 0;
  i2cWrite(0);

  reportBuffer[0] = 0;
  delay(10);

  while(Wire.available()) {
    Serial.println(Wire.read());
    i++;
    if(i>50) break;
  }

  // We can't request bytes unless we know how many.
  // The first step is to get the report size from the Pro Micro.
  // Ask for 2 bytes of size
  Wire.requestFrom(I2C_ADDRESS_PM,2);
  if(!Wire.available()) {
    Serial.println("Error getting report length...");
    return;
  }

  byte hibyte = Wire.read();
  byte lobyte = Wire.read();

  // This is the report length
  int reportLen = (hibyte&0xf)<<8|lobyte;

  if(reportLen > REPORT_BUFFER_SIZE) {
    Serial.println("Report is too large!");
    return;
  }

  // There are limitations in the Arduino Wire library
  // that mean we can only ask for 32 bytes or less per transfer.
  // This loop breaks the report into 32 byte chunks and requests it
  i = 0;
  int requestLeft = 0;
  while(1) {
    if(requestLeft == 0) {
      if(reportLen < I2C_REQUEST_MAX) requestLeft = reportLen;  
      else requestLeft = I2C_REQUEST_MAX;
      if(requestLeft == 0) break;
      reportLen -= requestLeft;
      Wire.requestFrom(I2C_ADDRESS_PM, requestLeft);
      if(!Wire.available()) {
        Serial.println("Error requesting report chunk...");
        reportBuffer[0] = 0;
        return;
      }
      while(Wire.available() && requestLeft != 0) {
        reportBuffer[i] = Wire.read();
        i++;
        requestLeft--;
      }
      requestLeft = 0;
    }
    // Make sure we 0-terminate the buffer, so we can calcluate the size
    // using strlen()
    reportBuffer[i] = 0;
  }
}

This function is the Oak side implementation of the I2C report transmission, so it’s the other half of receiveEvent() from the Pro Micro code. The first half of the function requests the length of the report, as we saw in the receiveEvent() code. Then, it chunks the report into 32 byte requests (or less, in the case of the final request) and fills the buffer with the Pro Micro’s report.

void sendLargeBuffer(WiFiClient client, const char* buf, int bufLen) {
  // The ESP8266 web client can't handle writing buffers that are too large,
  // perhaps around 1400 bytes or so?  This function just breaks it up into
  // MAX_HTTP_SEND_SIZE chunks.
  int bytesWritten = 0;
  while(bytesWritten < bufLen) {
    int curLen = MAX_HTTP_SEND_SIZE;
    if(curLen > (bufLen - bytesWritten)) curLen = bufLen - bytesWritten;
    client.write(&buf[bytesWritten], curLen);
    bytesWritten += curLen;
  }
}

This is a helper function that breaks longer responses down into 1400 byte chunks. For whatever reason, the write() function can’t accept buffers much larger than this. It doesn’t throw an error or a warning or anything, it just doesn’t send the whole buffer. This seems like something that should really be fixed at the lower levels, or at least called out in bold in the docs. This function is pretty easy to write, though, so as long as you know there’s a limitation, you can work around it.

void sendReport() {
  // Send the report we cached from the PC
  WiFiClient client = server.client();
  server.send(200, "text/json", "");
  sendLargeBuffer(client, (const char*)reportBuffer, strlen((const char*)reportBuffer));
}

There are several handlers in the Oak code, this one is responsible for sending the report to the client’s browser. Note that I set the response type to text/json (since the report was encoded as JSON back on the Python code on the PC), and I’m using the sendLargeBuffer function from the previous code snippet.

void usbState() {
  // Get the state of the Pro Micro, based on
  // whether or not it is responding to i2c requests
  String response = "{\"r\":\"200\",\"state\":\"";
  response = response+pmConnected+"\"}";
  server.send(200, "text/json", response);
}

Some of the handlers write their responses on the fly, so I encode these by hand as JSON. There is an Arduino JSON library, but it is so very, very slow! For these tiny JSON responses, doing it by hand isn’t the end of the world. Escaping all the quotes is painful and hard to read, though. We’ll see another method of handling encoding data in arrays when we get to the HTML/JS browser code.

void resetPM() {
  long now = millis();
  if(resetTime+10000 < now) {
    resetTime = now;
    Serial.println("Resetting Pro Micro...");
    digitalWrite(PIN_PM_RESET, LOW);
    delay(10);
    digitalWrite(PIN_PM_RESET, HIGH);
  }
}

If you remember from back when I discussed the hardware setup, I mentioned that the Oak had a GPIO tied to the Pro Micro’s reset pin. This code resets the Pro Micro, but only does so if it’s been 10 seconds since the last reset. That gives the Pro Micro a bit of time to come online and get the report from the PC.

Next: The Embedded UI in HTML + JS

Prev Page 4 of 5 Next
  • Pradeep Kumar

    This post is awesome. Great work!!!….I am inpsired to chase down my project. I was one of the backer of Digistump Oak. I want to use oak to wake up my AVR from stand-by mode (for starters) and be able to control over wifi once I am familiar. I am just getting started. Your post motivates me to chase this down.

    • agent86ix

      This was quite the project! Unfortunately, I broke my Pro Micro a few weeks afterwards and haven’t fixed it yet. All that work, sigh… :(

      Good luck with your project!

      • I profited 104 thousand dollars in 2016 by freelancing on-line a­n­d I was able to do it by w­orking in my own time f­­o­­r several hrs daily. I’m using work opportunity I found on-line and I am thrilled that I was able to earn so much extra income. It’s so user-friendly and I’m so grateful that i discovered it. Check out what I did… http://statictab.com/h8vxywm