tin.ng

Task Scheduler in Real-time Operating Systems (Part 1)

2019-05-19T00:00:00+07:00

Overview

This article is part 1 of an overview series specifically about the Scheduler in a Real-time Operating System (RTOS). Examples of such systems can be found in almost anything from automotive application such as airbags, emergency breaks to avionics, and also infotainment, multi-media systems like video playback and QoS in web servers.

A real-time system is one in which the correctness of the system depends not only on the logical results of computation, but also on the time at which the results are generated.

This means that they both have to look at the output of the system and at the time at which the output was generated to determine whether an execution is successful.

Prerequisite

Assuming you have foundation knowledge about
- Programming, Big-O Notation, Algorithm, Interrupt and Computer Architecture
- Real-time Operating System (RTOS) knowledge would make it more comprehensive

More about type of Real-time Systems coming later.

(Optional) Schedule Simulation software SimSo which offer all Platform Compatibility (Python-powered)

Understand the needs for Scheduler

That said, in the heart of RTOS is the Scheduler and as the software complexity advances, the scheduling become more complex and thus expect extreme scheduling methods. For instance, there are only two Electronic Control Units (ECUs) in luxury class cars in mid-80s to control most of the Engine, Exhaust, etc. and by the time this article was written, the BMW 7-series models have as many as 150 ECUs¹ and average at 80 ECUs in total.

There are primarily two distinction of Real-time Systems, a missed deadline

In Hard Real-time: is catastrophic
In Soft Real-time: can lead to significant loss

One important concern in such a system like RTOS is the predictability of system behavior, meaning there should be no space for un-accountability, everything must be accountable and predictable. This is often achieved by either static or dynamic scheduling of Real-time tasks to meet their deadlines.

Taxonomy of Real-time Systems Scheduling Scheme. Classified by system type and scheduling mechanism

Task Structure

The Task Structure in RTOS includes the planning and the scheduling of workload on a processor and guarantees that the workload timeline is never violated. That workload can be quantified into discrete pieces, i.e. a Task. The functionality of a Task will be executed over and over again and thus differentiate between each iteration by an instance of a task, a.k.a. a Job.

A periodic task, which will release a new job every time units, is usually defined by

Period (P): time interval between releases of jobs in a task
- Side Note: in reality, this interval is not necessarily constant with the presence of jitter between task transition.
Execution time ( $E_t$ ): how long the job is
Deadline (D): the deadline of a task

Example of Task $T_1$ with $P=3$ , $E_t=1s$ on x-axis of timeslice $t=1$

Since there is a latest time at which a task must be completed, all tasks must never violate its deadline. Although, a deadline can be defined as absolute or relative in Hard Real-time or Soft Real-time respectively.

The longer the execution time, the more CPU resources are needed and if $E_t$ becomes too large, the CPU gets overloaded and it cannot handle all the tasks within the time limit.

Task Jitter

From the Task Structure, it is necessary to ensure the consistency between tasks and the amount of time it takes to complete current task and move on to the next task while not overloading the CPU. In reality, the transition is delayed due to various reasons: current task not completely finished, resource is not released and furthermore.

The total amount of time for such delay is called Task Jitter. Ideally, no jitter is desirable, and Hard real-time scheduling ensures that with extremely precise scheduling, leaving no wasted time between task transition or context switching.

NOTE: A Hard Real-time OS has less jitter than a Soft Real-time OS in general.

Example from this SO discussion clearly demonstrate the scenario where Task Jitter is inevitable.

A task should be started after 10ms, but for whatever reason, it is started after 15ms. In our example the jitter is 5ms.

All tasks would use no more than 100ms per scheduled time, a task that requires and uses 150ms would cause significant scheduling jitter.

Data Inconsistency

When the scheduler switches the CPU from executing one task to execute another, the resource from the current running task is released for another task, usually higher priority, to take control and this process is called context switching.

A Context Switch enables multiple tasks to share a single CPU by storing/restoring the state and context so that the execution can be preempted and resumed later without interrupting current progress.

The context/shared memory require special attention since switching tasks will inevitably causing data inconsistency where any task could alter the values or worse completely corrupt it. This is partly solved by creating a temporary lock on the resources, buffer snapshot or other mechanisms such as Semaphore (Signaling) or Mutual Exclusion (Locking) to maintain the correct values between tasks.

The O(1) Task Scheduler in Linux

Another reason for the needs of a Scheduler is efficiency when it comes to task priority, what instruction should run first which would require more computing resources to calculate the order. This is what the O(1) Task Scheduler introduced in Linux 2.6 trying to solve.

Essentially, there is a First-In-First-Out (FIFO) Active Runqueues and another FIFO Expired Runqueues, each with their own tasks and their priorties. All tasks are scheduled based on priorities inside the Active Runqueue and ran based on a pre-defined timeslice, and swap in the Tasks from Expired Runqueue once similar priorities from Active Runqueue is emptied.

The time it takes for the Linux Task Scheduler to find a task to execute depends not on the number of active tasks but instead on the number of priorities which is constant since there are only 140 priorities.

This makes the Linux 2.6 Task Scheduler an O(1) process because the time to schedule is both fixed and deterministic regardless of the number of active tasks.

Side note: Linux 2.6 Task Scheduler also supports preemption, that is prioritize when a higher-priority task is ready to run and re-schedules the lower-priority task. It also offers Dynamic Task Prioritization at runtime in addition with the Static Task Prioritization defined at compile time.

Static Scheduling

As mentioned, predictability for an RTOS is an important characteristic and Static Scheduling provides just that, the ability to schedule, optimize for the tasks at runtime with very low complexity. Although, this also means that every modification requires execution of the scheduling algorithm again.

Taxonomy of Static Scheduling binded with Priority Scheduling Scheme. Different methodologies include timeslice, periodicity, and priority

For Static Scheduling, all parameters of all the tasks are assumed knownn in which the Scheduler built a schedule based on this information. This characterstic makes it simplier to schedule by timeslice based on task arrival pattern since there will be no change throughout the schedules.

Despite being fixed at runtime, Static Scheduling can be implemented in Search Tree where the tree level is Time Unit and the tree depth is Period in the schedule.

Another applications can be Process Control System where all sensory and actuator data range of all tasks are known before hand.

Clock-Driven Scheduling

Clock-Driven Scheduling is a Static Scheduling mechanism that consist of the following characteristics:

Schedule is created off-line at compile time
Precedence is treated explicitly between tasks
Scheduler operates when timer expires, i.e. Timer interrupt

Following is example of Clock-Driven Scheduling with a timeslice of Scheduling time $t=1s$ and three tasks $T_1$ , $T_2$ , $T_3$ scheduled at compile time.

Task ( $T$ )	Period ( $P$ )	Execution Time ( $E_t$ )	Deadline ( $D$ )
$T_1$	3	1	3
$T_2$	4	1	4
$T_3$	10	2	10

Table of Schedule for 3 Tasks with their Execution Time, Period and Deadline

Before any scheduling, a concept of CPU Utilization is important since tasks with fast period and long execution time will occupy a large amount of CPU resources.

CPU Utilization is defined as the sum of all tasks’ Execution times $\sum E_i$ divided by their Periods $\sum P_i$ . If the sum is greater than 1, then the system is guaranteed not feasible.

$Utilization = \sum_{i=1}^{n}\frac{E_i}{P_i}$

For this example of 3 Tasks, $U = \frac{1}{3} + \frac{1}{4} + \frac{1}{5} = 0.78$ , and thus the system will not be unfeasible. It does not mean the system is feasible yet, this point will be discussed in the schedule plot.

On the following plot, the Scheduler generates and allocates Tasks with start-of-task one direction arrow ( $\uparrow$ ) and end-of-task small dot (.) at the end of Execution Time (filled range).

Task $T_1$ , $T_2$ , $T_3$ and Processor $P_0$ plotted on Scheduling time ( $t$ ) x-axis in 20 seconds. Plotted on SimSo Simulation

On the first line of Task 1, the timer interrupts the system at time 0 and is selected. The job of is then executed through out its and come to a stop at then the next task is up.
- Side Note: similarly the second and third lines of Task 2 and 3 respectively.
Since start-of-task is , it takes control next.
- Side Note: there is no concept of priority yet, the only determinants are which Task available and ready.
At time $t=3$ , the Scheduler selects $T_1$ eventhough $T_2$ also starts at $t=3$ . That is because Task 3 Period is $P_3=4$ and thus not executed until the next timeslice $t=4$ .

From the schedule plot, it seems all tasks executed promptly and the schedule is feasible. However, as mentioned on CPU Utilization, because of task jitter between transition or some jobs occur simultaneously, the schedule will go wrong even if there are available CPU resources because the schedule is static.

Acknowledgement

This series is heavily influenced by a Coursera online course aiming for Master’s student created by the EIT-Digital University in Turku, Finland as part of the on-campus course on Real-Time Systems as given in their Embedded Systems master curriculum.

https://www.embitel.com/blog/embedded-blog/automotive-control-units-development-innovations-mechanical-to-electronics ↩

MQTT protocol analysis for embedded systems and IoT applications

2019-04-28T00:00:00+07:00

Overview

MQ Telemetry Transport is a Client – Server Publish/ Subscribe messaging transport protocol and was initially named after the MQ series of IBM where the protocol was first developed for proprietary embedded systems in the late 90’s.

Nowadays, MQTT is considered as the protocol name, as opposed to be an acronym previously, and is the de facto standard by OASIS and later ISO.

Throughout the development of MQTT, ease of use was always the key concern and thus the protocol are said to be easy to implement on the client side and excel at transferring data over the wire since it is lightweight, simple, and a widely adapted open standard.

Ultimately, the goal for this type of protocol is to ensure the communication channel in constrained environments, e.g. Machine to Machine (M2M) and Internet of Things (IoT) applications where a minimal code footprint is desirable and a limited network bandwidth.

Prerequisite

A MQTT Client (for Desktop), e.g. MQTT.fx, a cross-platform client (Windows, macOS, Linux) built under the Eclipse Paho Project. Another option is Mosquito MQTT.
(Optional) Setup Mathwork free account for ThingSpeak & MATLAB Online

If you have already understand MQTT, well done ! Skip to the second part of the article for instruction on how to setup and observe MQTT in action or read more about how the packets transmitted over the TCP/IP protocol.

Note: From this point onwards, the term MQTT is referring to the MQTT Version 3.1.1 of the OASIS standard. Official PDF can be downloaded here.

Understand how MQTT works

The Pub/ Sub pattern of MQTT provides a decoupling mechanism between the Client and Server where the Broker acting as a medium between them (this can be extrapolated from the preview image).

Space Decoupling: Publisher and Subscriber do not need to know each other, i.e. no exchange of IP address and port.
Time Decoupling: Publisher and Subscriber do not need to run at the same time, e.g. sensor update at specific interval and only retrieved to end-user when needed.
Synchronization Decoupling: Operations on both components do not need to be interrupted during sending or receiving.

There are three different aspects of MQTT include basic concepts (Publish/ Subscribe, Client/ Broker), basic functionalities (Connect, Publish, Subscribe) of MQTT and basic features such as Quality of Service, Retained Messages, Persistent Session, Last Will and Testament, Keep Alive and more.

Specifically, these functionalities (CONNECT, PUBLISH, SUBSCRIBE, UNSUBSCRIBE and DISCONNECT, etc.) are defined in a MQTT Control Package Type and comprise the Fixed Header (bit 7 - bit 4), i.e. the protocol first header.

Fixed Header format

In general, the protocol has three different headers vary by the package type.

Structure of an MQTT Control Packet

Side note: Full list of Control Packet Types.

The Variable Header

A Variable Header includes a 2 byte Packet Identifier field and resides between the Fixed Header and the Payload. Although, not every Control Package requires the identifier field.

An example of Variable Header of the CONNECT packet with Protocol Length, Protocol Name, Protocol Level, Connect Flag, and Keep Alive fields.

For the above example, every bits encoded in hexadecimal format, e.g. Keep Alive field of 60 seconds ( $60_{DEC} = 3C_{HEX}$ ).

The Payload

A Payload is the final part of the packet and also optional depends on the Control Package type. In the case of the PUBLISH packet this is optional and commonly named as the Application Message, whereas it’s required in the SUBSCRIBE package and contain Topic Filter, Quality of Service.

An example of Payload of the CONNECT packet with *Client Identifier fields*

Quality of Service

MQTT Protocol ensures high delivery guarantees with 3 levels of QoS serve to the constraint of embedded system communication in previous discussion.

QoS 0: Guarantees a best effort delivery but not result.

Image courtesy of MATLAB MQTT API

QoS 1: Guarantees that a message will be delivered at least once.

Image courtesy of MATLAB MQTT API

QoS 2: Guarantees that each message is received only once by the counterpart

Image courtesy of MATLAB MQTT API

Beside that, MQTT also provides other mechanism for quality control.

Last will means in case of unexpected disconnection of a client all subscribed clients will get a message from a broker.
Testament and Retained message means that a newly subscribed client will get an immediate status update.

Implement MQTT for practical uses

Now that you have an idea of MQTT, let’s examine it in action using MQTT.fx to establish the connection and Wireshark to capture the packet.

In MQTT.fx Setting tab, select “MQTT Broker” under Profile section and fill in ThingSpeak Server info as above

Note: If you have a MathWork ThingSpeak/MATLAB account, sign in under User Credentials sub-menu (Refer to Prerequisite). Otherwise, leave at default.

Under tab Subscribe, use the below syntax to Subscribe any Public channel

channels/<channel_ID>/subscribe/json

Once you receive the channel feed, it means that

The connection was requested & Server Acknowledged Request
Subscription requested & Server Acknowledged Subscription
Message Published from Server to MQTT.fx client

This can be verified in Wireshark by select “Start Capturing Packet” icon in the Top bar and re-do every steps.

MQTT packet captured while using MQTT.fx to the Broker on ThingSpeak server

Side note: Use “MQTT” (protocol) or “1883” (port) as the filter for better analysis.

Under network layer section, it can seen that MQTT indeed working on top of the TCP/IP layer. Beside that, Wireshark provides deeper look into the protocol and exactly how the packet was formed.

Analysis of MQTT CONNECT packet headers and flags setup

This can breakdown into several headers as discussed, i.e. Fixed Header with Control Packet Type, Variable Header with Protocol Length/ Name, Connect Flags, Keep Alive fields and lastly the Payload with Client ID and Authentication.

Also, these information can be interpreted in raw hexadecimal format too.

Same MQTT packet in raw HEX code with Headers highlighted and Fields separated.

Tips: You can hover over the HEX code in Wireshark to display the MQTT field that matches in the control bar at the bottom.

By using any DEC-HEX decoder, these code will yield similar result as expected.

Analysis of the same CONNECT packet in Microsoft Excel. Grab it here

Note: In Excel, the built-in function CODE() convert character to ASCII and DEX2HEX() could seamlessly achieve the task and of course modify to analyze other MQTT packets.

ThingSpeak & MATLAB Application

The great thing about ThingSpeak & MATLAB combination is that it provides an easy-to-implement approach without any restriction of further improvement.

ThingSpeak panels provide basic data collection and visualization

Many tools can be found under Apps top menu on your ThingSpeak profile with two main categories: Analytics and Actions.

For instances, you could use ThingSpeak to capture the temperature from sensor via MQTT, then import to MATLAB using script for Data Cleansing or use a Widget ThingTweet to inform public on sudden temperature changes.

On the other hand, MATLAB also provides excellent MQTT Support Toolbox for both reading and writing data directly to the ThingSpeak database.

Connect to MATLAB Online and try it out !

% thingSpeakRead(<channelID>, <Name>, <Value>)
[data,time] = thingSpeakRead(12397)

% Or specific Fields with Output Format flag
thingSpeakRead(12397,'Fields',[1 4],'NumPoints',10,'OutputFormat','table')

% Now try to write to your private channel
% NOTE: Example Key Value

% Update multiple fields at once 
thingSpeakWrite(<channelID>, 'WriteKey', <write-key>,...
		[field1_val, field2_val, field3_val]) 

% Or automated process for batch authentication from multiple channels
thingSpeakAuthenticate(<channelID>, <Name>, <Value>)

Tips: Use help or doc prefix in MATLAB Console Window on instruction how to implement the commands and its parameters

An Example of Bi-Histogram from ThingSpeak data imported for a Transportation Tracking application.

Troubleshooting (Extra)

Authorization issues in MQTT.fx

Make sure that you type in User-ID and MQTT-API-key for fields <User Name> and <Password> of User Credentials Tab respectively.

Both can be found under Account $\rightarrow$ My Profile on your ThingSpeak profile.

Decimal-Hexadecimal Conversion

In case you could not access the Excel file for reading the content of a MQTT packet, try to use directly in MATLAB Online with the Dec2Hex built-in function.

Configure Graphical UI access to VMs on Google Compute Engine

2019-04-16T00:00:00+07:00

Overview

Google Compute Engine is an Infrastructure as a Service (IaaS) component of the Google Cloud Platform (GCP) and aiming to provide specific hardware and utilize the cloud environment for development as well as production.

With Compute Engine, an instance could be generated with a couple of clicks with “a custom Machine Types optimized for your specific needs”. This is a great tool alongside with Docker, Kubernetes, etc. and is a game changer in the software industry and definitely a must-have tool.

Prerequisite

If you have already created your own VM instance, well done ! This is the next part to setup a Graphical User Interface (GUI) for your virtual computer.

VM setups

There are two essential components required in order for this to work includes a desktop environment and the Virtual Network Computing (VNC) tool.

Both can be installed via the Command Line Interface (CLI) from our previous SSH session (If you forgot, don’t worry ! Refer to “how to SSH into your vm instance”).

Install Desktop Environment

A CLI is differentiated based on the type of OS image and commonly is a Terminal, Console or some specific application with Shell (an interface) on it.

sudo apt-get update
apt-get install xfce4

sudo <Admin Privilege> ; install, update <Command>

apt-get: Package Manager, Others: aptitude; yum

xfce4: Desktop Environment, Others: gnome ; KDE

Install VNC Client

apt-get install tightvncserver

tightvncserver: VNC Client, Others: realvnc-vnc-server

After successful installation, run command tightvncserver to initialize the server and identify the xstartup file in order to register installed desktop environment.

Register newly installed desktop environment in VNC server startup file.

sudo vim /home/user-name/.vnc/xstartup

vim: Text Editor, :wq to Save & Quit. Trust me you will need it !

Hint: The syntax is start<desktop-environment> and an ampersand & to link with the rest of the script.

Ultimately, follow the instruction to setup credential key, this will be the access password for the upcoming VNC session.

Network Connectivity

After you have successfully installed all required components, test run the VNC server and its connectivity:

tightvncserver
nc localhost 5901

Success initialization of the VNC Server. Port <5901> returns value meaning it is listening and RFB indicates our server is up and running.

nc net cat <Command>. More on the command manual with man nc

<5901>: Default VNC Connection Port (5800/5900), increment 1 for every new session (next, X:2 5902, X:3 5903, etc.)

RFB: Remote Frame Buffer, this means the port is up and listening

Note: Although the server is up, the port still need to be configured forwarding to the Internet (out of the VM instance, on GCP) for remote access.

Side note: On your local computer, you can try to test the connectivity to the instance (which will be unconnected since the VNC server port is not forwarding). Hint: nc <global-ip> <port>.

Configure port for remote access

Before configure any settings, always backup or at least write down every step so that it’s easier for you to trace back later. Also, turn off VNC server and close port tightvncserver -kill: X with X essentially the number of your session.

From the Compute Engine page, go to your vm tab - click on more (at the right end) and select view network details.

From the Firewall Rules page, select Create Firewall Rule on the top bar.

Modify these fields:

Name: default-vnc-server
Description: instance-name ; vnc-server
Source Filter: Select IP Range
Source IP Range: Type “0.0.0.0/0”, allow all incoming connection
Specified Protocol and Ports: Type “tcp:5901”, specify for VNC

Then leave the rest as default (modify only when you are sure what it does) and hit create. And voila now everything is running and establishing a connection !

Establish VNC Connection

The only step left is to find your VM External (Global) IP Address and access into it from your local computer.

External IP from the Developer console

Retrieve External IP Address of your instance from Compute Engine page on the Cloud console.

External IP from 3rd party provider

curl ifconfig.me

Note: This method contains security risk for obvious reason.

Once having the External IP of your instance, access to it using any of the available VNC Viewer for your computer OS.

Built-in VNC viewer for macOS

Every macOS computer contains a built-in VNC viewer and can be quickly activate via Finder shortcut ⌘ + K:

VNC session using native tool Screen Sharing on macOS 10.14.1.

Hint: The syntax is <protocol>://<global-ip>:<port>

Third-party VNC Viewer

VNC session using Jump Desktop on macOS 10.14.1. Others: VNC Viewer (RealVNC), Screen 4.

Troubleshooting (Extra)

Issue with Desktop Environment configuration

Re-check if the Default Session, i.e. which DE to start up, is correct in the Display Managers: lightdm, GDM

Forgot VNC Server Password (tightvncserver)

On vm instance, type vncpasswd into the CLI and follow instruction to reset the password.

VNC Server port not forwarding

Re-check the Firewall settings of your computer for issue such as SSH permission, SSH tunneling config.

Also, from Compute Engine page, click on your VM and select edit. In network tag, add the default-vnc-server tag or the equivalent name that you created before.

Jekyll Category and Tag Taxonomies for custom Collection

2019-03-17T00:00:00+07:00

Update

As of August $20^{th}$ 2019, Jekyll 4.0 release will support natively category and tag for custom collection. This means the work-around is no longer necessary. However, this tutorial is left here in case anyone still need it. For instance, static-blog hosts support only older Jekyll version.

Overview

The main purpose of this category / tag iterator is to create an index page, or a taxonomy. Brief review about Collection will be mentioned, but at this point, you should be able to build, host your own site and understand basic functionality of layout, include and post folders.

Quick View of the result

An alternative would be using the Ruby plugin jekyll-archives, although as mentioned earlier these additional / non-native plugin could be not supported depend on build on the server host.

Prerequisite

Define a custom Collection

In case you do not have a custom collection already, it can be quickly configured. In _config.xml, a custom collection of recipe is defined as below.

collections:
  category: recipe
    output: true

defaults:
  - scope:
      path: ""
      type: recipe
    values:
      layout: single

Usage in article YAML Front Matter

Once the configuration is defined, simply add it in your article. For example, this tutorial has the following tags defined in its Front Matter.

NOTE: YAML Front Matter is essentially everything inside the divider (3-dashes) at the beginning of your article.

---
categories: [Tutorial, Productivity]
tags: [Jekyll, Taxonomy]
---

Category Tag Iterator

It depends on your purpose to place the following code snippet in the correct place, bu typically this would be in file _layout/categories.html if you want to create an Index / Taxonomy page.

Collect all data from collection

Assign variable thisCollection for your custom collection data
Apply filter to keep only category
- group_by: Re-group field data by their category
- join, split: Order field data into an array with , delimeter
- map: Keep only data belongs to site.categories i.e. our interested categories.

NOTE: The | is the pipe special character from Liquid syntax.

{% assign thisCollection = site.portfolio %}	
{% assign allCats =  thisCollection | map: 'categories' | join: ','  | split: ','  | group_by: category %}

Array for unique values

This stage is not necessary but recommended, if you’re doing indexer, it is best to count duplicated values and later display that Category only once.

{% capture category_array %}
  {% for eachCat in allCats %}
    {{ eachCat.name }}|
  {% endfor %}
{% endcapture %}

Iterate all categories

Now it is essentially a nested for-loop to match all of categories of the current article (post.categories) with the one collected earlier from all articles of a collection.

{% for eachCat in allCats %}
  {% for post in thisCollection %}
    
    {% if post.categories contains eachCat.name %}

      {% assign thisCat = eachCat.name | append: '|' %}

      {% if category_array contains thisCat %}
        <!-- Apply your code here to display the category list -->
      {% endif %}

      {% assign category_array = category_array | remove: thisCat %}
    {% endif %}

  {% endfor %}
{% endfor %}

Final implementation for this website.

Troubleshooting (Extra)

Custom Plugin Jekyll Hook

Another approach is writing your own plugin with the Jekyll.Hook method inspired from this StackOverflow Discussion.

How to setup and remotely access to VM instances on Cloud

2019-03-05T00:00:00+07:00

Overview

Prerequisite

Follow the official documentations to setup your Google account and create a Google Cloud Platform Project with Billing Enabled.

Google Free Tier introduces a 12 months, $300 trial ¹ for new user to learn the basics with several limitations which will not be an issue for our setup.

Well ! To be perfectly candid, it does depend on how much resources you want for your computer, so make sure to verify it.

For part 2 of this tutorial: “Gain access to your VM instance”, make sure your computer have a version of Python installed, preferably Python 2.7, for the best compatibility with the Software Development Kit from GCP.

If you have already created your own VM instance, well done ! Skip to the second part of the tutorial for instruction on how to gain access to the instance.

Configure VM step-by-step

There will be two main steps:

Create and enable a VM on GCP Server
Install development tools for access using laptop’s terminal or simply access via web client

1. Create VM Instance: Image Option

First, access the Cloud Console of your GCP Project and proceed to the Navigation menu (top-left corner) to tab Compute - Compute Engine. There will be a button to create the instance at the top bar.

Four steps to navigate VM Instance Manager section. Hold to enlarge.

Now, it is advised that you left the settings as default in case you do not know what it does. The main settings for your instance are Name, Region, Specifications.

*Hint: You can monitor how much is the Operation Cost , both estimated in monthly and hourly*

Also, once you got familiar with the process, do explore the Instance Template section for a more robust process of generating virtual machines from pre-defined sets of template.

Boot Disk allows you to choose a variety of Operating System from Windows to Linux and even specific application image of SQL server built upon.

Once hit create, half of our tutorial is done. The next part is to gain access into your VM instance.

2. Access VM instance: SSH

The Easy Way: Web Client

By now you have got the instance up and running, as in the same tab Compute Engine - VM instances. In column Connect, simply expand and chose the appropriate option to initiate a session with your instance.

Enable Secure Shell connection using a browser window which result in a direct access to the command line of your instance.

If you are looking for a way to gain access using your laptop terminal, keep scrolling to next part. Otherwise, that is the whole point of this tutorial, congratulation !

Hint: You might want to have a quick look into the “View gcloud command” tab for our next approach.

The Not-So-Easy Way: CLI

For those of you want a CLI-based developing environment, the official SDK powered by Python from GCP is definitely a fine choice. Refer to the Prerequisite section at the beginning for Python installation requirements.

Note: The following instruction is specifically for macOS, however, majority of other Operating Systems have either similar installation process or conveniently built directly from a package manager. Refer to the official documentation for your OS specific Installation Guide

From that very page, download the package for macOS 64-bit (x86_64). Navigate to the downloaded location and extract the package. Ultimately, you can do this in terminal using the built-in tar command. Then execute the install Shell script.

tar -xvf google-cloud-sdk-241.0.0-darwin-x86_64.tar.gz

google-cloud-sdk/install.sh

Side note: The option flags stand for eXtract, Verbose (optional), Filename

After install successfully, execute the initialization command gcloud init and follow the instruction to set up account and verify the connection credential keys.

Re-initialize previous configuration, when you first initialize it will be different but will essentially defining these parameters

You are almost there !!! Run the previous “View gcloud command” to establish SSH connection to the VM instance and you are all set. FYI, the command has the format of

gcloud compute --project "YOUR-PROJECT" ssh --zone "YOUR-ZONE" "YOUR-INSTANCE-NAME"

Successfully established SSH connection to my Debian VM instance. Now it’s your turn !!.

Troubleshooting (Extra)

“Command Not Found” for missing Shell Path

export PATH="/usr/lib/google-cloud-sdk/bin:$PATH"

This is due to the current Shell not knowing the location of gcloud and requires a re-open of Shell, i.e. Terminal.

Relatively, looking into .bashrc/.zshrc for automatic loading shell command for gcloud if the issue was not completely resolved.

Lost/Forgot Credential keys

rm -rf /Users/YOUR-USER-NAME/.config/gcloud/*.db

This attempts to delete the stored access_token and credentials local database. Run gcloud init to re-initialize and set up for new credential keys.

As of the written time ↩

Embedded Systems and Communication Protocols

2019-02-26T00:00:00+07:00

Overview

Embedded system is a system that has software embedded into the hardware and is dedicated to a particular part of a computer application regarding one or many specific features.

Note: Embedded systems are commonly referred as microprocessor-based systems (MPU) or microcontroller-based systems (MCU). More on this later.

Understanding connection between Hardware and Software Interfaces. Source: RMIT VNM– EEET2490 (2017)

So, what is with the interests in embedded systems?

99% of newly manufactured microprocessors were designed for Embedded Systems, whilst only 1% left for the rest of all desktop processors out in the market.

Over 31 billion MCUs was sold in 2015 and forecast to grow to almost 50 billion in 2019.

$10.46 billion in Embedded Software Market size valued in 2015 and is anticipated to grow at over 7% CAGR from 2016 to 2023.

System Architectures

Before discussing different types of system, it is important to understand that a system essentially consists of data and instruction units which can loosely defined as human memory and brain respectively. From there, the idea is to access data, i.e. knowledge, to analyze and make instruction.

As previously mentioned, embedded systems are classified as MPU and MCU with primary differences which are important in design including application perspective, connectivity standpoints, power consumption and whether it is consumer electronic or industrial automation. More on that, detailed analysis by Product Managers from Microchip - a microcontroller manufacturer.

Beside that, their functionality are tabulated as below.

Microprocessor and Microcontroller specification and comparison. Source: ElectronicForum

That being said, depends on the specific architecture, the medium for such transmission can serve different purposes (data/code) as well as interface with the CPU, memory and I/O units.

Hardware components and transmission interfaces. Source: RMIT VNM– EEET2490 (2017)

Harvard and Von Neumann

The system architectures are commonly categorized essentially based on the design of their transmission interface. That is, a Harvard architecture with separate busses for data and instruction allowing simultaneous transfers on both busses. On the other hand, a Von Neumann with a single bus used for both data transfers and instruction fetches hence must be scheduled.

Design Concept, Memory Allocation, Computing Speed and Purpose comparison between architectures.

Communication Protocol

The aforementioned medium typically a pre-defined sets of signal (voltage) leveling which referred as protocol and is a way for electronic components to exchange information hence the communication part.

There are two typical types of Electronic Communication Protocols, classified as:

Inter System Protocol: Between (External) Interconnected Different Devices. E.g: USB, RS232, UART/USART.
Intra System Protocol: Between (Internal) Intraconnected Entities, Submodules. E.g: SPI, I2C, CAN.

Note: There is another classification (and will be discussed in another article): Serial and Parallel which can be inferred from their name, are transmitting the data sequentially or simultaneously.

Some of the protocols are commonly implemented in industrial automation include USB, RS485, CAN. Although, three fundamental serial communication protocols will be examined for the purpose of an introduction.

UART

Universal Asynchronous Receiver Transmitter (UART) Hardware Setup and Interface.

It is a 2-wire, full-duplex, asynchronous and one-to-one serial protocol. Meaning it requires only 2 wires:

Rx: Receiver
Tx: Transmitter

to receive AND transmit at the same time (Duplex) while using their own CPU clocks, hence asynchronous and is typically used in scenario with two devices communicate to each other.

Note: Commonly the 2-wire could be replaced with any general purpose input/output (GPIO) pin, a process supported by software library emulating the similar process at the hardware level, i.e. voltage. Hint: Software Serial.

SPI

Serial Peripheral Interface (SPI) Hardware Setup and Interface.

It is a 4-wire, full-duplex, synchronous and one-to-(at least one) serial protocol and follow the master-slave interface scheme. Meaning it requires at least 4 wires:

SS: Slave Select (aka. CS: Chip Select)
SCLK: Serial Clock (aka. SCL)
MOSI: Master Output, Slave Input
MISO: Master Input, Slave Output

to receive AND transmit at the same time similar to UART with a designated CPU clock and a specific selected slave all of which initiated by the one-and-only master, hence synchronous and is usually implemented for scenario with only one master broadcast data, not necessarily identical, to different destinations.

Note: The number of SPI-slaves is technically unlimited, however, realistically it is depend on the number of Slave Select (SS) pins, i.e. usually from 2 to 4 pins, and shift-register scheme for sequential interconnected slaves. More on that, SPI Direct and Daisy linked chains explanation from Maxim Integrated (Advanced).

Side note: SPI is a trademark of Motorola and later became the standard but not to be confused with Texas Instrument interpretation of SPI.

Definitely not when working with DSP application (E.g. TMS320C/OMAP-L series), it’s a painful memory.

I2C

Inter-Integrated Circuit ( $I^2 C$ , I-squared-C ) Hardware Setup and Interface.

It is a 2-wire, half-duplex, synchronous and (at least one)-to-(at least one) serial protocol and also follow the master-slave interface scheme similar to SPI. Meaning it requires only 2 wires:

SDA: Serial Data
SCL: Serial Clock

to receive OR transmit sequentially, i.e. scheduled (half-duplex) different than UART or SPI. In spite of that, the beauty of I2C comes from both the world of UART and SPI with only two wire, multiple slaves, synchronous and especially multiple masters.

Note: On a single bus (1 byte - 8 bits), I2C theoretically supports $2^7 = 127$ different addresses (slaves) which is a huge advantage over SPI with the Slave Select limitation dilemma. Realistically, the address of each slave is usually pre-defined under hardware level and conflict is inevitable, i.e. much less than 127 addresses.

Side note: SPI is proprietary product of Phillips and as such Two-Wire Interface (TWI) is another variation of it which introduced by Atmel as more of an open standard and is almost identical in term of interface and protocol.

Advantages / Trade-offs

Summary and comparison between Serial protocols. More on details.

How to setup and burn the Bootloader to any Arduino board

2019-02-02T00:00:00+07:00

Overview

The idea of using Arduino and how it accelerates the process of programming and uploading code to the board was explained in the Makefile tutorial, check it out if you have a vague idea of what is the ultimate goal here.

The concept of Bootloader is similar to the BIOS of a computer if you have ever heard of. The Basic Input/Output System (BIOS) on your computer typically waits for a user input at startup (the ubiquitous “Press F1 to continue, Del to enter SETUP” screen) and provide options to modify Boot settings, i.e. CPU clock, peripherals, etc. and then proceed to load the Operating System (OS), e.g. Floating flag of Windows or an apple in macOS.

The bootloader program is written in the bootloader section, and the application program is written in the application section. Source: ElectronicWings

Here, in the embedded system, Bootloader is an initial piece of code (usually 512 bytes) which runs whenever a microcontroller is powered up or resets. It also looks for external inputs, i.e. uploading new program. Similarly, in case of no input, it executes the program that was uploaded previously.

Prerequisite

Arduino IDE: Only the IDE is enough, if you want to know other toolchains, look into previous instruction
Two (2) Arduino boards: One as the target board and one as the programmer board
USB-AVR-ISP (Optional): Advance, Bootload without the need for another programmer board

Understanding Bootloader

Arduino board implements AVR microcontrollers which has program memory and bootload memory sections. They are used to simplify the process of uploading programs to the microcontrollers as well as initializing IO devices before they begin the main application program, e.g. Initialize Analog-to-Digital-Conversion (ADC) pins.

Essentially, the bootloader is a .hex code and it initially watching dedicated pins for a new program to be uploaded, then using simple serial communication (in case of Arduino Uno, UART. More about the protocol) to download the program and write it into the application flash memory.

Arduino Uno (ATmega328p) Memory Map. Source: Arduino.cc

Luckily, Arduino provides you a built-in written Bootloader program within the IDE serving for this exact purpose at “File $\rightarrow$ Examples $\rightarrow$ ArduinoISP”.

Bootloading any board

Now with the Bootloader program ready, the rest is to physically setup a way for the programmer board and target board to communicate.

Circuit Examination

The setup utilizes Serial Peripheral Interface, a synchronous four-wire communication protocol with master-slave scheme, i.e. the programmer-target board. More about SPI protocol here.

Beside power pins, peripheral pins such as ADC, GPIO on any Arduino board, another set of pins called In Circuit Serial Programming (ICSP) header. These will give direct (serial) access to the main microcontroller on the board and will be the entry point.

From top left: *Master In Slave Out, +VCC Positive Supply, Serial Clock, Master Out Slave In, Reset and *Ground**

Side note: Read more about the functionality of ATMega16u2 on Arduino Uno.

Not only that, for some boards such as the ubiquitous Arduino Uno or Mega, they contains another set of SPI pins beside the ICSP header and are completely functional which can also be utilized for Bootloading.

It is important to notice that some only contain either and not both sets of pin, wire your board accordingly to either Digital pins scheme or ICSP header.

Hardware Setup

No matter how the Digital pins or ICSP headers were connected, the idea remains the same. That is, connect pins in SPI scheme from the Programmer board to that of the Target board in order to upload Bootloader on the microcontroller of the Programmer board to the one on the Target board.

It is also important to remember that, the power comes from the Programmer board which can originated from your USB port or an external supply and then will be shared with the Target board.

Bootloader procedure pin map using Digital Pin (applicable for Uno, Mega) Refer to the Pin Map in previous section. Schematic: Fritzing on macOS.

Note: In case of using a Programmer board of Arduino Mega, a 10 to $100 \mu F$ electrolytic capacitor must be placed (positive, long leg) between the Reset and (negative, short leg) Ground pins.

SELF CHECK

Arduino Mega as the Programmer board

This is to diminish the unwanted reset signal sent every time the Programmer attempts to establish connection with the Target in Bootload mode.

(Extra) ICSP Header Setup

Bootloading Procedure

Now that everything is clear, the procedure first took place with ONLY the Programmer WITHOUT any connection to the Target board.

Step 1: From Arduino IDE, Open the previously mentioned Bootloader firmware in the Arduino IDE in “File $\rightarrow$ Examples $\rightarrow$ ArduinoISP”.
Step 2: Select the model of Programmer board in “Tool $\rightarrow$ Board” (E.g. Arduino Mega) menu and the Serial Port in “Tool $\rightarrow$ Port” (E.g. /dev/cu.usbserial1410) menu.

Side note: Connected device port on macOS can also be identified using list command directly in Terminal, either ls /dev/cu.* or less common ls /dev/tty.* command.

Step 3: Select Upload button in the IDE and proceed to load the ArduinoISP sketch to your Programmer board.

Successfully upload Bootloader Firmware onto the Programmer’s microcontroller.

Now, the Bootloader is currently on the microcontroller of your Programmer board and is ready to be burned onto the Target board.

Step 4: Wiring the Target Board to the Programmer as in previous Hardware Setup, either ICSP header or SPI Digital Pins.

Side note: Also remember the capacitor for Arduino Mega which was explained previously.

Step 5: Back to the IDE, Select the model of Target board in “Tool $\rightarrow$ Board” (E.g. Arduino Uno) menu and the Programming Mode as Arduino as ISP in “Tool $\rightarrow$ Programmer” (Originally AVRISP mkII).
Step 6: Lastly, select the Burn Bootloader command, i.e. “Tool $\rightarrow$ Burn Bootloader”.

Successfully bootloaded a Target Arduino Uno (ATmega328p) using a Programmer Arduino Mega (ATmega2560).

Troubleshooting (Extra)

“avrdude: Yikes! Invalid device signature.”

This is usually due to mismatch in hardware interface as well as software configuration and is an indication of selected wrong Target board. There are some issues need to be taken into consideration:

Board Selection: Re-trace any step to select the correct board model, microcontroller variant (some have different architecture and signature)
Schematic Setup: Re-trace the setup for SPI protocol and/or the ICSP header pin map (if used)
Metallic Contact: Avoid any contact with the spikes on the back of the board. Trust me, I learned that the hard way !

Define Compiling workflow for Arduino sketch with Makefile

2019-01-18T00:00:00+07:00

Overview

This method is a wonderful bridge between beginner and intermediate in a sense from automated script from Arduino IDE to define your very own flags to feed the compiler. That is, before you just “need to” (or to be perfectly candid “only-able-to”) hit compile button.

After this, you will be able to comprehend a concept of Makefile and further working with Makefile in raw .c and .cpp format and even with other compiler, e.g. GNU C++ Compiler (GCC).

That being said, this approach utilizes pre-defined routine from Arduino IDE itself such as board variant, microcontroller architecture to monitor port on your computer and furthermore.

Prerequisite

An application running Shell (Command Line Interpreter) typically Terminal on macOS.
A Package Manager, typically homebrew (macOS) or apt-get, yum on a variety of Linux distros.
A C Compiler, specifically OSX-GCC Compiler if you are on macOS which will be installed along with the Xcode application.
A Python interpreter and Python-specific Package Manager: pip for Python 2.7 or pip3 for Python 3.

Don’t worry if you are missing any of those !! A comprised instruction can be found

INSTRUCTION HERE

Install homebrew package manager Copy and Paste this command into your Terminal to install homebrew.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Once installed, update the symlink of homebrew directory in .bash_profile

vim ~/.bash_profile
# Place this line into the profile
export PATH="/usr/local/bin:/usr/local/sbin:$PATH

:wq to Save & Quit; Trust me you will need it. Restart Terminal for changes effect.

Install OSX-GCC Compiler Still in Terminal, verify if you have XCode and its Command Line Tools yet.

/usr/bin/xcodebuild -version

If not, then proceed to install the Command Line Tools (necessary) or the entire application (optional)

xcode-select --install

Once installed, the compiler can be verified using command

whereis gcc
~ /usr/bin/gcc

Install Python Interpreter Some of the toolchains have reported discrepancies with Python > 3.5 . For the sake of this tutorial, Python 2.7 will be installed instead.

brew install python@2

Similarly, update the python@2 keg directory

vim ~/.bash_profile
# Place this line into the profile
export PATH="/usr/local/opt/python@2/bin:$PATH"

Toolchain Installation

In case you are an intrinsic newbie, definitely check out the Prerequisite instruction above. Once having all the application and dependencies, the main toolchains are required to setup.

Arduino IDE & AVR-G++ Compiler

Download the latest version of Arduino IDE from arduino.cc which includes the AVR-G++ Compiler. Drag & Drop the Arduino IDE into Application folder (important).

Once installed, the compiler can be verified by using command

/Applications/Arduino.app/Contents/Java/hardware/tools/avr/bin/avr-g++ --version

Note: The output should indicates compiler version and flag -v can be used to find more about it, e.g. configuration, target, thread mode and etc.

Makefile workflow

(Recommended) Using homebrew or apt-get to install directly with command

brew update
brew tap sudar/arduino-mk
brew install arduino-mk

Side note: As of the time writing this article, the arduino-mk package has already been in the default homebrew 2.1.2 inventory and does not need to tap anymore.

Another way is if you are familiar with GitHub, clone the directory sudar/Arduino-Makefile and extract the .ZIP into /usr/bin/Cellar.

PySerial tool

The Arduino IDE provides avrdude to work with uploading code to the board and by using Pyserial program, this process can be further automated.

(Recommended) Install using the pip, Package Manager for Python 2.7 package with the command.

pip install pyserial

Another way is to install it as a package with apt-get install python-serial.

Understanding Arduino

Now that every dependencies are satisfied, let’s take a look into how components on Arduino board and controller peripherals operated as well as compiler directive.

Hardware Level

At Hardware Level, an Arduino Board, e.g. Uno, Leonardo, Mega, etc.; typically contains a main AVR microcontroller (ATmega328p, ATmega32u4, ATmega2560 respectively) from Atmel and hence the controller names.

Beside that, a common microcontroller ATmega16u2 is usually in charge of convert between the computer’s USB port and the main controller’s serial port.

Arduino Uno and Hardware Interface between computer to the microcontroller.

This bridge is essential for uploading code from the computer to the main controller’s memory and communicating with it via serial protocol as discussed in the embedded system article.

Software Level

At Software Level, Arduino IDE provides a medium between the user and the AVR-G++ compiler. That is, it completes function pre-define, header, library linking and ultimately export machine-understandable directive, usually in .hex format.

The nature of an Arduino sketch is nothing else but C, C++ code and parameter directive as well as other “under the hood” declaration, for instance look at the way CPU clock speed defined below.

Hint: AVR-C expected param defined in the script whereas Arduino stored a default clock speed in a file and automated that for you (unless modified, of course).

Typical AVR C script and Arduino sketch depict the same behavior, blinking LEDs every 1 second

Note: A misconception should be cleared is that the Arduino programming language is essentially the C (and C++) code compiled and not a different language.

Makefile Flags

In the Makefile, some usual parameters are defined for the board, port, user Library and one important line to include the directive Arduino.mk file.

Hint: The full list of Arduino boards, AVR microcontroller can be found inside the Arduino.app or simply make show_boards and make show_submenu respectively.

BOARD_TAG    = mega
BOARD_SUB    = atmega2560

ARDUINO_LIBS = SoftwareSerial

MONITOR_PORT = /dev/*.usbserial*
MONITOR_BAUDRATE = 115200

include ../../Arduino.mk

Typical Makefile script for Arduino Mega board*

Note: Full list of make commands can be found in the arduino-mk-vars.md

Side note: On macOS, the MONITOR_PORT is either tty/cu and modem/serial depend on the OS version.

Routine Usage

The usage is quite straight forward with a few commands and a Makefile in the same directory as the Arduino sketch .ino.

These commands are pretty self-explanatory using a prefix make with flags:

Output of Compile-Only command make

Output of Compile-and-Upload command make upload using PySerial to trigger reset.

Establish serial connection command make monitor using native app screen on macOS 10.14.1.

Advance to AVR Development

If you are feeling confident enough, explores these new aspects:

Compiling plain AVR C programs: Also look up Atmel Studio for an IDE with troubleshooting.
Program using Arduino as ISP: Bootload firmware form one to another board (More on upcoming article).
Generate assembly and symbol files: Understand more of file linking and how compiler works.

Troubleshooting (Extra)

“Command Not Found” for missing Shell Path In case of homebrew or Python commands not found, try to refresh it

export PATH="/usr/local/bin:/usr/local/sbin:$PATH”
source /.bashprofile

Or better yet, look into ~/.bashrc instead.

Python 3 and Python 2 compatibility

Python > 3.5 is NOT backwards compatible and thus some of the dependencies might be newer than wanted. One solution is to switch using homebrew.

brew switch python 2.7.x

Also, make sure to use the correct commands in execution, i.e. pip/pip2 or python/python2.

Quick Service to copy code format and reserve syntax highlight

2018-12-23T00:00:00+07:00

Overview

This short tutorial aims to simplify the process of reserve the syntax highlight property of any code blocks and paste them into other application such as Microsoft Word, Keynote, etc. conveniently with System Service routines and couple of clicks.

Update: [2019-06-12] Visit this repo to get the available Service workflow or type

svn checkout https://github.com/tinng81/CopyHighlightService.git

in your Terminal app to directly download it.

Prerequisite

1. Install Python-powered Syntax Highlighter Pygments via Terminal app

# Python 2.7
pip install pygments

# Python 3.x
pip3 install pygments

pip/ pip3: Python Package Managers

Side note: Python 2.7 end of life is on January $1^{st}$ , 2020 and Python 3.x is backward-incompatible, i.e. you cannot keep using obsolete modules. It is highly recommended to install Python 3.x instead.

2. (Optional) In case you don’t have Python installed yet. Download Python 3 Latest Release here. Or follow the below script to install via command line.

#! /bin/bash

# One-liner install homebrew using default Ruby shipped with your macOS Version
mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew

# Then update the default PATH to use homebrew with your Shell
# Or other config file, E.g. '.bashrc'
test -f ~/.zshrc && echo "export PATH="/usr/local/bin:/usr/local/sbin:$PATH"" >> ~/.zshrc && source ~/.zshrc

# Verify successful installation
brew --version

# Install Python via Brew
brew install python

# Or specify a version
brew install python@2

# Lastly, install the Syntax Highlighter as in step 1
pip3 install pygments

Side note: Have a look into similar tutorials for Python installation and settings.

About Pygments

Python is an intepreted, high-level and without a doubt, the most common general-purpose programming language as of 2018.

And Pygments is simply a syntax highlighter suitable for code hosting, forums, wikis or other applications that need to prettify source code.

From the Terminal, use command pygmentize to initiate and provide input flags to process your code block. If you have experience with Python, head to the next section on how to use Pygment in Python Environment.

touch test.py && echo 'print("Success") # Just a comment' > test.py

pygmentize -f html -O style=friendly,full=True -o result.html test.py 

open result.html

touch: Create a file named test.py
echo: Insert ‘{string}’ of text to designated file > {output}. This would also create file (if not exist) when invoked
open: Open input file

Pygment flags explanation. Type pygmentize --help for a detailed manual.

-f: Format flag
-O: Option flags, comma separated values
-o: Output file (stdout)

Lastly, the test.py file was given as input (stdin) for the command.

Grab another example array.py and try it yourself.

Automate with Services

Now that you have an initial idea of how Pygments works, let’s try to automate copy code format with a couple of clicks.

Services provide convenient routines visibly on a right click. Screen captured on macOS Mojave 10.14

Essentially, Services are pre-defined routines that aim to automate tasks using AppleScript. Open Automator app via Spotlight using ⌘-Space Bar keystroke.

Step 1 – Select Quick Action workflow

Simply put, the flow is to Copy the selected text (Step 1) $\rightarrow$ Prompt for highlighting language (Step 2) $\rightarrow$ Set variable value (Step 3) $\rightarrow$ Run command to

Step 2 – Select Input as Text. On the left panel, search & drag Copy to Clipboard action

Note: The transition between Step 2 and 3. The Input(1) and Copy(2) actions are inter-connected (Above, middle), whereas the Copy(2) and Ask for Text (3) actions below are not connected.

Step 3 – Similarly, search & drag Set Value of Variable action, then select New Variable. This will serve as input for the final command

Ultimately, the whole workflow should look similar to this.

Step 4 – Finally, Add a Run Shell Script action, select your Shell and make sure it _Pass input_ as arguments

pbpaste & pbcopy: Command lines to provide text from Clipboard and add text to Clipboard. This setup serves our purpose of copy code blocks and later able to paste a highlighted version of it.

New Pygments flags are added as well as the full path in order to ensure all cases.

-g: Automatically detect Lexer from the content, i.e. the programming language.
-l: Manually input Lexer, i.e. the user-input in Step 3

Note: The -f formatter flag now utilizes rtf (Rich Text Format) instead.

export LC_CTYPE=UTF-8; 
pbpaste | /usr/local/bin/pygmentize -g -O style=colorful -f rtf -l  $1 | pbcopy

The command here has been revised and is a little bit different than the ‘raw’ one from previous section. Most importantly, the export command ensures that the copied text is in UTF-8 encode in order for this to work.

Command Line variant

As a bonus, if you’re more comfortable working with CLI, there is a quick function for you. Usage is simply as passing any file into your CLI, for instance highlight array.py from the previous example.

echo -e 'function highlight() {/usr/local/bin/pygmentize -g -O style=colorful -f rtf $1 | pbcopy}' >> ~/.zshrc
source ~/.zshrc

Again replace the config file based on your Shell, the default is usually bashrc*

⚠️ Important: Use of echo command with >> will append to file. Do not use >, this will replace all your text. I learned that the hard way !

The result will be copy onto your Clipboard and is ready to paste in any other application that support RTF.

Side note: With the -g flag enabled and file extension, there is no need to provide a Lexer as Pygments will automatically detect the language and highlight accordingly.

Explore highlight style

To find more about available color scheme, simply type python3 to access Python Environment or use Python3 IDLE if you use a Graphical UI application and the following code

from pygments.styles import get_all_styles
styles = list(get_all_styles())
print(styles)

Troubleshooting (Extra)

Command not found with brew and/or python

Make sure that you have installed successfully, try source ~/.zshrc to refresh your Shell config file or simply open a new window.

Command not found with Pygments

This is usually due to inproper installation of multiple Python version on your machine. Run

brew doctor && brew prune

to eliminate any empty keg and un-link unused dependencies.

Also, install Pygments with Python pip and NOT to be confuse with brew.

Study Matlab Script

2018-05-02T00:00:00+07:00

Workflow Overview

graph TB node1(Load Variables From Workspace) click node1 "#1" node1(Load Variables From Workspace) --> node2(Obtain 2-Point Statistics) click node2 "#2" node2(Obtain 2-Point Statistics) --> node3(PCA) click node3 "#3" node3(PCA) --> node4(Function Estimation) click node4 "#4"

Obtain the direct to Project-Pages publish function here.

Load Variables From Workspace

Loading the microstructures and the effective property.

load GDL200Volumes.mat
load MPL290Volumes.mat
load EffectiveProperty290MPL200GDLinThatOrder.mat

Obtain 2-Point Statistics

GG = zeros(490,49^3);

for i=1:290
    current = MPL{i};
    GGcurrent = TwoPointMaster('full','auto',25,current);
    GG(i,:) = GGcurrent(:)/numel(current);
    PP(i)=GGcurrent(25,25,25)/numel(current);
end

for i=1:200
    current = GDL{i};
    GGcurrent = TwoPointMaster('full','auto',25,current);
    GG(i+290,:) = GGcurrent(:)/numel(current);
    PP(i+290)=GGcurrent(25,25,25)/numel(current);
end

PCA

Via SVD:

[PC1,Var1,Basis1]=PCAConstruct(GG,10);
scatter(PC1(:,1),PC1(:,2),30,[ones(290,1);2*ones(200,1)],'filled'); colormap jet;

Function Estimation

Using Multivariate Polynomial Regression

reg = MultiPolyRegress(PC1(:,1:2),R,3,'figure')

reg = 

           FitParameters: '-----------------'
             PowerMatrix: [10x2 double]
                  Scores: [490x10 double]
    PolynomialExpression: [10x2 table]
            Coefficients: [10x1 double]
                    yhat: [490x1 double]
               Residuals: [490x1 double]
           GoodnessOfFit: '-----------------'
                 RSquare: 0.9572
                     MAE: 0.0571
                  MAESTD: 0.0561
           Normalization: '1-to-1 (Default)'
      LOOCVGoodnessOfFit: '-----------------'
               CVRSquare: 0.9556
                   CVMAE: 0.0583
                CVMAESTD: 0.0570
         CVNormalization: '1-to-1 (Default)'

Published with MATLAB® R2014b

tin.ng

Task Scheduler in Real-time Operating Systems (Part 1)

Overview

Prerequisite

Understand the needs for Scheduler

Task Structure

Task Jitter

Data Inconsistency

The O(1) Task Scheduler in Linux

Static Scheduling

Clock-Driven Scheduling

Acknowledgement

MQTT protocol analysis for embedded systems and IoT applications

Overview

Prerequisite

Understand how MQTT works

The Fixed Header

The Variable Header

The Payload

Quality of Service

Implement MQTT for practical uses

ThingSpeak & MATLAB Application

Troubleshooting (Extra)

Configure Graphical UI access to VMs on Google Compute Engine

Overview

Prerequisite

VM setups

Install Desktop Environment

Install VNC Client

Network Connectivity

Configure port for remote access

Establish VNC Connection

Troubleshooting (Extra)

Jekyll Category and Tag Taxonomies for custom Collection

Update

Overview

Prerequisite

Category Tag Iterator

Collect all data from collection

Array for unique values

Iterate all categories

Final implementation for this website.

Troubleshooting (Extra)

How to setup and remotely access to VM instances on Cloud

Overview

Prerequisite

Configure VM step-by-step

1. Create VM Instance: Image Option

2. Access VM instance: SSH

The Easy Way: Web Client

The Not-So-Easy Way: CLI

Troubleshooting (Extra)

Embedded Systems and Communication Protocols

Overview

System Architectures

Harvard and Von Neumann

Communication Protocol

UART

SPI

I2C

Advantages / Trade-offs

How to setup and burn the Bootloader to any Arduino board

Overview

Prerequisite

Understanding Bootloader

Bootloading any board

Circuit Examination

Hardware Setup

Arduino Mega as the Programmer board

(Extra) ICSP Header Setup

Bootloading Procedure

Troubleshooting (Extra)

Define Compiling workflow for Arduino sketch with Makefile

Overview

Prerequisite

Toolchain Installation

Understanding Arduino

Hardware Level

Software Level

Makefile Flags