IoT: In the Cloud

Enter Your Project for a chance to win an Arduino MKR WAN 1300 Kit with a Compatable Lora Gateway or an MKR 1400 GSM with a Shield Combo + a $100 shopping cart!

Back to The Project14 homepage

Project14 Home
Monthly Themes
Monthly Theme Poll

 

Home Automation

Introduction

Home automation allows us to control household electrical appliances like light, door, fan, and so on. Home automation not only refers to reduce human efforts but also energy efficiency and time saving.

The aim of the project is to implement home automation using Raspberry Pi, Matrix Voice and Snips platform. In this project, Raspberry Pi is connected with Matrix Voice via GPIO pins and all the five lights (such as bedroom, kitchen lights) which we are controlling through voice command are interfaced with expansion GPIO pins of Matrix Voice. We are providing input(voice command) through Matrix voice mic and taking output (response) as light on/off from GPIO of Matrix Voice.

 

Component Needed

1. Matrix Voice ==> $55

2. Raspberry Pi 3 B+ ==> $35

3. 5V 2.5A Power Supply Adapter ==> $4.78

 

 

4. 5V Relay single channel ==> $4

5.16 GB SD card ==> $7.68

Total cost of the project is around $ 106

 

How to Connect Matrix Voice With RPi 3

The Matrix Voice is a development board for building sound driven behaviors and interfaces. It is an user-friendly tool for simple to complex Internet of Things (IoT) voice app creation. It comes under two versions

1. Matrix Voice version

2. Matrix Voice ESP32 version

Both versions of these boards run the same on a Raspberry Pi, however, the Matrix Voice ESP32 version has the option to run standalone by programming the ESP32 module.

Here I am using Matrix Voice.

 

      

 

Matrix Voice expansion GPIO

 

DF## pins are equivalent to GPIO## pins.

  

 

Raspberry Pi GPIO

 

 

Note:- The expansion GPIOs on the Matrix Voice are not connected to the RPi GPIOs. The expansion GPIOs mostly come from the FPGA.

 

Before installing Matrix OS make sure you have setup Matrix voice. For that you need

  • Raspberry Pi 3 Model B
  • +5V 2.5A Micro USB Power Supply
  • Matrix voice

1. Insert flashed micro SD card into Raspberry Pi

2. Attach Matrix Voice onto Raspberry Pi GPIO pins

3. Power Raspberry Pi with micro USB power supply

Once done next is installing Matrix OS on your personal computer. You can use Windows or macOS or Linux. In this project I am going to use Linux (Ubuntu) to install Matrix OS

 

Matrix core installation on RPi

 

Before starting installing make sure you have installed NodeJS and npm

1. To install NodeJS, go to NodeJS and download Linux binaries(x64) of version 10.15.1

and then extract it with

tar -xvf node-v10.15.1-linux-x64.tar.xz 
cd node-v10.15.1-linux-x64
sudo cp -R * /usr/local/
sudo apt-get update
sudo apt-get upgrade

and check NodeJS and npm version

node -v 
npm -v

 

 

1. Add the Matrix repository and key.

curl https://apt.matrix.one/doc/apt-key.gpg | sudo apt-key add - 
echo "deb https://apt.matrix.one/raspbian $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/matrixlabs.list
sudo apt-get update
sudo apt-get upgrade

2. Install the the Matrix Core packages.

sudo apt-get install matrixio-malos 

 

sudo reboot

 

 

3. To install ZeroMQ

echo "deb http://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/Debian_9.0/ ./" | sudo tee /etc/apt/sources.list.d/zeromq.list

wget https://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/Debian_9.0/Release.key -O- | sudo apt-key add

 

4. JavaScript setup

 

Create a node project folder in the home directory of RPi

cd ~/ mkdir js-matrix-core-app (whatever name you want) 
cd js-matrix-core-app
npm init

 

5. Installing npm Packages for ZMQ and Protocol Buffers

 

To install the ZMQ and MATRIX Protocol Buffers npm packages. Make sure you are in directory which you created above (name you give). Here I gave it js-matrix-core-app. This allows you to interact with Matrix Core through Node.js.

npm install zeromq --save
npm install matrix-protos --save 

 

Now you are done with Matrix core.

 

What is snips

Snips is an AI voice platform for connected devices. It is an end-to-end solution with Speech Recognition and Natural Language Understanding, enabling anyone to add a customizable voice assistant to their product. Snips is the only deep-learning based voice platform that runs fully on-device. It can run without the need for a server or the cloud. It offers Privacy by Design.

How to install the Snips using SAM CLI tool

For this make sure all the steps mentioned above are working

I. Raspberry Pi

1. To Install the matrix Kernel Modules. Run the following commands in your Raspberry Pi's terminal to add the matrix repository & key and update your repository packages.

 

curl https://apt.matrix.one/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.matrix.one/raspbian $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/matrixlabs.list
sudo apt-get update
sudo apt-get upgrade
sudo reboot

 

2. To Install the Matrix Kernel modules. Use the command below

sudo apt install matrixio-kernel-modules 

 

 

This allows the microphones on your matrix device to register as an ALSA microphone on your Raspberry Pi.

sudo reboot 

3. Next command will test your microphones by raising your output volume, creating a 5 second long audio recording, and then playing that recording.

Make sure you have select audio jack as output for that

sudo raspi-config 

then blue screen appears, go to advanced option then go to audio and select analog jack.

 

 

 

amixer set PCM 100% && arecord recording.wav -f S16_LE -r 16000 -d 5 && aplay recording.wav

 

 

If you are able to hear the audio then you are done.

Next we are going to install the SAM tool on desktop

 

II.Installing the SAM CLI tool on personal computer(Ubuntu)

 

We are using the SAM CLI tool to create, manage and deploy our Snips assistants, to configure hardware, to view the logs of assistants on the Raspberry Pi, as well as to run your application locally for quick prototyping.

1. Install SAM into your computer with the following command.

sudo npm install -g snips-sam 

2. Installing Snips.ai through SAM

We are going to install Snips.ai on our Raspberry Pi, Snips will be installed through the SAM CLI tool from our personal computer.

sam devices 

This command should log all of the Raspberry Pis on your network with their IP address and host name.

Note:- sam devices not required if you already know your RPi IP address

Connect to your Raspberry Pi using command

sam connect ip address of Pi 

eg :- sam connect 192.168.1.68

you'll then be prompted to insert a username and password of RPi.

 

Once you connected use the sudo init command, from your personal computer, to install Snips onto your Raspberry Pi.

sudo sam init 

 

III. Configuring Snips.ai on Raspberry Pi

 

Now, edit the snips.toml file for configuring the mics on RPi.

Open terminal and use command to open the snips.toml file

sudo nano /etc/snips.toml 

You can use any editor to do changes.

Scroll down to where you see [snips-audio-server] and replace

# mike = "Built-in Microphone" 
with the following:
mike = "MATRIXIO SOUND: - (hw:2,0)"

IV.Testing Snips.ai on personal Computer(Ubuntu)

 

Through your personal computer's terminal, you can use SAM to test the speaker and microphone of your Raspberry Pi.

 

1. To Test Raspberry Pi's speaker

sam test speaker 

 

2. To Test the microphones on your Matrix device

sam test microphone 

 

If both commands seem to work, you've properly configured Snips.ai on your Raspberry Pi.

If there is some problem use this command

sam setup audio 

Then it will ask whether you are using snips kit or not (I choose no)

Then it will ask you to select the microphone. Here I am using Matrix Voice microphone

Then it will ask you to select the speaker. Here I am using RPi audio jack as speaker.

To test Snips.ai is properly working by installing a demo assistant for the weather or not.

sam install demo 

 

 

After installing as you see snips assistant is running.

Now, test the assistant by saying

"Hey Snips!" "What will the weather be like in Madagascar in two days?" 

The response from Snips should be:

"You asked for the weather in Madagascar in two days." 

Congratulations we have successfully installed Snips on RPi.

 

Control lights with Snips.ai

 

1. Creating a Snips assistant & app

Sign into your Snips.ai and create an assistant. After sign in you page will like this

 

2. Create a new assistant

 

3. Give name to an assistant, select language and then create it.Once created you will see page like this

 

4. Then click on add an app and you will see page like this

5. Select create a new app and give it a name whatever you want and create it

Once created you will see page like this

 

 

6. After that open the app, click on edit app.

You will see page like this. As you see I already created intent (lights). You will see no intent on your page.

 

7. Create a new intent

 

After creating you will see page like this. My intent name is lights

 

 

8. Give training examples (you can give training examples after creating slots also).

9. Then create a new slots and give it name whatever you want. Here I am using three slots

  • switch for on and off state
  • room for different rooms such as bedroom, bathroom etc
  • device for devices such as light, fan etc

10. Then give slot type name. Here I am building custom slot type.

  • For room slot

 

After that add slot values "bathroom", "bedroom" and "kitchen".

 

  • For switch slot

 

After that add slot values "on" and "off".

  • For device slot

Give any name whatever you want.

 

After that add slot values such as "light", "fan" etc

11. Close the app, in order to tell the training example which slot is used, double click on off and on and select slot name(you provided earlier). Then save it and you can test it by deploying it.

 

Then save it and you can test it in window provided right hand side of the page.

 

Then deploy it

 

Connect SAM to Raspberry Pi on personal computer

 

1. From your computer's terminal, sign in through the Sam CLI Tool

sam login 

Login using your snips account

2. Connect to your Raspberry Pi. When prompted for a username & password, insert the Raspberry Pi credentials.

sam connect YOUR.PI.IP.HERE 

3. Copy the command and paste on your local computer.

After deploying on your Pi you will see like this

 

Creating Assistant on RPi

 

Assistant will be used to listen and respond to events from your Snips assistant

1. Go to the directory which we made during JavaScript setup in Matrix core. cd ~/js-matrix-core-app and create 4 new JavaScript files.

 touch assistant.js relay.js everloop.js func.js

2. Install MQTT to listen in on events from Snips.

npm install mqtt --save

3. Open everloop.js using any editorand paste the code given in program. Code is used to control the LEDs on the MATRIX Creator/Voice.

 

4. Open relay.js using any editor and paste the code given in program. It is used to turn the relay on and off.

 

5. Open assistant.js using any editor and paste the code given in program. It combines all the code from everloop.js & relay.js to control the everloop and relay switch through voice. It shows you how to listen to lightState intents and read slots.

var snipsUserName = 'ruchir'; // username of snips account 
var lightState = 'hermes/intent/'+snipsUserName+':lightState'; // name of intent you give while creating an intent

 

Once done now it is time to check.Open your personal computer(Ubuntu) terminal and run

sam watch

On the other hand, open RPi terminal and go to the directory

cd js-matrix-core-app/
node assistant.js  // run it 

 

Connections

Relay ==> Matrix Voice

  • 5V ==> 5V
  • GND ==>GND
  • Signal1 ==> GPIO0 (DF0)
  • Signal2 ==> GPIO1 (DF1)
  • Signal3 ==> GPIO2 (DF2)
  • Signal4 ==> GPIO3 (DF3)
  • Signal5 ==> GPIO5 (DF5)

 

 

Program

relay.js

/////////////
//VARIABLES//
/////////////
var zmq = require('zeromq');// Asynchronous Messaging Framework
var matrix_io = require('matrix-protos').matrix_io;// Protocol Buffers for MATRIX function
var matrix_ip = '127.0.0.1';// Local IP
var matrix_gpio_base_port = 20049;// Port for GPIO driver
var methods = {};// Declaration of method controls at the end
var relayPin = 0;// The GPIO pin connected to your relay
var roomname=['bathroom','bedroom','kitchen','living room','drawing room'];// array


///////////////////
//KEEP ALIVE PORT//
///////////////////
// Create a Pusher socket
var pingSocket = zmq.socket('push');
// Connect Pusher to Keep-alive port
pingSocket.connect('tcp://' + matrix_ip + ':' + (matrix_gpio_base_port + 1));
// Send initial ping
pingSocket.send('');
// Send ping & toggle pin value every 2 seconds
setInterval(function(){
  pingSocket.send('');// Send ping
}, 2000);
//////////////
//ERROR PORT//
//////////////
// Create a Subscriber socket
var errorSocket = zmq.socket('sub');
// Connect Subscriber to Error port
errorSocket.connect('tcp://' + matrix_ip + ':' + (matrix_gpio_base_port + 2));
// Connect Subscriber to Error port
errorSocket.subscribe('');
// On Message
errorSocket.on('message', function(error_message){
  console.log('Error received: ' + error_message.toString('utf8'));// Log error
});
/////////////
//BASE PORT//
/////////////
// Create a Pusher socket
var configSocket = zmq.socket('push');
// Connect Pusher to Base port
configSocket.connect('tcp://' + matrix_ip + ':' + matrix_gpio_base_port);
// Create driver configuration
var outputConfig = matrix_io.malos.v1.driver.DriverConfig.create({
 // Update rate configuration
  delayBetweenUpdates: 0.01,// 0.01 seconds between updates
  timeoutAfterLastPing: 6.0,// Stop sending updates 6 seconds after pings
  //GPIO Configuration
  gpio: matrix_io.malos.v1.io.GpioParams.create({
  pin: relayPin,// Use pin 0
  mode: matrix_io.malos.v1.io.GpioParams.EnumMode.OUTPUT,// Set as output mode
  value: 0// Set initial pin value as off
  })
});
////////////////////
//ON & OFF METHODS//
////////////////////
methods.lightsOff = function( pin1) { //pin1 is variable
  outputConfig.gpio.value = 0; //1
  outputConfig.gpio.pin=pin1;// 
  configSocket.send(matrix_io.malos.v1.driver.DriverConfig.encode(outputConfig).finish());
  console.log("Lights Have Been Turned Off ",roomname[pin1]);
};
methods.lightsOn = function(pin1) {
  outputConfig.gpio.value = 1; //0
  outputConfig.gpio.pin=pin1;
  configSocket.send(matrix_io.malos.v1.driver.DriverConfig.encode(outputConfig).finish());
  console.log("Lights Have Been Turned On ",roomname[pin1]);
};
module.exports = methods;

 

everloop.js

/////////////
//VARIABLES//
/////////////
var zmq = require('zeromq');// Asynchronous Messaging Framework
var matrix_io = require('matrix-protos').matrix_io;// Protocol Buffers for MATRIX function
var matrix_ip = '127.0.0.1';// Local IP
var matrix_everloop_base_port = 20021;// Port for Everloop driver
var matrix_device_leds = 0;// Holds amount of LEDs on MATRIX device
var methods = {};// Declaration of method controls at the end
var waitingToggle = false;
var counter = 0;
///////////////////
//KEEP ALIVE PORT//
///////////////////
// Create a Pusher socket
var pingSocket = zmq.socket('push')
// Connect Pusher to Keep-alive port
pingSocket.connect('tcp://' + matrix_ip + ':' + (matrix_everloop_base_port + 1));
// Send a single ping
pingSocket.send('');
//////////////
//ERROR PORT//
//////////////
// Create a Subscriber socket
var errorSocket = zmq.socket('sub');
// Connect Subscriber to Error port
errorSocket.connect('tcp://' + matrix_ip + ':' + (matrix_everloop_base_port + 2));
// Connect Subscriber to Error port
errorSocket.subscribe('');
// On Message
errorSocket.on('message', function(error_message){
 console.log('Error received: ' + error_message.toString('utf8'));// Log error
});
////////////////////
//DATA UPDATE PORT//
////////////////////
// Create a Subscriber socket
var updateSocket = zmq.socket('sub');
// Connect Subscriber to Data Update port
updateSocket.connect('tcp://' + matrix_ip + ':' + (matrix_everloop_base_port + 3));
// Subscribe to messages
updateSocket.subscribe('');
// On Message
updateSocket.on('message', function(buffer){
 var data = matrix_io.malos.v1.io.EverloopImage.decode(buffer);// Extract message
 matrix_device_leds = data.everloopLength;// Save MATRIX device LED count
});
/////////////
//BASE PORT//
/////////////
// Create a Pusher socket
var configSocket = zmq.socket('push');
// Connect Pusher to Base Port
configSocket.connect('tcp://' + matrix_ip + ':' + matrix_everloop_base_port);
// Create an empty Everloop image
var image = matrix_io.malos.v1.io.EverloopImage.create();
setInterval(function(){
  // Turns off all LEDs
  if (waitingToggle == false) {
  for (var i = 0; i < matrix_device_leds; ++i) {
  // Set individual LED value
  image.led[i] = {
  red: 0,
  green: 0,
  blue: 0,
  white: 0
  };
  }
  };
  // Creates pulsing LED effect
  if (waitingToggle == true) {
  for (var i = 0; i < matrix_device_leds; ++i) {
  // Set individual LED value
  image.led[i] = {
  red: (Math.round((Math.sin(counter) + 1) * 100) + 10),
  green: (Math.round((Math.sin(counter) + 1) * 100) + 10),
  blue: 0,// Math used to make pulsing effect
  white: 0
  };
  }
  };
  counter = counter + 0.2;
  // Store the Everloop image in MATRIX configuration
  var config = matrix_io.malos.v1.driver.DriverConfig.create({
  'image': image
  });
  // Send MATRIX configuration to MATRIX device
  if(matrix_device_leds > 0) {
  configSocket.send(matrix_io.malos.v1.driver.DriverConfig.encode(config).finish());
  };
},50);
///////////////////
//WAITING METHODS//
///////////////////
methods.startWaiting = function() {
  waitingToggle = true;
};
methods.stopWaiting = function() {
  waitingToggle = false;
};
module.exports = methods;

 

func.js

var everloop = require('./everloop.js');
var relay = require('./relay.js');
var methods={};
methods.roomtype = function(pin,state){
  if (state === 'on'){
  relay.lightsOn(pin);// passing arguments
  everloop.stopWaiting();
  console.log('light is turned On');
  }
  else{
  relay.lightsOff(pin);
  everloop.stopWaiting();
  console.log('light is turned Off');
  }
};
module.exports = methods;

 

assistant.js

/////////////
//VARIABLES//
/////////////

var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://localhost', { port: 1883 });
var func=require('./func.js'); 
var relay = require('./relay.js');
var everloop = require('./everloop.js');
var snipsUserName = ''; // your snips username
var wakeword = 'hermes/hotword/default/detected';
var sessionEnd = 'hermes/dialogueManager/sessionEnded';
var lightState = 'hermes/intent/'+snipsUserName+':intent name';
//////////////
//ON CONNECT//
//////////////
client.on('connect', function() {
  console.log('Connected to Snips MQTT server\n');
  client.subscribe('hermes/hotword/default/detected');
  client.subscribe('hermes/dialogueManager/sessionEnded');
  client.subscribe(lightState);
});


//////////////
//ON MESSAGE//
//////////////
client.on('message', function(topic,message) {
  var message = JSON.parse(message);
  switch(topic) {
  // * On Wakeword
  case wakeword:
  everloop.startWaiting();
  console.log('Wakeword Detected');
  break;
  // * On Light State Change
  case lightState:
  // Turn lights On/Off
  try{

   
var room=message.slots[1].rawValue;
var state=message.slots[0].rawValue;
var device=message.slots[2].rawValue;
switch(room){
// Bathroom light on/off
case 'bathroom':
  if(device=='light'){
  func.roomtype(0,state);
  console.log("Bathroom light");
  }
  else if(device=='fan'){
  func.roomtype(5,state);
  console.log("Bathroom fan");
  }

//case 'bathroom':func.roomtype(0,state);
//console.log("Bathroom light");
break;

// Bedroom light on/off
//case 'bedroom':func.roomtype(1,state);
//console.log("Bedroom light");
case 'bedroom':
if(device=='light'){
func.roomtype(1,state);
  console.log("Bedroom light");
}
else if(device=='fan'){
func.roomtype(6,state);
console.log("bedroom fan");
}
break;
// Kitchen light on/off

case 'Kitchen':
  if(device=='light'){
  func.roomtype(2,state);
  console.log("Kitchen light");
  }
  else if(device=='fan'){
  func.roomtype(7,state);
  console.log("Kitchen chimney");
  }

/*case 'kitchen':func.roomtype(2,state);
console.log("Kitchen light");*/
break;
// Living room light on/off
case 'living room':func.roomtype(3,state);
console.log("Living room light");
break;
// Drawing room light on/off
case 'drawing room':func.roomtype(4,state);
console.log("Drawing room light");
break;
// for turning all lights on/off
case 'all':func.roomtype(0,state);
  console.log("Bathroom light");
  func.roomtype(1,state);
  console.log("Bedroom light");
  func.roomtype(2,state);
console.log("Kitchen light");
func.roomtype(3,state);
console.log("Living room light");
func.roomtype(4,state);
console.log("Drawing room light");
}
  }
   
  // Expect error if `on` or `off` is not heard
  catch(e){
  console.log('Did not receive an On/Off state')
  }
  break;
  // * On Conversation End
  case sessionEnd:
  everloop.stopWaiting();
  console.log('Session Ended\n');
  break;
  }
});

 

Result

 

 

Training examples

  • turn on bathroom light
  • turn on the kitchen lights
  • turn off the bedroom lights
  • turn on all the light
  • turn on living room light

 

Update in May