Using the raspberry pi for video monitoring

The Raspberry PI is a good hardware for making a simple video monitoring device, such as watching at any time what's happening at home, if cats have enough to eat, chicken safe from fox attacks, or simply see at distance what's the weather like. The linphone console tools (linphonec and linphone-daemon) can be used to automatically accept a SIP call with video, so that it becomes possible from Linphone android or iOS app to call home at any time.

From a hardware standpoint, a raspberry pi2 is a minimum for a decent image quality. Indeed, video software encoding is a cpu consuming task that a pi 1 can hardly achieve as it doesn't have the NEON (multimedia) instruction set.

The PI's camera has excellent quality, however plugging a USB camera can be more interesting as it will provide sound recording thanks to its integrated microphone, which the raspberry doesn't have.

An ethernet connection is preferred, though wifi would work decently for video transmission as long as the raspberry is not too far from the wifi router.

Displaying the video received by the raspberry is out of this article's scope : we will just focus on the capture, sound and video, as well as the transmission via SIP to another SIP phone.

The following section explains how to setup the video monitoring on the raspberry pi. Several options are possible: 

• use of the python wrapper
• compilation on the raspberry to use linphone console tools directly

Prequisites

We recommend to use Raspbian, as a base installation to run linphone.

Since linphone heavily makes uses of dual-stack (ipv6 and ipv4) sockets, it is required to enable ipv6 on the raspberry by doing (as root):

echo ipv6 >> /etc/modules

And reboot.

To be clear, there is absolutely no need to have an IPv6 connection, the concern is only to enable the IPv6 APIs of the kernel (which are compatible with IPv4 operation of course).

Using the python wrapper to develop a simple script

First of all, you need to install the python wrapper for liblinphone API on your raspberry. You can follow the Linux instructions (there are raspberry pi specifics) from this page .

Here's a simple script (a bit more advanced than the sample from the Linphone Python Wrapper page).

Dependencies

In addition to the linphone python wrapper, this script uses daemon, so you'll have to install it first:

pip install python-daemon

Code

Here are the features:

• Obviously it will answer if it receives an incoming call from a whitelisted account ;
• If you send it a SIP message, it will take a capture from the camera and send it back to you as a message with an attachement ;
• Everything will be logged into linphonecam.log.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import linphone
import logging
import signal
import time
import daemon
import os
import sys

class SecurityCamera:
 def __init__(self, whitelist=[]):
   self.quit = False
   self.whitelist = whitelist

    callbacks = linphone.Factory.get().create_core_cbs()
    callbacks.call_state_changed = self.call_state_changed
    callbacks.registration_state_changed = self.call_state_changed
    callbacks.message_received = self.message_received

    path = os.path.dirname(os.path.abspath(__file__))
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
   self.logfile = logging.FileHandler(path + '/linphonecam.log')
    logger.addHandler(self.logfile)

    signal.signal(signal.SIGINT, self.signal_handler)
    linphone.set_log_handler(self.log_handler)

   self.quit_when_registered = False
   self.core = linphone.Factory.get().create_core(callbacks, path + '/config.rc', None)
   self.path = path

 def signal_handler(self, signal, frame):
   self.core.terminate_all_calls()
   self.quit = True

 def log_handler(self, level, msg):
    method = getattr(logging, level)
    method(msg)

 def registration_state_changed(self, core, proxy, state, message):
   if self.quit_when_registered:
     if state == linphone.RegistrationState.Ok:
       print 'Account configuration OK'
       self.core.config.sync()
       self.quit = True
     elif state == linphone.RegistrationState.Failed:
       print 'Account configuration failure: {0}'.format(message)
       self.quit = True

 def call_state_changed(self, core, call, state, message):
   if state == linphone.CallState.IncomingReceived:
     if call.remote_address.as_string_uri_only() in self.whitelist:
        params = core.create_call_params(call)
        core.accept_call_with_params(call, params)
     else:
        core.decline_call(call, linphone.Reason.Declined)

 def message_received(self, core, room, message):
    sender = message.from_address
     if sender.as_string_uri_only() in self.whitelist:
        capture_file = self.path + '/capture.jpg'
       self.core.take_preview_snapshot(capture_file)
        time.sleep(2)
        content = self.core.create_content()
        content.name = 'capture.jpg'
        capture = open(capture_file, 'rb')
        content.buffer = bytearray(capture.read())
        msg = room.create_file_transfer_message(content)
        room.send_chat_message(msg)
   
 def configure_sip_account(self, username, password):
   self.quit_when_registered = True
 
    proxy_cfg = self.core.create_proxy_config()
    proxy_cfg.identity_address = self.core.create_address('sip:{username}@sip.linphone.org'.format(username=username))
    proxy_cfg.server_addr = 'sip:sip.linphone.org;transport=tls'
    proxy_cfg.register_enabled = True
    proxy_cfg.avpf_mode = 1
    proxy_cfg.publish_enabled = True
   self.core.add_proxy_config(proxy_cfg)
   self.core.default_proxy_config = proxy_cfg
 
    auth_info = self.core.create_auth_info(username, None, password, None, None, 'sip.linphone.org')
   self.core.add_auth_info(auth_info)
   
 def run(self):
   while not self.quit:
     self.core.iterate()
      time.sleep(0.03)

if __name__ == '__main__':
  cam = SecurityCamera(whitelist=[''])
 if len(sys.argv) == 4 and sys.argv[1] == 'configure_account':
    cam.configure_sip_account(sys.argv[2], sys.argv[3])
    cam.run()
 else:
    context = daemon.DaemonContext(files_preserve = [ cam.logfile.stream, ],)
    context.open()
    cam.run()

How to use it

First of all, edit the script itself to set which account will be allowed to call your raspberry. For example, I'll allow myself:

cam = SecurityCamera(whitelist=['sip:sylvain.berfini@sip.linphone.org'])

The script simplifies the configuration of the Linphone SIP account you'll use to contact your raspberry-pi. To configure the account, use the following command:

./linphonecam.py configure_account <username> <password>

If you don't have yet a Linphone SIP account, you can create one for free at https://www.linphone.org/free-sip-service.html

You're good to go, just call the account used to on the previous configuration step using a Linphone application (or another SIP client) on which you have your whitelisted account configured on:

Automatic start at boot

If you want your daemon to be started automatically when your raspberry pi boots, you can use the following script:

sudo cp linphonecam /etc/init.d/

Adjust the DIR accordingly (and possibly some other values depending on the name you gave the script):

# Provides:          linphonecam
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Surveillance camera using Linphone
# Description:       Linphonecam allows you to use your device as a surveillance camera using Linphone
### END INIT INFO

# Change the next 3 lines to suit where you install your script and what you want to call it
DIR=/home/pi/linphonecam
DAEMON=$DIR/linphonecam.py
DAEMON_NAME=linphonecam

# Add any command line options for your daemon here
DAEMON_OPTS=""

# This next line determines what user the script runs as.
# Root generally not recommended but necessary if you are using the Raspberry Pi GPIO from Python.
DAEMON_USER=pi

# The process ID of the script when it runs is stored here:
PIDFILE=/var/run/$DAEMON_NAME.pid

. /lib/lsb/init-functions

do_start () {
    log_daemon_msg "Starting system $DAEMON_NAME daemon"
    start-stop-daemon --start --background --pidfile $PIDFILE --make-pidfile --user $DAEMON_USER --chuid $DAEMON_USER --startas $DAEMON -- $DAEMON_OPTS
    log_end_msg $?
}
do_stop () {
    log_daemon_msg "Stopping system $DAEMON_NAME daemon"
    start-stop-daemon --stop --pidfile $PIDFILE --retry 10
    log_end_msg $?
}

case "$1" in

    start|stop)
        do_${1}
        ;;

    restart|reload|force-reload)
        do_stop
        do_start
        ;;

    status)
        status_of_proc "$DAEMON_NAME" "$DAEMON" && exit 0 || exit $?
        ;;

    *)
       echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|restart|status}"
       exit 1
        ;;

esac
exit 0

Finally, apply the changes:

sudo update-rc.d linphonecam defaults

Compiling linphone on the rasberry pi

This procedure is for using the linphone console tools (linphonec, linphonecsh or linphone-daemon) on the raspberry pi.

It assumes that the raspberry pi is installed with a Raspbian image. 

• Install build dependencies from raspbian repositories

sudo apt-get install cmake automake autoconf libtool intltool yasm libasound2-dev libpulse-dev libv4l-dev nasm git libglew-dev

• Clone the linphone-desktop git repository. This repository comprises the linphone source code plus all required dependencies, with an overall build script that builds everything in order.

git clone git://git.linphone.org/linphone-desktop.git --recursive

• Setup compilation options for the whole linphone-desktop project. The line below sets options for a minimal linphone build, without graphical user interface.

cd linphone-desktop
./prepare.py no-ui -DENABLE_OPENH264=ON -DENABLE_WEBRTC_AEC=OFF -DENABLE_UNIT_TESTS=OFF -DENABLE_MKV=OFF -DENABLE_FFMPEG=ON -DENABLE_CXX_WRAPPER=OFF -DENABLE_NON_FREE_CODECS=ON -DENABLE_VCARD=OFF -DENABLE_BV16=OFF -DENABLE_V4L=OFF

• Now proceed with the compilation. This step may take around half an hour and will warm the raspberry !

make -j4

• Once completed, the output executables are in OUTPUT/no-ui/bin. The intermediate compilation products are all located in the WORK directory, that you can safely remove. Should any compilation problem happen, in order to restart the compilation from the beginning, removing the WORK directory is sufficient.

Now the software is ready to be used !

Configuration

First run linphonec once in order to configure your SIP account, so that you can place calls to your raspberry from another place (typically the linphone app on android or iOS!). We recommend to use our free sip.linphone.org service, on which accounts can be created using this online form .

cd OUTPUT/no-ui/bin

./linphonec

[.....]

linphonec> proxy add

[…enter sip account configuration.
We recommend to set proxy address to <sip:sip.linphone.org;transport=tls>
Once done, you should be prompted for your sip account password, and then it should indicate that it is successfully registered. ]
quit

You may also, within the linphonec command prompt, set the sound card to use for capturing sound. The raspberry-pi has no microphone, but if your plan is to plug a USB camera onto the raspberry pi, it probably has a built-in microphone. To tell linphonec to use the microphone from the usb camera, use the "soundcard list" and "soundcard use" commands to select the sound card index to use.

Now open ~/.linphonerc file with an editor (vim, nano...) in order to tweak a few things:

• In section [sound], set

echocancellation=0

Indeed, echo cancellation is not needed, our raspberry pi has no speaker. No need to spend cpu cycles on this.

• In section [video], set vga video size to achieve decent quality, compatible with the pi's processing capabilities:

 

size=vga

 

720p is also possible but the pi2 cpu is a bit too slow for this image format with VP8 codec. svga tends to work not so bad as well.

• Turn on ICE, in section [net] section:

stun_server=stun.linphone.org
firewall_policy=3

Starting linphonec

You can then launch linphonec in background mode in with auto-answer mode. We assume the current directory is already in the OUTPUT/no-ui/bin directory.

export PATH=$PATH:`pwd`
linphonecsh init -a -C -c ~/.linphonerc -d 6 -l /tmp/log.txt

Notice the "-d 6 -l /tmp/log.txt" which are there to tell linphonec to output all debug messages into /tmp/log.txt. Looking into this file might be useful should any trouble arrive.

To stop it, do:

linphonecsh exit

Automatic start at boot

In order to have linphonec automatically started when the raspberry boots: you can add this line to /etc/rc.local :

export PATH=/home/pi/linphone-desktop/OUTPUT/no-ui/bin:$PATH
sudo -u pi linphonecsh init -a -C -c /home/pi/.linphonerc -d 6 -l /tmp/log.txt

The lines above assume that your linphone-desktop source tree is in /home/pi/linphone-desktop. Please adapt if it is not the case.

Just a final reboot and now you can place calls from your favourite linphone app (mobile or desktop) to your raspberry, by calling its sip address !

Common hardware problems

If you are using the RaspberryPi Camera, you'll notice it won't show up as a /dev/video entry, thus it won't be available in linphone for use. To fix that, use the following:

• first enable the camera thanks to the raspi-config utility included in your raspberry.
• tell the raspberry to load the driver specific to the raspberry camera and do it automatically and next reboot:

sudo bash
modprobe bcm2835-v4l2
echo "bcm2835-v4l2" >> /etc/modules
exit

Sadly, the raspberry loosing the network connectivity is a frequent occurrence. Unfortunately, all that NetworkManager stuff included with Raspbian was designed for Linux workstation or servers, and  is not robust to temporary network connectivity losses. They happen quite frequently because of various reasons:

• The internet box looses DSL connection and reboots
• The wifi signal is lost temporarily due to interferences
• The wifi driver has bugs and looses connection after some hours
• The house general power shuts down and comes back due to a thunderstorm, but the raspberry starts faster than the DSL box and will have no internet at boot time.

I recommend to plug the DSL box onto a programable power timer so that it is restarted every day : indeed it is not so rare that a DSL box hangs forever and needs a manual reboot.

In order to force the raspberry to check the network periodically and force a re-connection, I suggest these two scripts, that can be invoked periodically from cron daemon:

• The first one called "restart_wlan0_if_necessary.sh". It just tries a ping to linphone.org, and in absence of response, trigers a shutdown/restart of the wlan0 network interface.

#!/bin/sh
date
if test -z "`ping -w 5 linphone.org |grep time`" ; then
       echo "broken connection, restarting wlan0 now"
        /sbin/ifdown wlan0
        sleep 1
        /sbin/ifup wlan0
       echo "wlan0 restarted."
else
       echo "Everything ok with network."
fi

• The second one, called "reboot_if_necessary.sh". Its goal is to reboot the raspberry if network is still not working, which is the case when the wifi driver or hardware has entered an irrecoverabilly corrupted state.

Sadly, the wifi devices and drivers are generally so bad in terms of robustness, that this kind of trick is necessary.

#!/bin/sh
date
if test -z "`ping -w 5 linphone.org |grep time`" ; then
       echo "broken connection, rebooting now"
        /sbin/reboot
else
       echo "Everything ok with network."
fi

And here's the crontab file:

5 12 * * *       /home/pi/reboot_if_necessary >> /home/pi/reboots.log 2>&1
0 *  * * *       /home/pi/restart_wlan0_if_necessary.sh >> /home/pi/restarts.log 2>&1

Use sudo crontab -e to write these lines aboves.

This will schedule restart_wlan0_if_necessary.sh every hour, and reboot_if_necessary every day at 12:05.