How to avoid core dump when exiting the process in linux/unix

Sometimes in an application, we want to exit the process forcefully (no graceful termination), but do not want the core dump to be generated. One example could be a service that is being monitored by “monit” and if killed is restarted automatically by monit. If this service has multiple threads, it might happen that calling “exit(0)” from main thread will make the memory invalid which might still be being accessed by the threads. This will cause SIGSEGV (Segmentation fault) and will generate core dump. To avoid that, rather than calling exit(0) from the main thread, raise SIGKILL signal i.e. “raise(SIGKILL)”. SIGKILL signal does not generate core dump and will also terminate the process instantly.

boost multithreading and shared mutex example: multiple reader, multiple writer

Here is a simple example of boost multithreading and shared mutex. Hope it helps. The code is quite simple and self explanatory:

#include <iostream> 
#include <boost/thread.hpp>
#include <boost/date_time.hpp>

boost::shared_mutex g_mutex;

using namespace std;

void reader(string id)
{
    cout << id << ": running" << endl;

    boost::shared_lock<boost::shared_mutex> lock(g_mutex);
    if (!lock.owns_lock())
        cout << id << ": Writer has the lock" << endl;
    else
        cout << id << ": Got the lock" << endl;

    // pretend doing something
    boost::this_thread::sleep(boost::posix_time::seconds(3));

    cout << id << ": finished" << endl;
}

void writer(string id)
{
    cout << id << ": running" << endl;

    boost::upgrade_lock<boost::shared_mutex> lock(g_mutex);
    boost::upgrade_to_unique_lock<boost::shared_mutex>  unique_lock(lock);

    cout << id << ": Got the lock" << endl;

    // pretend doing something
    boost::this_thread::sleep(boost::posix_time::seconds(3));

    cout << id << ": finished" << endl;
    boost::this_thread::sleep(boost::posix_time::seconds(1));
}

int main(int argc, char* argv[])
{
    const int writerthreadcount = 2;
    const int readerthreadcount = 2;

    vector writerthread;
    vector readerthread;

    for(int i=0; i<writerthreadcount; i++){
        ostringstream id;  id << "writer" << i+1;
        writerthread.push_back(new boost::thread(writer, id.str()));
    }
    for(int i=0; i<readerthreadcount; i++){
        ostringstream id;  id << "reader" << i+1;
        readerthread.push_back(new boost::thread(reader, id.str()));
    }

//    boost::this_thread::sleep(boost::posix_time::seconds(1));

    cout << "main: waiting for thread" << endl;

    for(int i=0; i<writerthreadcount; i++){         writerthread[i]->join(); delete writerthread[i];
    }
    for(int i=0; i<readerthreadcount; i++){         readerthread[i]->join(); delete readerthread[i];
    }

    cout << "main: done" << endl;

    return 0;
}

FFmpeg sample to decode video using libavformat API

Here is a very simple example to use libavformat and libavcodec library API to decode video:

// ffmpeg_sample.c
// Date: Sep 05, 2013
// Code based on a https://raw.github.com/phamquy/FFmpeg-tutorial-samples/master/tutorial01.c
// Tested on CentOS 5.7, GCC 4.1.2,FFMPEG 0.10.1
// libavcodec.so.53.60.100  libavdevice.so.53.4.100  libavfilter.so.2.60.100
// libavformat.so.53.31.100  libavutil.so.51.34.101  libswresample.so.0.6.100
// libswscale.so.2.1.100
//
// A small sample program that shows how to use libavformat to decode a video file and save it as Y frames.
//
// Build (assuming libavformat, libavcodec, libavutils are correctly installed on your system).
//
// gcc -o sample ffmpeg_sample.c -lavformat
//
// Run using
//
// ./sample myvideofile.mpg
//
// To view the generated output
//
// mplayer -demuxer rawvideo -rawvideo w=[LINESIZE]:h=[HEIGHT]:format=y8 out.raw -loop 0

#include 
#include 

int readsave_frames(int videoStreamIdx
                , AVFormatContext *pFormatCtx
                , AVCodecContext  *pCodecCtx
                , AVCodec         *pCodec
)
{
    int             y, i;
    FILE           *pFile;
    AVPacket        packet;
    int             frameFinished;
    AVFrame        *pFrame;


    // Open file
    pFile=fopen("out.raw", "wb");
    if(pFile==NULL)
    {
        printf("Unable to open output file\n");
        return -1;
    }

    /// Allocate video frame
    pFrame = avcodec_alloc_frame();

    printf("\n");
    for(i=0; av_read_frame(pFormatCtx, &packet) >= 0;) {

        // Is this a packet from the video stream?
        if(packet.stream_index==videoStreamIdx) {
            i++;

            /// Decode video frame
            avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);

            // Did we get a video frame?
            if(frameFinished) {
                printf("\rFrame [%d]: pts=%lld, pkt_pts=%lld, pkt_dts=%lld", i, pFrame->pts, pFrame->pkt_pts, pFrame->pkt_dts);
                // Write pixel data
                for(y=0; yheight; y++)
                    fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, pFrame->linesize[0], pFile);
            }
        }

        // Free the packet that was allocated by av_read_frame
        av_free_packet(&packet);
    }
    printf("\n");

    /// Free the Y frame
    av_free(pFrame);

    // Close file
    fclose(pFile);

    return 0;
}


int main(int argc, char *argv[]) {
    AVFormatContext *pFormatCtx;
    int             videoStreamIdx;
    AVCodecContext  *pCodecCtx;
    AVCodec         *pCodec;
    
    if(argc < 2) {
        printf("Please provide a movie file\n");
        return -1;
    }
    // Register all formats and codecs
    av_register_all();

    pFormatCtx = avformat_alloc_context();

    /// Open video file
    if(avformat_open_input(&pFormatCtx, argv[1], 0, NULL) != 0)
    {
        printf("avformat_open_input failed: Couldn't open file\n");
        return -1; // Couldn't open file
    }

    /// Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL) < 0)
    {
        printf("avformat_find_stream_info failed: Couldn't find stream information\n");
        return -1; // Couldn't find stream information
    }

    /// Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);


    /// Find the first video stream
    {
        int i = 0;
        videoStreamIdx=-1;
        for(i=0; inb_streams; i++)
        {
            if(pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) { //CODEC_TYPE_VIDEO
                videoStreamIdx=i;
                break;
            }
        }
    }
    /// Check if video stream is found
    if(videoStreamIdx==-1)
        return -1; // Didn't find a video stream
    

    /// Get a pointer to the codec context for the video stream
    pCodecCtx = pFormatCtx->streams[videoStreamIdx]->codec;


    /// Find the decoder for the video stream
    pCodec = avcodec_find_decoder( pCodecCtx->codec_id);
    if(pCodec==NULL) {
        fprintf(stderr, "Unsupported codec!\n");
        return -1; // Codec not found
    }

    /// Open codec
    if( avcodec_open2(pCodecCtx, pCodec, NULL) < 0 )
        return -1; // Could not open codec
    
    // Read frames and save them to disk
    if ( readsave_frames(videoStreamIdx, pFormatCtx, pCodecCtx, pCodec) < 0)
    {
        return -1;
    }

    /// Close the codec
    avcodec_close(pCodecCtx);
    
    /// Close the video file
    avformat_close_input(&pFormatCtx);
    
    return 0;
}

To run/view the Y frame output following View Y Frames using mplayer.

View Y frames in Linux using mplayer

To view a file with only Y frames, following command can be used on Linux to view the frames:


mplayer -demuxer rawvideo -rawvideo w=[LINESIZE]:h=[HEIGHT]:format=y8 in.raw -loop 0


To view all the formats available for rawvideo:

mplayer -rawvideo format=help


Reference:
mplayer man page

Writing Simple HTTP Server in Python (With REST and JSON)

There are many already existing powerful http servers that can be used in python e.g. gevent, twisted web server. However, they are a bit complex to use and you cannot start them in a thread other than the main thread.

Here is a sample of basic http server using “BaseHTTPRequestHandler”. The example exposed two rest interfaces:

1. To ingest records into the web server
2. To retrieve already ingested records from the web server

The records are assumed to be JSON based, however, any type of records can be ingested.

from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
from SocketServer import ThreadingMixIn
import threading
import argparse
import re
import cgi

class LocalData(object):
  records = {}

class HTTPRequestHandler(BaseHTTPRequestHandler):
  def do_POST(self):
    if None != re.search('/api/v1/addrecord/*', self.path):
      ctype, pdict = cgi.parse_header(self.headers.getheader('content-type'))
      if ctype == 'application/json':
        length = int(self.headers.getheader('content-length'))
        data = cgi.parse_qs(self.rfile.read(length), keep_blank_values=1)
        recordID = self.path.split('/')[-1]
        LocalData.records[recordID] = data
        print "record %s is added successfully" % recordID
      else:
        data = {}
      self.send_response(200)
      self.end_headers()
    else:
      self.send_response(403)
      self.send_header('Content-Type', 'application/json')
      self.end_headers()
    return

  def do_GET(self):
    if None != re.search('/api/v1/getrecord/*', self.path):
      recordID = self.path.split('/')[-1]
      if LocalData.records.has_key(recordID):
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(LocalData.records[recordID])
      else:
        self.send_response(400, 'Bad Request: record does not exist')
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
    else:
      self.send_response(403)
      self.send_header('Content-Type', 'application/json')
      self.end_headers()
    return

class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
  allow_reuse_address = True

  def shutdown(self):
    self.socket.close()
    HTTPServer.shutdown(self)

class SimpleHttpServer():
  def __init__(self, ip, port):
    self.server = ThreadedHTTPServer((ip,port), HTTPRequestHandler)

  def start(self):
    self.server_thread = threading.Thread(target=self.server.serve_forever)
    self.server_thread.daemon = True
    self.server_thread.start()

  def waitForThread(self):
    self.server_thread.join()

  def addRecord(self, recordID, jsonEncodedRecord):
    LocalData.records[recordID] = jsonEncodedRecord

  def stop(self):
    self.server.shutdown()
    self.waitForThread()

if __name__=='__main__':
  parser = argparse.ArgumentParser(description='HTTP Server')
  parser.add_argument('port', type=int, help='Listening port for HTTP Server')
  parser.add_argument('ip', help='HTTP Server IP')
  args = parser.parse_args()

  server = SimpleHttpServer(args.ip, args.port)
  print 'HTTP Server Running...........'
  server.start()
  server.waitForThread()

Usage:

• Copy paste the code above and name it simplewebserver.py
• Starting WebServer:
  python simplewebserver.py
  
• POST addrecord example using curl:
  curl -X POST http://localhost:8080/api/v1/addrecord/1 -d ‘{\”asif1\”:\”test1\”}’ -H “Content-Type: application/json”
• GET record example using curl:
  curl -X GET http://localhost:8080/api/v1/getrecord/test -H “Content-Type: application/json”

linux bash: login vs non-login shell

I always had confusion about login vs non-login shell and today decided to clear this up. Here is what I have understood. Please refer to the following site for more explanation.

Reference: http://www.linuxfromscratch.org/blfs/view/cvs/postlfs/profile.html

When a user starts The shell program /bin/bash (hereafter referred to as just “the shell”), shell uses a sequence of startup files to create an environment for the user. This includes PATH, PS1, HOME etc. Each file has a specific use and may affect login and interactive environments differently. 

First Some definitions:

Login Shell:

A login shell is one whose first character of argument zero is a -, or one started with the –login option.     It starts after a successful login using /bin/login and by reading the /etc/passwd file.  

Interactive Shell:

An interactive shell is one started without non-option arguments and without the -c option whose standard input and error are both connected to terminals, or one started with the -i option. 

 

Three types of shell sessions:

1. interactive login shell:

Following files are read in the specified sequence to establish user environment.

/etc/profile
~/.bash_profile
~/.bash_login
~/.profile

When a login shell exits, to cleanup, following file is used:

~/.bash_logout

 

2. Interactive non-login shell:

Normally started at the command-line using $/bin/bash) or by the /bin/su command.

This copies the parent environment and then reads the following files, in the specified order, for additional startup configuration instructions:

/etc/bash.bashrc
~/.bashrc

 

3. Non-Interactive Shell:

A non-interactive shell is usually present when a shell script is running. It is non-interactive because it is processing a script and not waiting for user input between commands. For these shell invocations, only the environment inherited from the parent shell is used.

 

Summary:

1. In a Interactive non-login shell, parent environment is used and few settings which are specified in “/etc/bash.bashrc and ~/.bashrc” are updated.

2. Whereas, in an interactive login shell, environment is not inherited from parent and is created from scratch.

 

 

 

FFMPEG stops receiving multicast stream after group membership lost

Recently I came across an issue under CENTOS 5.7 kernel 2.6.18 regarding multicast group membership. I was using ffmpeg 0.10 to receive udp stream over multicast sockets. There happened some disruption in the network and ffmpeg stopped receiving any incoming stream. I checked the incoming stream by starting another ffmpeg instance and that seemed to be working. After some investigation I found that the multicast group membership is lost (does not exist anymore). To check group membership I used:

$ netstat -g

After investigation I found that ffmpeg keeps on waiting for udp multicast packets without checking if the group membership is lost.

To fix this issue, I patched libavformat/udp.c code to check for group membership.

Here are some more investigation results:

1. When network is restarted on Centos 5.7, group membership table is reset. So we have the problem mentioned above.

2. This issue does not appear on ubuntu 11.10 (linux kernel 3.0.0 )as it retains its group membership during network restart.

3. Looking at the linux kernel logs, it seems that this behavior is already fixed in the later version of linux kernel > 2.6.27 and because of that ubuntu 11.10 does not have this issue anymore.

Checking multicast group membership

Following code can be used to check multicast group member under Linux. This code is only for ipv4, but can be extended for ipv6.

Note: I have used netstat source as a reference to write this code.

#define PATH_PROCNET_IGMP              "/proc/net/igmp"
//TODO: Support ipv6
static int udp_check_multicastgroupmembership(struct sockaddr *addr)
{
    FILE *f_igmp;
    char igmp_line[8192];
    char target_addr[10];
    int result = -1;

    if(addr->sa_family != AF_INET)
    {
    	printf("udp.c: IPPROTO_IPV6 NOT SUPPORTED\n");
	return -1; //NOT SUPPORTED
    }

    if(NULL == (f_igmp = fopen(PATH_PROCNET_IGMP, "r")))
    {
    	printf("udp.c: Unable to open %s\n", PATH_PROCNET_IGMP);
        return -1;
    }

    snprintf(target_addr, 9, "%X", ((struct sockaddr_in *)addr)->sin_addr.s_addr);

    if(fgets(igmp_line, sizeof(igmp_line), f_igmp)) {
        if(strstr(igmp_line, "Device") == NULL) {
        	printf("udp.c: IPPROTO_IPV6 NOT SUPPORTED\n");
            fclose(f_igmp);
            return -1;
        }
    }

    result = 0;
    while (!feof(f_igmp)) {
       if(fgets(igmp_line, sizeof(igmp_line), f_igmp)){
           if(NULL != strstr(igmp_line, target_addr)){
            result = 1;
            break;
           }
       }

    };

    fclose(f_igmp);
    return result;
}

How to compile VLC media player source on Ubuntu 10.10 Maverick

Recently I compiled VLC media player source in my Ubuntu 10.10 Maverick. Although there is an online documentation available at “http://wiki.videolan.org/UnixCompile”, yet I faced many dependencies problems during compilation. So, I decided to document most of those issues. The whole process is as below:

1. Prepare your build environment

• setup git (a free open source version control system):
  $ sudo apt-get install git
• setup build tools:
  $ sudo apt-get install build-essential

2. Download source

• change to directory where the source need to be downloaded e.g. $cd ~
• download source using git:
  $ git clone git://git.videolan.org/vlc.git
• a folder ‘vlc’ will be created in your ~ directory. change to that directory: $ cd vlc

3. Bootstrap
$ ./bootstrap
Issues that I faced

• autoreconf: not found:
  sudo apt-get install autoconf
• libtool error:
  sudo apt-get install libtool

After these commands, bootstrap process was successful.

4. Configure
$ ./configure
During configuration, there were many dependency issues reported. Some of them are as below:

• configure: WARNING: libhal >= 0.5.0 was not found. Install libhal-dev ?  &  configure: error: Couldn’t find DBus >= 1.0.0, install libdbus-dev ?
  For both of these use: $ sudo apt-get install libhal
• configure: error: Could not find lua. Lua is needed for some interfaces (rc, telnet, http)
  $ sudo apt-get install liblua5.1-dev
• configure: error: Could not find libmad on your system:

For all of the above and further issues, the command that I used was:
$ sudo apt-get build-dep vlc
This command will install all the dependencies. After running this command ‘configure’ worked without further issues.

5. Make
$ make

6. Run
$ ./vlc