Thursday, 2 July 2009

make variable introspection

Ever wondered how to get value of some Makefile variable? Just invoke:
makespy Makefile MAKEVAR
makespy script below:
#!/bin/sh
original_makefile=`readlink -f $1`
name=$2

shift
shift

get_name_target=_makespy_get_${name}

(
        cat << EOF
include ${original_makefile}

${get_name_target}:
        echo \$($name)
EOF
) | make -C$(dirname $original_makefile) -s -f- ${get_name_target} "$@"

(remember about TAB character at the beginning of echo \$($name))

Usage: makespy MAKEFILENAME VARIABLENAME [makefile options...]

Prints actual value of variable in some makefile.

(works with GNU Make).

Update1 (2012-06-22): fixed a little, note about TAB

Tuesday, 17 March 2009

FreeBSD as NIS client of Linux

For record: FreeBSD has a little different way of storing important login information - Linux stores it in /etc/passwd and /etc/shadow FreeBSD uses only /etc/master.passwd (/etc/passwd is generated from the former for reference).

/etc/master.passwd contains both traditional "passwd" data like name, uid, gid, home shell + password and some administrative values.

So, FreeBSD expects NIS server to provide master.passwd file but and OOB Linux NIS server installation doesn't have it. One must add it to YP database. It's not so hard.

At least on Debian/Ubuntu YP domain data are stored in /var/yp. There is one makefile that simply generates all YP server data files from your ... (choose passwd, group, shadow, hosts ...). Regarding passwords we must generate two databases master.passwd.byuid and master.passwd.byname very similarly as passwd.{byname,byuid} are created. So let's go: Here is python script that takes custom merger output (shadow+passwd) and makes "master.passwd" compatible file.

#!/usr/bin/python

shell_map = {
    "/bin/bash": "/usr/local/bin/bash"
}
def map_shell(shell):
    return shell_map.get(shell, shell)

import sys
for line in sys.stdin.readlines():
    line = line.strip();    
    name, passwd, uid, gid, desc, home, shell = line.split(':')
    desc_fields = desc.split(",")
    gecos = desc_fields[0]
    class_name = ""
    password_change_time = "0"
    account_expiration_time = "0"
    shell = map_shell(shell)
    all = ":".join([name, passwd, uid, gid, class_name, password_change_time, account_expiration_time, gecos, home, shell])
    print all

And here is Makefile snippet that can be inserted to main Makefile or included:

#
# Makefile part for /var/yp/Makefile
#
ALL += master.passwd

MASTER_PASSWD_CONVERTER=python /home/zbigg/projects/nis-linux-freebsd/master.passwd-converter.py ### this is the path to former python script

master.passwd: master.passwd.byname master.passwd.byuid

master.passwd.byuid: $(PASSWD) $(SHADOW) $(YPDIR)/Makefile
 @echo "Updating $@"
 @$(MERGER) -p $(PASSWD) $(SHADOW) | $(MASTER_PASSWD_CONVERTER) | \
 $(AWK) -F: '!/^[-+#]/ { if ($$1 != "" && $$3 >= $(MINUID) && $$3 <= $(MAXUID) && $$3 != $(NFSNOBODYUID) ) \
 print $$3"\t"$$0 }' | $(DBLOAD) -i $(GROUP) -o $(YPMAPDIR)/$@ - $@
 -@$(NOPUSH) || $(YPPUSH) -d $(DOMAIN) $@

master.passwd.byname: $(PASSWD) $(SHADOW) $(YPDIR)/Makefile
 @echo "Updating $@"
 @$(MERGER) -p $(PASSWD) $(SHADOW) | $(MASTER_PASSWD_CONVERTER) | \
 $(AWK) -F: '!/^[-+#]/ { if ($$1 != "" && $$3 >= $(MINUID) && $$3 <= $(MAXUID) && $$3 != $(NFSNOBODYUID) ) \
 print $$1"\t"$$0 }' | $(DBLOAD) -i $(GROUP) -o $(YPMAPDIR)/$@ - $@

 -@$(NOPUSH) || $(YPPUSH) -d $(DOMAIN) $@

Works for me :)

When configuring NIS on hosts i've followed these instructions:

Wednesday, 11 March 2009

hijacking throw

Here's continuation about internals of g++ throw mechanism.

Recent post showed how to effectively detect any exception type at runtime. It lacked one important information - where was the exception thrown. Who have made this.

Who is responsible for "bad argument", "logic_error" or anything. Who? Who nows?

On g++ exceptions are raised using __cxa_throw call. This is is runtime call that given given exception object (and some stuff) starts searching for exception handler and starts unwinding stack. Eventually it aborts if no handler was found.

How about hijacking this call? In gnu tool chain it's easy. __cxa_throw resides in libstc++.so. Lazy linkage and dlopen permits us to ovoerride __cxa_throw and the load original version from appropriate libstdc++.so. Well look at source code:

#include <typeinfo>
#include <stdexcept>
#include <iostream>
#include <dlfcn.h>

typedef void (*cxa_throw_type)(void* , void *, void (*) (void *));
cxa_throw_type orig_cxa_throw = 0;

void load_orig_throw_code()
{
 void* orig_libcxx = dlopen("/usr/lib64/libstdc++.so.6", RTLD_LAZY);
 std::cerr << "loaded orig libc++ " << orig_libcxx<< "\n";
 orig_cxa_throw = (cxa_throw_type)( dlsym(orig_libcxx, "__cxa_throw") );
 std::cerr << "loaded orig_cxa_throw " << (void*)orig_cxa_throw << "\n";
 dlclose(orig_libcxx);
}

extern "C" 
void __cxa_throw(void *thrown_exception, void *pvtinfo, void (*dest) (void *) ) {
 std::type_info const* tinfo = reinterpret_cast<std::type_info const*>(pvtinfo); 
 std::cerr << "YEAH! detected throw of exception of " << tinfo->name()<< "\n";
 if( orig_cxa_throw == 0 )
  load_orig_throw_code();
 orig_cxa_throw(thrown_exception, pvtinfo, dest);
}

void foo() {
 std::cerr << "throwing exception\n";
 throw std::runtime_error("akuku");
}

int main()
{
 try { foo(); }
 catch(std::exception& e) {
  std::cerr << "exception successfully caught at main\n";
 }
 return 0;
}
Voila. The output:
$ ./hijack
loaded orig libc++ 0x2a95587000
loaded orig_cxa_throw 0x34e35af1a0
throwing exception
YEAH! detected throw of exception of St13runtime_error
exception successfully caught at main
How there are some problems and things to do:
  • how the hell should we know what is the path to libstdc++.so (it can't be hardcoded as in this proof-of-concept)
  • we must use backtrace and some "debug info" retriever to show where is the exception throws
  • initialization of orig_cxa_throw must be more safe

But for now it works at least on 32&64 bit linuxes. I've got to test in on some production code.

Tuesday, 10 March 2009

boost size

I acknowledge that Boost is a great library. Great interfaces, stability, design and what not. Great people behind ...

But think a little:

#include <boost/format.hpp>
using g++ and boost 1.37 yields literally
  • 147 includes
  • 15899 lines of source code!
  • 509257 bytes of source code!
Madness!

Friday, 6 March 2009

catch(...) on g++

Ever wondered how to get information what was actually caught you enter catch(...) clause ?

With g++ (on some popular platforms) it's possible retrieve pointer to type_info record of base exception object type and what more important, pointer to exception object itself.

[Note, all information comes from http://www.codesourcery.com/public/cxx-abi/abi-eh.html]

Let's go ...

First, of all g++ runtime stores exception handling state in some structure that is different for each thread:

struct __cxa_eh_globals {
   __cxa_exception * caughtExceptions;
   unsigned int        uncaughtExceptions;
};

You can obtain pointer to instance via

extern "C" __cxa_eh_globals* __cxa_get_globals();
call.

Next, when you got this you can look at first exception that you have - it's the one currently handled (i assume you're still in catch(...) clause). The __cxa_exception structure is quite almost obvious:

struct __cxa_exception {
   std::type_info * exceptionType;
   void (*exceptionDestructor) (void *);
   void* unexpectedHandler;
   void* terminateHandler;
   __cxa_exception * nextException;

   int   handlerCount;
   int   handlerSwitchValue;
   const char *        actionRecord;
   const char * languageSpecificData;
   void *  catchTemp;
   void *  adjustedPtr;

   void*         unwindHeader;
};

Here most important member are:

adjustedPtr
pointer to exception object
exceptionType
type_info of exception object

Having that we can retrieve information for any type of exception object we know. std::exception - just call what(), std::string (yeah who throws them) - just print it, CORBA::exception, all messy Xerces exceptions ... whoa!

Sample proof of concept. Tested on following configurations

Meat:

#include <map>
#include <string>
#include <typeinfo>
#include <iostream>
#include <sstream>

// catch clause analysis

struct __cxa_exception { 
    std::type_info * exceptionType;
    void (*exceptionDestructor) (void *); 
    void* unexpectedHandler;
    void* terminateHandler;
    __cxa_exception * nextException;

    int   handlerCount;
    int   handlerSwitchValue;
    const char *        actionRecord;
    const char * languageSpecificData;
    void *  catchTemp;
    void *  adjustedPtr;

    void*         unwindHeader;
};

struct __cxa_eh_globals {
    __cxa_exception * caughtExceptions;
    unsigned int        uncaughtExceptions;
};

extern "C" __cxa_eh_globals* __cxa_get_globals();

// yet another rtti like attempt
struct extended_type_info {
    virtual const char* type_name() = 0;
    virtual std::string to_string(void*) = 0;
    
    virtual ~extended_type_info() {}
};

typedef std::map<std::string, extended_type_info*> handler_map;

template <typename Callable>
void execute(Callable c, handler_map const& exc_handlers)
{
    try {
        c();
    } catch( ... ) {
        __cxa_eh_globals* gp = __cxa_get_globals();
        
        if( gp == 0 && gp->caughtExceptions == 0 ) {
            std::cerr << "unknown exception caught (unable to find exception record)" << "\n";
        } else {
            __cxa_exception* cp = gp->caughtExceptions;
            std::type_info* t = cp->exceptionType;
            void* exc_obj_ptr = cp->adjustedPtr;
            std::string exc_name(t->name());
            handler_map::const_iterator ieh = exc_handlers.find(exc_name);
            if( ieh != exc_handlers.end() ) {
                extended_type_info* ti = ieh->second;
                std::cerr << "exception caught " << ti->type_name() << "(" << ti->to_string(exc_obj_ptr) << ")\n";
            } else {
                std::cerr << "exception caught: " << t->name() << ", no handler found\n";
            }                
        }
    }
}

struct int_extended_type_info: public extended_type_info  {
    const char* type_name() { return "int"; }
    std::string to_string(void* p) {
        int* v = reinterpret_cast<int*>(p);
        std::ostringstream ss;
        ss << *v;
        return ss.str();
    }
};

struct std_string_extended_type_info: public extended_type_info  {
    const char* type_name() { return "std::string"; }
    std::string to_string(void* p) {
        std::string* v = reinterpret_cast<std::string*>(p);
        return *v;
    }
};

template <typename T>
class singletonof {
public:
    static T* get_ptr() { return &instance; }
private:
    static T instance;
};

template <typename T>
T singletonof<T>::instance;

// the code

void akuku()
{
    throw 321;
}

void akuku_string()
{
    throw std::string("akuku");
}

int main(int argc, char** argv)
{
    handler_map hm;
    hm["i"] = singletonof<int_extended_type_info>::get_ptr();
    hm["Ss"] = singletonof<std_string_extended_type_info>::get_ptr();
    execute(&akuku, hm);
    execute(&akuku_string, hm);
}
And the output is:
$ ./execution_monitor.exe
exception caught int(321)
exception caught std::string(akuku)
Cool!

Update 1 [2009-03-06]: typo(sstream), added info about tested platforms

Update 2 [2009-03-09]: tested also on gcc/solaris 9