image

MMSoftware - Software

 image

Main

Main page

MMSoftware

News
Software
Documentation
Repositories

Various

Image Gallery

Links

Software I Enjoy
MMMail
MMBookmark
Open source software

Various other software may also be found within the repository, although these may merit some highlight.

mmstatd | apache-mmstat | mmftpd | mmmail | mmspawnd2 | mmserver-mmserver2 | mmlist-mmpool-mmhash | pthread_util | js-appserv-sh | ecl-unix | bktr | tap-bridge | utf-8 | xisop | mystic-ships | paradise-adventure | symbol-game | irclog | crow-httpd | analog-term

mmstatd

 (top)

mmstatd is a small database server, C library and command-line client designed to store key-based counters for statistics or simple configuration storage. These up to date statistics may then be queried in real-time. The C library allows C software to query and modify those using transactions, and the server internally uses a recovery log.

mmstatd is used by mmftpd, mmmail and apache-mmstat. It may be found at the following locations:

apache-mmstat

 (top)

This utility can easily be used with apache so that it pipes a custom log format trough it, for various statistics to become maintained by mmstatd.

mmftpd

 (top)

At the time I wrote this FTP server, none that I knew were open source for unix, properly and easily supported virtual users (simulating chroot for each), running fully unprivileged and under a real chroot(2). I also wanted a single file to serve as an accounts database, and another simple configuration file. Internally using the mmserver library, it initially used GNU Pth threads. Over time, the pthread_util library was written to solve some missing functionality from POSIX threads, and the mmserver library was modified to use pthread. Later on, IPv6 support was also added.

Other features include configurable per-address rate and connection limits as well as bandwidth throttling.

mmmail

 (top)

Following the writing of mmftpd, I also wanted a non-privileged, chroot-capable non-relaying SMTP server and its POP3 server companion. Originally, MySQL was used as the backend, but eventually the code was modified to use PostgreSQL and file storage for message content, which the setups running it had migrated to. Like mmftpd it also uses the mmserver library so now uses POSIX threads and supports IPv6 as well.

This mail server is highly tuned against spam. It performs various sanity checking and allows filtering based on headers or user/mailbox specific envelope based filters which users may configure via an HTTP interface. The latest HTTP interface to MMMail is unfortunately not open source, however.

Other features include configurable per-address rate and connection limits as well as bandwidth throttling.

mmspawnd2

 (top)

mmspawnd2 consists of an inetd-like server but an instance of the server is needed for every service. However, more control is allowed on a service, such as setrlimit(2), chroot(2), running unprivileged, binding privileged ports to then drop privileges before starting serving connections, doing some sanity checking before executing the target executable, etc. This server uses the mmserver2 library and thus supports IPv6 and maintains a dynamic pool of pre-forked processes for performance. Connection and rate limits are also supported. Has been used a long time to host anoncvs CVS services with the help of mmanoncvs.

mmserver and mmserver2

 (top)

mmserver was initially a library for writing unprivileged network IPv4 TCP servers using the GNU Pth portable user-threads library. Although mainly I/O bound, which user-threads can be suited for, a special facility was designed using a pool of processes along with a messaging system so that parallel tasks, or ones which might require a fair amount of stack, be served by non-threaded processes.

As POSIX threads implementations became more widely available and more mature, conforming to POSIX more and more (LinuxThreads didn't conform and NetBSD had no kernel threads at the time), the pthread_util library was eventually written and mmserver was modified to use POSIX threads. The non-threaded pool of processes is still available and may still be used for cases where threads are unappropriate, like for using non-threadsafe library calls, tasks needing a lot of stack, etc. Later on, support for IPv6 was also added. This library is used by mmftpd and mmmail.

mmserver2 consists of another similar library but which is designed to use processes rather than threads. This model is more adequate for some types of applications and provides enhanced reliability. A pool of processes is maintained to serve the clients in parallel, such as is done in Apache 1.3, and mmserver2 supports AF_LOCAL, IPv4 and IPv6 TCP/Stream and UDP/Datagram. mmserver2 is used by mmspawnd2 and js-appserv, and has also been used in proprietary server code. This library also permits to reuse, and to eventually recycle processes of the pool to for instance recover memory leaks from a third party library, and this is tunable by every application. Connection/request rate and limits are also supported.

mmlist, mmpool, mmhash

 (top)

mmlist is a library of C macros to help dealing with doubly linked lists, which are a common data structure used in software.

mmpool consists of an efficient fixed-sized objet pool-based allocator, with some slab-allocator features such as asynchroneous object creation/destruction and reuse of recently used memory pages.

mmhash is a general purpose hash table library, also a frequently used data structure. Custom 32-bit hash functions, as well as object-specific comparision functions may be provided, and the tables may be tuned/customized via various options. Collisions are dealt with, as well as automatic table resizing and rehashing.

pthread_util

 (top)

The pthread_util library provides utility features for use with POSIX threads. It first implements an efficient Amiga/PTH like message passing interface for inter-thread synchronization, where rings can be used for polling. It then also implements a poll(2) variant which can be used to poll on both file descriptors and message rings at a time. Along with this new polling primitive, asynchroneous connect(2)/accept(2) variants are implemented. Other facilities include a managed pool of threads to which tasks may easily be dispatched in parallel. Finally, some thread sleep variants are also implemented which may also wake up upon events on a ring.

This library was initially written to replace missing POSIX threads library features which the GNU Pth library allowed, to then allow migrating mmserver (and thus, mmftpd and mmmail), to POSIX threads. It also has found other uses since.

js-appserv and js-sh

 (top)

This was written in C, using the SpiderMonkey Mozilla JavaScript engine. The shell (js-sh) allows to run various programs written in JavaScript on unix systems. As for the unix server (js-appserv), it consists of a versatile server supporting IPv4, IPv6 and local sockets. This server is written in C but delegates control to JavaScript server applications in an elegant way. It uses the mmserver2(3) library.

Various libraries are provided such as the following:

Eventual support might be provided for:

Also provided are example programs:

Here are the relevant links:

ecl-unix

 (top)

ecl-unix is an interface to NetBSD syscalls and various other important libc interfaces for the ECL implementation of Common Lisp. It is written to use ECL's native C inline features. This allows ECL to be used on NetBSD to develop various kinds of server and client applications easily, with good performance and minimal dependencies. Support for kqueue(2) is also included.

It makes uses of custom SETF expansions and lexically scoped MACROLETs to override DEFMACRO-defined global macros to very efficiently inline C structure accessors where the code is intended to be compiled. In cases where serializing whole structures at once is more convenient, this is used instead of working via structure accessors. For interpreted or third party code, where random access to a C structure object is necessary, accessor functions are provided for C structures.

This code is still a work in progress.

bktr_multicast, bktr_client and bktr_httpd

 (top)

bktr_multicast consists of an UDP multicasting server, listening to BSD bktr(4) interfaces (device for bt8x8 video capture devices). It marks images with timestamp and source, then compresses them in mjpeg format. It then broadcasts those frames over the LAN, after tagging them with the current timestamp, source and a type of noonce (to prevent replay attacks) and an RSA signature (to prevent third party frame insertion, allowing clients to assert their authenticity). Capture rate and parameters are configurable, as well as the simplistic motion detection system. Was used on NetBSD with a PCI 4 bktr ports board.

bktr_client is an SDL-based client which displays frames sent by bktr_multicast for the wanted source. It supports pause and save features. It also ensures to only decode frames with a valid RSA signature. This client was used successfully on NetBSD, Linux and Windows.

bktr_httpd is an efficient, I/O bound single-threaded HTTP server using kqueue(2), which can stream image frames originating from a wanted bktr_multicast source to many web clients. It supports HTTP authentication and Keep-Alive, as well as some type of bandwidth modulation to adapt the frame rate to the link speed of every client. Only frames with a valid RSA signature are streamed.

tap-bridge

 (top)

This is a modular daemon designed to act as a bridge/proxy between a physical ethernet device and a NetBSD tap(4) virtual one to learn more about network protocols and permit to easily write prototype network stacks and low-level applications as userspace modules. Packet capture and injection are possible, and a few facilities are provided to modules for ethernet/IP header deserializing/serializing as well as delayed send queues and a facility for persistent property/variable which may last when modules are reloaded (i.e. at reception of SIGHUP). A diagram of the system can be found here.

UTF-8/UTF-32 encoder/decoder

 (top)

More and more applications make use of the UTF-8 external format and of unicode, and UTF-8 is also the standard format of several network protocols. This library implements a UTF-8 to UTF-32 decoder and an UTF-32 to UTF-8 encoder. I needed a comprehensive and efficient C interface which was under a non-restrictive license, for use as third party extensions to software which could not be under the GPL, and had to write this implementation.

Xisop

 (top)

Once upon a time, the author was an Amiga user. This was a microkernel with intefaces and design similar to that of AmigaOS. An eventual offspring of Xisop had to be licensed under more restrictive terms and development stopped on the open source original, but here is the old code anyway. This version only runs on Amiga (it was developped on NetBSD using an m68k cross-compiler targetted at the UAE emulator at the time).

The Mystic Ships

 (top)

After discovering Netrek, playing it, running a bronco server for some time, and hosting a third party CVS repository for bronco-server and cow-client, I got interested in possibly developing a similar game from scratch. There also was an interleving research project which could have benefited from it.

Some work was done on the basic system including an encrypted and compressed, adaptively streamed TCP-based client-server protocol, a kqueue(2) based server allowing a large number of concurrent clients efficiently and an SDL-based portable client (NetBSD, Linux, OSX, Windows). However, as the challenging parts which were interesting to me were fully implemented, interest in the game itself diminished and development was stopped. However, a little effort was put in 2014 to revive it, so that it is at least playable.

Paradise Adventure

 (top)

Since a few of my family members have a drawing, painting and music hobby, and that I have a type of nostalgy for old adventure games and am myself musician and compositor, I started writing an SDL-based text+image adventure game. It started out well, but lack of time and lack of family support stalled development. The existing part is more a demo than anything else, and is portable (tested on NetBSD, Linux and Windows). An early screenshot is also available.

If I ever pick the project up seriously again, I'll most probably use ECL with it such that the world and events system be implemented in Common Lisp.

Please note that although the code is BSD-licensed, the artwork (images and music) is not licensed for public distribution or use.

Symbol Game

 (top)

This was a test I wrote when learing Common Lisp. The main goals were to learn how to develop an application interactively, how to make custom use of the package system, dynamic symbols manipulation and simple macros. symbol-game runs on ECL or SBCL.

irclog

 (top)

I previously had written irclog.js, but wanted to rewrite it in Common Lisp. This implementation is also more complete as it can humanize output. It was nice to develop as a fair amount of progress could be made interactively in SLIME while not having to reconnect the client to the IRCd. It was designed for ECL. It will be used to log certain important events happening on IRC.

Crow-httpd

 (top)

Although I wrote various HTTP servers before (in C and Javascript), this is a work in progress server written in Common Lisp, specifically for ECL on unix. It attempts to properly implement HTTP/1.1 and remain lightweight, minimizing its external dependencies. Its performance is very acceptable, and using ECL makes it easy to use and produce OS-friendly modules and executables.

Because of the power of Lisp, neat tricks are possible like interactive development, compile-time template to text/html generation, embedding those HTML templates elegantly into code (not as text, but as hierarchical s-expression objects with conditional and loop commands), providing an HTTP-LET for dynamic application user code to easily bind GET/POST/Cookie variables, etc.

Future possibilities could also include compilation of lispy code to JavaScript, a live transactional object storage system, the development of very high-level web frameworks using CLOS, etc.

It thus consists of an interesting learning experience. If I'm satisfied with it once It's more complete, future actual applications might get written with it. Crow-httpd also currently hosts this site, replacing the previous js-httpd that hosted it for years.

AnalogTerm I & ][

 (top)

AnalogTerm 1 is based on XScreenSaver 5.35's Apple2 module, that was turned into a more complete 80 columns terminal emulating xterm. It is nice although it requires some CPU time whenever the window is visible. It has a more realistic NTSC emulation than AnalogTerm ][ does.

AnalogTerm ][ is a reemplementation from scratch, using a simpler analog-like scanline emulation system. Other than the cursor, blinking text or application output, the screen doesn't need a high framerate constant refresh and the CPU can be idle. Higher resolutions and larger windows are also possible. An attempt is also made to produce partial screen refreshes for performance. XShm is also supported (AnalogTerm 1 also used it), which enhances performance when running non-remotely. This terminal also supports copy/paste and optional colors, unlike version 1. Its functionality is also expected to be more complete, as the old AnalogTerm will not be updated often. Other functionality include unicode for Latin and graphics characters and dead keys to enter accents, (like version I).